Noun
Web crawler (plural Web crawlers)
(Internet) Synonym of Web spider
Architectures High-level architecture of a standard Web crawler A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture. Source: Internet
By the time a Web crawler has finished its crawl, many events could have happened, including creations, updates and deletions. Source: Internet