Crawl
Updated: 10/02/2017 by Computer Hope
Crawl describes a bot, script, or program that visits a web page and grabs its content and links. Once completed, the program visits the next link in the list or a link obtained from the web page it had recently visited.