Robots.txt

File used to direct or tell web bots which pages and directories to index or not index. This file must be placed in the root directory on the server hosting your pages. The file should be named robots.txt and should have read permissions. Below is an example of a robots.txt file.

#robots.txt file for https://www.computerhope.com
User-agent: *
Disallow: /directory1/
Disallow: /directory2/

In the above robots.txt example, the first line is a nonexecutable statement that describes the file. Next, the User-agent line tells what robots or bots should follow this section of the file. In the above example, the * represents all bots. Finally, the next two lines are the two directories that you want the bots not to index.

To allow a bot to query all URLs in your website, enter the bot name followed by disallow: with a blank value. An example of this configuration is shown below.

User-agent: Somebot*
Disallow:

Internet terms, Meta tags, Robot, SEO terms, Site map, Web design terms