To stay away from undesirable articles during the search indexes, site owners can instruct spiders never to crawl certain information or directories from the standard robots.txt file in the root directory of your domain. Furthermore, a site is usually explicitly excluded from a search engine's database through the use of a meta tag certain to robot