Alex Rivera | Logout

Questions tagged robots.txt

11 questions

Sort by: Newest Votes Active
24
votes
2
answers

robots.txt file for different domains of same site

asp.net-mvc-4 seo robots.txt
asked 2013-06-10T22:21:53.577
21
votes
4
answers

How to stop search engines from crawling the whole website?

security .htaccess robots.txt
asked 2012-02-01T20:34:07.153
15
votes
2
answers

Web Crawler - Ignore Robots.txt file?

python web-crawler mechanize robots.txt
asked 2011-12-05T14:05:02.620
59
votes
3
answers

robots.txt allow root only, disallow everything else?

robots.txt
asked 2011-08-29T05:31:18.880
9
votes
1
answers

How to configure robots.txt file to block all but 2 directories

seo robots.txt google-search
asked 2011-06-23T21:35:23.417
13
votes
2
answers

Listing both sitemaps and sitemap index files in robots.txt?

sitemap robots.txt
asked 2011-06-15T19:09:12.417
13
votes
2
answers

Robots.txt, how to allow access only to domain root, and no deeper?

robots.txt
asked 2011-03-05T20:32:30.200
14
votes
5
answers

Disallow or Noindex on Subdomain with robots.txt

robots.txt
asked 2011-02-05T01:58:50.163
10
votes
3
answers

Sitemap for a site with a large number of dynamic subdomains

search-engine sitemap robots.txt google-search-console
asked 2010-10-07T10:11:59.190
42
votes
3
answers

How do I disallow specific page from robots.txt

robots.txt
asked 2010-08-15T06:29:50.520
38
votes
10
answers

Ethics of robots.txt

robots.txt
asked 2009-06-16T00:02:22.443