How do I disallow specific page from robots.txt

Asked 2010-08-15T06:29:50.520

42

I am creating two pages on my site that are very similar but serve different purposes. One is to thank users for leaving a comment and the other is to encourage users to subscribe.

I don't want the duplicate content but I do want the pages to be available. Can I set the sitemap to hide one? Would I do this in the robots.txt file?

The disallow looks like this:

Disallow: /wp-admin

How would I customize to the a specific page like:

http://sweatingthebigstuff.com/thank-you-for-commenting

robots.txt

Edit Report

1 Answer

65

Disallow: /thank-you-for-commenting$

in robots.txt

Take a look at last.fm robots.txt file for inspiration.

The dollar sign marks an end. We need that so other pages starting with the same url, for example /thank-you-for-commenting-another-page, will be indexed.

answered 2010-08-15T06:43:32.577

Your Answer