13
I want to allow crawlers to access my domain's root directory (i.e. the index.html file), but nothing deeper (i.e. no subdirectories). I do not want to have to list and deny every subdirectory individually within the robots.txt file. Currently I have the following, but I think it is blocking everything, including stuff in the domain's root.
User-agent: *
Allow: /$
Disallow: /
How can I write my robots.txt to accomplish what I am trying for?
Thanks in advance!