WordPress has a virtual robots.txt file; best not to muck around with the filesystem. There are a bunch of plugins to allow you to edit the robots.txt.

I choose to use WP Robots.txt, it is simple and the least intrusive that I found, its control panel is added to Settings -> Reading dashboard panel. This is explained in the support forum.

I took advice from stackflow and spiderwebpress to add the following line

sitemap: http://wiki.davelevy.info/sitemap.xml

The spiderweb article has some more advice about disallowing the filesystem since the content isn’t there.

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.