Use this tool to generate a simple robots file for your website, allowing you to hide files or directories that you dont wish to be spidered. This generator tool is designed to create the text for the file for you and for you to then make changes should you feel necessary to afterwards.
This robots.txt syntax checker checks the contents of a site's robots.txt against that contained in the latest specification, along with providing warnings on the use of new features that are not yet widely deployed.
