I found this really good website for information about managing website crawlers using a robot.txt file (and meta tags). Really nice website with great information.

http://www.robotstxt.org