Search Crawler Engine
Robots.txt Generator
Generate a perfectly formatted robots.txt file to instruct Googlebot and other search engines on how to crawl your website.
Crawling Rules
Global Settings
Restricted Directories
Specific Search Bots
Googlebot
Google Image
Bingbot
Yahoo (Slurp)
YandexBot
Baiduspider
Live File Output