robots.txt Generator

Generate robots.txt files to control how search engines crawl your website.

Configuration

Sitemap URL (optional)

User Agents

Crawl Delay (seconds, optional) Time between requests. Not all bots support this.

Quick Presets

robots.txt

About robots.txt

What is robots.txt?

A text file that tells search engine crawlers which pages or files they can or can't request from your site.

Common Directives

User-agent: Target bot name
Disallow: Block path
Allow: Allow path
Sitemap: Sitemap location

Important Notes

Place at website root
Not a security measure
Not all bots respect it
Case sensitive paths