robots.txt Generator

Build a clean robots.txt from a curated catalog of 36 known bots — search engines, AI crawlers, social previewers, SEO tools, uptime monitors, and shopping bots. Pick each bot's mode (Allow all, Block all, or Custom rules), add your sitemap URLs and an optional Yandex-style Host directive, and see a live preview of the canonical file with a lint pass. Copy or download. Everything stays in your browser.

Wildcard User-agent: *

Rules here apply to every bot that doesn't have its own block. Most crawlers honour the wildcard by default.

Bots

(no bot selected)

Sitemaps & Host

    Import existing robots.txt

    Paste an existing file to seed the picker. Wildcard rules become the wildcard block; each named user-agent group becomes a bot block.

    robots.txt

    — 
     
     
     
     
    Lint
     
     
     
     
     How the file is built 
       
    • Wildcard block is always emitted first. Empty rules → just User-agent: * with no directives (most crawlers treat that as “no block”).
      •  
    • Allow all mode: bot block is omitted entirely (it inherits the wildcard rules).
      •  
    • Block all mode: emits Disallow: / on a fresh user-agent line.
      •  
    • Custom mode: emits one Allow: or Disallow: line per rule, in source order. Add rules with the editor on the left.
      •  
    • Sitemaps are file-scoped, emitted at the end (one Sitemap: line each). Must be absolute http(s)://.
      •  
    • Host is the Yandex-style canonical-host directive. Only Yandex and a few Russian crawlers honour it.
      •  
    • Crawl-delay is emitted when set on a bot block; the wildcard block does not currently get one (move it to a per-bot block if needed).
      •  
    • Everything stays in this tab. Nothing is uploaded, nothing is sent to a server.
      •