Robots.txt Generator
Rules for specific agents, plus a sitemap line. Output is plain text you can paste and serve at the root.
robots.txt
—
Worked example
Allow and Disallow are matched by longest prefix, not by order. A Disallow: / followed by Allow: /tools/ lets crawlers into /tools/ — the allow is more specific and therefore wins.
robots.txt tells cooperative crawlers what not to fetch. It is not access control: a crawler that ignores it, or a scraper written by someone who wants your data, is unaffected. Anything genuinely private needs authentication.
Rules that bite
Disallow: / with no matching allow blocks the whole site, and it is the single most common cause of a site vanishing from search. A stray Disallow: / in a staging copy that gets deployed is a classic outage.