Robots.txt Generator

Generate a robots.txt file with allow, disallow and sitemap rules — free robots.txt generator.

About

About the Robots.txt Generator

Builds a robots.txt file with allow/disallow rules, an optional sitemap link and crawl-delay, instead of hand-writing it yourself.

Site owners use it to keep crawlers out of admin areas or staging paths while still pointing them to a sitemap, while developers launching a new site use it to make sure it isn't accidentally left fully open — or fully blocked — to search engines. It's one of many free SEO tools on Toolapt, and like every tool here, it's free to use with no sign-up and no limit on how many times you run it.

Choose what to view about this tool

Guide

Using the Robots.txt Generator

  1. 1

    Step 1

    Choose the default access: allow all pages, or disallow all pages.

  2. 2

    Step 2

    List any paths to disallow, and optionally any paths to explicitly allow, one per line.

  3. 3

    Step 3

    Add a sitemap URL and/or a crawl-delay in seconds if you want them included.

  4. 4

    Step 4

    Copy the generated robots.txt and upload it to your site's root directory.

  • Sets a clear default — allow everything or disallow everything — as the starting point
  • Adds specific Disallow and Allow path rules on top of that default
  • Includes an optional Sitemap directive pointing search engines to your sitemap.xml
  • Includes an optional Crawl-delay directive to slow down aggressive crawlers
  • Runs entirely in your browser — your rules are never uploaded anywhere
  • Free to use, with no limit on how many times you run it
  • Fully responsive, so it works just as well on a phone as on a desktop

Site owners

Site owners use the Robots.txt Generator for keeping crawlers out of admin areas or staging paths while still pointing them to the site's sitemap.

Developers launching a new site

Developers launching a new site use the Robots.txt Generator for making sure the site isn't accidentally left fully blocked to search engines at launch.

SEO specialists

SEO specialists use the Robots.txt Generator for controlling exactly which paths crawlers are allowed to index across a large site.

DevOps engineers

DevOps engineers use the Robots.txt Generator for adding a crawl-delay when a server needs to limit how aggressively bots hit it.

Examples

Robots.txt Generator examples

3 worked examples

01

Blocking an admin area

Set default access to Allow all pages, add /admin/ and /private/ under Disallow, and a sitemap URL, to generate lines like Disallow: /admin/, Disallow: /private/ and Sitemap: https://example.com/sitemap.xml.

02

Locking down a staging site

Set default access to Disallow all pages to generate User-agent: * followed by Disallow: /, blocking every well-behaved crawler from the entire site.

03

Slowing down aggressive bots

Set Crawl-delay to 10 seconds to add a Crawl-delay: 10 line, asking crawlers that respect it to wait between requests.

Related tools

SERP Snippet Preview

See how a page's title, URL and meta description will look in Google search results, with pixel-width checks.

Use it

Open Graph Preview

Preview how a link will look when shared on Facebook, LinkedIn and X, and get the Open Graph tags for it.

Use it

Schema Markup Generator

Generate JSON-LD structured data for an article, FAQ page, organization or breadcrumb trail.

Use it

UTM Link Builder

Add utm_source, utm_medium and utm_campaign tracking parameters to a link for Google Analytics.

Use it

Slug Generator

Turn a title into a clean URL slug, with accents stripped.

Use it

Word Frequency Counter

Which words appear most often, with counts and percentages.

Use it

FAQ

Frequently asked questions

Quick answers to common questions about the Robots.txt Generator.

What does the Robots.txt Generator do?

It builds a robots.txt file with allow/disallow rules, an optional sitemap link and an optional crawl-delay, based on the choices you make.

Is the Robots.txt Generator free to use?

Yes. It's free, with no sign-up, no watermark and no limit on how many times you use it.

Is my input safe when I use this tool?

Yes. The file is generated entirely on your own device — nothing you enter is sent to a server.

Where does robots.txt need to go?

In the root directory of your site, so it's reachable at yoursite.com/robots.txt — crawlers won't find it anywhere else.

Does robots.txt guarantee a page won't be indexed?

No. It's a request that well-behaved crawlers respect, not an enforcement mechanism — a page can still be indexed if it's linked to from elsewhere, and it doesn't stop crawlers that ignore the file.

What does Crawl-delay do?

It asks crawlers to wait a given number of seconds between requests to your site, which can help reduce server load — though not every crawler honors it.

Can I use the Robots.txt Generator on my phone?

Yes. The page is fully responsive and works on mobile, tablet and desktop browsers alike.

Does the Robots.txt Generator have any limitations?

It builds a single, general rule set rather than per-crawler (per-user-agent) rules, which covers the vast majority of everyday robots.txt needs.

Still have a question? Contact us

Report a problem

Robots.txt Generator

What went wrong?
    or paste a screenshot