Dashboard

Robots.txt generator

Robots.txt generator and rule tester

Build a valid robots.txt file for your site in seconds. Choose which common paths to block, block AI scrapers with one click, add custom rules, and test any URL with the live tester.

  • Free
  • No signup
  • Runs in your browser
Larky standing guard beside an open robots.txt file window with allow and disallow rule badges

Website configuration

Used to format absolute sitemap addresses and check upload locations. robots.txt lives at the root: /robots.txt.

Common crawler paths

Checked paths will be blocked from every crawler. Safe defaults protect admin dashboards and search spam without risking public pages.

AI crawlers and scrapers

Choose which automated bots to shut out. Blocking AI training stops model dataset scrapes; blocking AI search stops conversational citations.

AI trainingCollect pages to train AI models. Blocking them does not affect Google or Bing search.
AI searchFetch pages so AI assistants can show and cite them in answers. Blocking them can cost you that traffic.
SEO toolsBuild the link databases of SEO tools. Blocking saves server load; your links from other sites still show up in them.
Custom rules (0)

Specific directives for any user agent. Use * for all crawlers, or specify a crawler like Googlebot.

No custom rules added. Click “+ Add rule” to add custom paths.

Separate multiple sitemaps with line breaks or commas. Defaults to https://yourproduct.com/sitemap.xml.

Live URL tester

Test how search engines evaluate any page on your domain against your current robots.txt rules (RFC 9309).

✗ Crawling blockedGooglebot → /admin/

Matches DISALLOW: /admin/ in User-agent: *.

Validation checks 1 to check

  • CheckDomain: Enter your domain so the sitemap address and upload location are right.
  • GoodCrawling: Search engines can crawl your homepage and public pages.

Generated robots.txt

Place this file at the root of your web server: /robots.txt

# All crawlers
User-agent: *
Disallow: /admin/
Disallow: /search/
Disallow: /search?

Crawl rules ready? Get indexed and discovered

Launch on LarkLaunch for free: get a full week on the launch leaderboard, a permanent dofollow backlink, and visibility among founders and early adopters.

Launch your product

How to use the robots.txt generator

  1. Enter your domain: Type your domain (e.g. yourproduct.com). The generator automatically extracts your host origin and constructs the right sitemap URL.
  2. Select common crawler paths: Enable or disable recommended blocks for back offices, internal search result pages, checkout funnels, or CMS admin areas.
  3. Configure AI scrapers and search bots: Choose whether to block AI model training crawlers (like GPTBot and ClaudeBot) while keeping conversational search bots (like OAI-SearchBot and PerplexityBot) open for citations.
  4. Add custom rules: Add specific Allow or Disallow directives for individual crawlers or all user agents.
  5. Test and download: Use the live URL tester to verify that your paths match expected rules according to RFC 9309, then copy the text or download your robots.txt file.

Safe defaults vs critical robots.txt mistakes

A single typo in a robots.txt file can accidentally remove an entire website from search engine results. Here are the crucial rules to protect your site’s organic search presence:

  • Never block CSS, JS, or fonts: Search engines like Googlebot render your pages using an evergreen browser engine. Blocking stylesheets or script bundles prevents Google from rendering your page layout, often causing indexing drops or mobile-friendliness penalties.
  • robots.txt is NOT access control: robots.txt is a publicly viewable plain text file. Listing sensitive private directories like /secret-admin-folder-xyz/ informs anyone reading the file exactly where your internal endpoints reside. Secure sensitive pages with authentication, not robots.txt directives.
  • Disallow does not guarantee de-indexing: If other sites link to a URL blocked in robots.txt, search engines can still index the link and show the URL in search results without reading page content. To ensure a page is omitted from search results, allow it to be crawled and place a <meta name="robots" content="noindex" /> tag on the page.
  • Always use trailing slashes for directories: Writing Disallow: /admin blocks not just /admin/, but also public pages like /administration-guide or /administer-payments. Writing Disallow: /admin/ restricts the block to the intended folder.

AI training scrapers vs AI search engines

As generative AI and answer engines grow, crawlers now fall into two distinct groups with different implications for your traffic:

  • AI model training scrapers (such as GPTBot, ClaudeBot, Google-Extended, and CCBot): These fetch content to train future foundation models. Blocking them prevents your content from being ingested without harming your rankings in traditional Google or Bing search.
  • AI conversational search crawlers (such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot): These power real-time conversational search engines and direct user queries. If you block these bots, assistants cannot browse your content or cite your product as a source in answer cards.

How rule matching works (RFC 9309)

Modern search engines follow the official IETF standard for robots exclusion (RFC 9309):

  • Group precedence: A crawler first searches for a User-agent group matching its specific name (e.g. Googlebot). If a matching group exists, it reads only that group and completely ignores the generic User-agent: * group.
  • Longest match wins: Within a group, rules are evaluated by specificity. The longest matching path pattern decides whether a URL is allowed. For example, if you have Disallow: /blog/ (6 characters) and Allow: /blog/announcement (19 characters), the URL/blog/announcement is allowed.
  • Allow breaks ties: If an Allow rule and a Disallow rule have the exact same length and match the same URL, the standard dictates thatAllow wins.

Where to upload your file

Save your file as robots.txt (in lowercase) and upload it to the exact top-level root of your site:https://yourdomain.com/robots.txt. Crawlers never look for robots.txt in subdirectories.

If your product uses subdomains (such as app.yourdomain.com and docs.yourdomain.com), each subdomain requires its own independent robots.txt file.

In Next.js App Router, you can create a static file at app/robots.txt, or export a dynamic function from app/robots.ts returning a MetadataRoute.Robots object.

After uploading your robots.txt, audit your complete setup using the Website SEO checker, generate rich previews and Open Graph tags with the Meta tag generator, and check your domain’s backlink authority with the Domain rating checker.

Frequently asked questions

What is robots.txt and does my site need one?

robots.txt is a plain text file at the root of your domain that tells search engines and automated crawlers which pages they may request. You need one to stop crawlers from wasting time on private dashboards or internal search results, and to point them directly to your sitemap.

Does robots.txt remove pages from Google search results?

No. If other websites link to a blocked address, Google can still index the URL and display it in search results without reading the page content. To remove a page completely from search engines, keep it crawlable and add a "noindex" meta robots tag or X-Robots-Tag header instead.

Which paths should I block by default?

Block back-office areas like /admin/, internal search result queries, and checkout funnels. Never block public pages, and never block CSS, JavaScript, or image assets: search engine crawlers need stylesheets and scripts to render and index your pages accurately.

Should I block AI training crawlers?

Blocking AI training bots like GPTBot, ClaudeBot, and Google-Extended stops your content from being ingested into foundation model datasets without affecting your regular organic search rankings. Be cautious when blocking AI search bots like PerplexityBot or OAI-SearchBot if you want your site cited in conversational answers.

What is the difference between Allow and Disallow rules?

Disallow instructs matching user agents not to access URLs starting with that prefix. Allow creates an exception inside a disallowed directory. Under RFC 9309 and Google rules, when both match a URL, the longer, more specific rule wins, with Allow breaking ties.

Where should I upload my robots.txt file?

Upload it to the exact root of your website: https://yourdomain.com/robots.txt. It cannot be located in a subfolder like /blog/robots.txt. Each subdomain (e.g. app.yourdomain.com) also requires its own separate robots.txt file.

Does Google respect Crawl-delay in robots.txt?

Googlebot does not support the Crawl-delay directive. Bing, Yandex, and Yahoo do honor Crawl-delay. If you need to manage Googlebot crawl rates, adjust your server capacity or use Google Search Console rate controls.