Robots.txt is one of the oldest and simplest files in web development — literally a text file in the root directory. But it has gained renewed relevance in 2026. With the rise of AI crawlers like GPTBot, ClaudeBot, and PerplexityBot, robots.txt is no longer just about controlling Googlebot — it is about who gets access to your content at all. This guide covers everything from basic syntax to advanced AI bot strategy.
What robots.txt is and how it works
Robots.txt is a text file located in the root directory of your website (example.com/robots.txt). It tells search engines and other crawlers which parts of the website they are allowed to visit, and which they should avoid. It is important to understand that this is a request, not a blocking mechanism — well-behaved crawlers respect it, but malicious bots can ignore it.
When a crawler visits your website, robots.txt is the first thing it looks for. If the file does not exist, the crawler assumes everything is accessible. If the file exists but is empty, the same applies. Only explicit Disallow rules restrict crawling.
In 2026, robots.txt is more important than ever because the number of active crawlers has exploded. Beyond Googlebot, Bingbot, and other search engines, you now have AI training crawlers, AI search crawlers, SEO tool crawlers, and countless other bots. Without a well-thought-out robots.txt, you spend server resources on crawlers that provide no value.
Syntax: User-agent, Allow, Disallow, and Sitemap
The syntax in robots.txt is simple, but precision is critical. A single wrong character can block the entire website from indexing. Here are the key directives you need to know.
User-agent specifies which crawler the rule applies to. Use * for all crawlers, or specify by name: Googlebot, Bingbot, GPTBot, etc. Disallow specifies paths that should not be crawled. Allow specifies exceptions to Disallow rules. Sitemap points the crawler to your XML sitemap.
- User-agent: * — applies to all crawlers
- User-agent: Googlebot — applies to Google only
- Disallow: /admin/ — blocks everything under /admin/
- Allow: /admin/public/ — allows exception within blocked folder
- Sitemap: https://example.com/sitemap.xml — points to sitemap
- Crawl-delay: 10 — asks crawlers to wait 10 seconds between requests (ignored by Google)
Strategic blocking: What you should block
Not everything on your website should be crawled. Strategic blocking is about directing crawl budget to pages that actually provide value in search results, and keeping crawlers away from pages that waste resources.
Block admin pages, staging environments, shopping cart and checkout, user dashboards, internal search results pages, filter and sort pages with parameters, print versions of pages, and API endpoints that should not be indexed. For e-commerce sites, it is especially important to block faceted navigation that generates thousands of parameter-based URLs.
Read our index bloat guide to understand why this is so important for larger websites.
Robots.txt vs noindex: When to use which
This is one of the most common confusions in technical SEO. Robots.txt and noindex do fundamentally different things, and using the wrong tool can produce catastrophic results.
Robots.txt prevents crawling — meaning Google never sees the page content. Noindex (via meta tag or HTTP header) asks Google not to include the page in the index, but Google must first crawl the page to discover the noindex tag. The problem arises when you block crawling in robots.txt AND add noindex — Google can never see the noindex tag because it never crawls the page.
Use robots.txt for pages that consume crawl budget without value (filtering pages, duplicate content). Use noindex for pages that Google already crawls but that you do not want in the index (thank you pages, internal search). Read more in our technical SEO checklist.
AI bot management: GPTBot, ClaudeBot, and PerplexityBot
This is where things get really interesting in 2026. You now have the ability to control whether AI companies can use your content for training and whether AI search engines can display your content in generated answers. These are two entirely different decisions.
GPTBot (OpenAI) crawls for language model training. ChatGPT-User crawls for ChatGPT Search (real-time answers). ClaudeBot (Anthropic) crawls for training. PerplexityBot crawls for Perplexity search answers. Google-Extended controls whether Google uses your content for AI training (separate from regular search).
Our recommendation: Block AI training crawlers (GPTBot, ClaudeBot, Google-Extended) unless you have a specific reason to let them train on your content. But allow AI search crawlers (ChatGPT-User, PerplexityBot) — they send traffic back to your website with citations. Read our LLM visibility guide for a deeper discussion of this trade-off.
- User-agent: GPTBot / Disallow: / — block OpenAI training
- User-agent: ChatGPT-User / Allow: / — allow ChatGPT Search
- User-agent: ClaudeBot / Disallow: / — block Anthropic training
- User-agent: PerplexityBot / Allow: / — allow Perplexity search
- User-agent: Google-Extended / Disallow: / — block Google AI training
Testing with Google Search Console
Google Search Console has a built-in robots.txt testing tool under "Settings > Crawling > robots.txt." Use this tool to verify that your rules work as expected before publishing changes.
Test specific URLs to see if they are blocked or allowed. Check especially the edge cases — pages that sit near the boundary between allowed and blocked. A single misplaced slash can cause an entire folder to be blocked or allowed unintentionally.
Want professional help with technical SEO and crawl management? LeapJet conducts thorough technical audits covering robots.txt, crawl budget, and indexing strategy. Book a free analysis. Also read our Google Search Console guide.
Robots.txt mistakes that kill SEO
The worst mistake is the simplest: Disallow: / under User-agent: * blocks the entire website from all crawling. We have seen this happen after new website launches where developers forget to remove the staging robots.txt. It often takes weeks before anyone notices — and by then the website has fallen out of the index.
Other critical mistakes include: blocking CSS and JavaScript files (Google needs these to render pages correctly), blocking images (they cannot appear in Google Images), and incorrect use of Crawl-delay that slows indexing unnecessarily.
A subtle but common mistake is using robots.txt to block pages you want removed from the index. Remember: robots.txt prevents crawling, not indexing. If other websites link to a page you have blocked in robots.txt, Google can still index the URL (without the content) based on anchor text and other signals.
Robots.txt for WordPress, Next.js, and Shopify
Implementation varies depending on platform. In WordPress, robots.txt is generated automatically, but you should customize it with a plugin like Yoast SEO or by editing the file manually. Default WordPress robots.txt is too simple for most websites.
In Next.js, you can place a static robots.txt in the public folder, or generate it dynamically via an API route. Dynamic generation is useful if you want different rules for staging and production. In Shopify, the robots.txt file is automatically generated and difficult to modify — you can do it via theme.liquid, but changes are limited.
Regardless of platform: always test changes in a staging environment before publishing to production. And have a backup plan — an error in robots.txt can do more damage in a short time than almost any other technical SEO change.
Example templates for common scenarios
Here are practical templates you can use as a starting point. Customize them for your specific website — there is no universal robots.txt that fits everyone.
For a business website: Block /admin/, /wp-admin/, /api/, /internal/ and search result pages. Allow everything else. Add a Sitemap reference. For an e-commerce site: In addition to the above, block filtering URLs, shopping cart, checkout, and user account pages.
For AI bot strategy: Add separate blocks for each AI crawler you want to manage. Combine with our glossary for explanations of all terms used in this article, and see /services/seo for professional help with implementation.
Sources
Google Search Central, "Robots.txt Specifications," 2025. Google Search Central, "Large Site Management and Crawl Budget," 2024. OpenAI, "GPTBot Documentation," 2024. Anthropic, "ClaudeBot Documentation," 2024. Perplexity, "PerplexityBot FAQ," 2025. Yoast, "WordPress Robots.txt Best Practices," 2025.
