robots.txt: What to Block, What Never to Block

A robots.txt file lives at the root of your domain — https://example.com/robots.txt — and is plain text. The structure is a series of records, each consisting of one or more User-agent lines followed by directives.

• User-agent: * — applies to all crawlers.
• User-agent: Googlebot — applies only to Google's crawler.
• Disallow: /path/ — blocks that path for the named agent.
• Allow: /path/ — explicitly permits a path, even inside a disallowed parent.
• Crawl-delay: 5 — asks crawlers to wait 5 seconds between requests (Googlebot ignores this; use Google Search Console instead).

Rules are case-sensitive. /Admin and /admin are different paths. A blank Disallow: line means allow everything, which is the default behavior anyway. Comments start with #.

Minimal valid file that allows all crawlers everywhere:

User-agent: *
Allow: /

Block anything that creates duplicate content, exposes private data, or wastes crawl budget on pages with zero search value.

• Admin and login pages — Disallow: /admin/, Disallow: /wp-admin/. These have no ranking value and you do not want them indexed.
• Internal search results — Disallow: /search?. Search-result pages are near-duplicate and Google's own guidelines recommend blocking them.
• Faceted navigation / filter URLs — e.g., Disallow: /shop/?sort= or Disallow: /products/?color=. Each parameter combination can create thousands of near-identical pages.
• Cart and checkout paths — Disallow: /cart/, Disallow: /checkout/.
• Staging or dev environments — block the entire site with Disallow: / on staging subdomains or add HTTP auth so robots.txt is irrelevant.
• Duplicate pagination beyond a threshold — Disallow: /*?page= if you have thousands of archive pages with thin content.

These mistakes show up regularly in SEO audits and can tank rankings silently:

• CSS, JS, and image files — Blocking /wp-content/ or /assets/ prevents Googlebot from rendering your pages correctly. Google needs to see the same resources a browser sees in order to understand your content.
• Your sitemap — Some generators add a Disallow: /sitemap.xml by accident. Never do this.
• Canonical pages — If /product-name/ is the canonical URL you want ranked, it must not be disallowed anywhere in the file.
• Hreflang pages — If you rely on hreflang tags for international SEO, all language variants must be crawlable; otherwise Google cannot verify and process the hreflang annotations.
• Structured data pages — Pages with schema markup (Product, Recipe, Article) need to be crawled for rich results to appear.

This is the most consequential confusion in technical SEO. Robots.txt and noindex do completely different things, and using the wrong one backfires.

The critical trap: if you block a page in robots.txt and add a noindex tag, Google cannot see the noindex because it respects the robots.txt block. The page may stay in the index indefinitely. Choose one mechanism per page based on the goal.

robots.txt is the standard place to declare your sitemap location. Add this line anywhere in the file (it is not part of any User-agent block):

Sitemap: https://example.com/sitemap.xml

You can list multiple sitemaps:

Sitemap: https://example.com/sitemap-blog.xml
Sitemap: https://example.com/sitemap-products.xml

Google, Bing, and most major crawlers read this directive. It is the fastest way to ensure your sitemap is discovered without manually submitting it in every search console. If you use a sitemap index file, point to the index — you do not need to list every individual sitemap.

Crawl budget is the number of URLs Googlebot will crawl on your site within a given time window. For most small sites under ~1,000 pages, crawl budget is not a concern. It becomes critical when:

• Your site has more than ~10,000 URLs.
• You have large e-commerce catalogs with filter and sort parameters generating millions of URL variations.
• Your server is slow and Googlebot is backing off due to load.
• Important new pages are taking weeks to get indexed.

robots.txt is the right tool to reclaim wasted crawl budget because blocked URLs are not fetched at all. By contrast, a noindex page still gets crawled — it just is not indexed. Blocking parameter-heavy faceted navigation with robots.txt can reduce crawl waste by 60–80% on large retail sites, freeing Googlebot to discover and index product pages faster.

A practical audit workflow: export your server logs, identify which URLs Googlebot is hitting most, check whether those URLs have search value, then disallow the patterns that do not. Do not guess — use data.

Wildcards: the * character matches any sequence of characters. The $ character anchors a match to the end of the URL. These are supported by Google but not all crawlers — check documentation for any third-party bot you are targeting specifically.