Robots.txt Generator

You give it two things: the paths to block, one per line, and your sitemap address. It writes a single rule group that applies to every crawler (User-agent: *), turns each path into a Disallow line, and adds the Sitemap line at the end. The starting values produce:

Blank lines in the paths box are skipped and spaces around each path are trimmed. Clear the sitemap box and the Sitemap line is left out.

The output box has a copy button but no download. Paste the text into a plain-text editor, save it as robots.txt, and upload it to the root of your site so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored.

Each path is copied into the file exactly as you type it, so small details matter:

• Start every path with a slash. /admin is a valid rule. admin is copied as typed and doesn't match the way you intend.
• A rule matches everything that starts with it. The default /cart also blocks /cart-guide and /cartoon-prints. To block only the folder, use /cart/. To block the cart page and its folder without catching /cart-guide, list /cart$ and /cart/; the $ marks the end of the URL for Google and other major crawlers.
• Wildcards pass straight through. Typing /*?sort= gives Disallow: /*?sort=, which blocks sorted listing pages with any path.
• Paths are case-sensitive. /Admin and /admin are different rules.

The generator only writes Disallow rules for all crawlers. Two common additions have to be typed into the file after you copy it:

• An exception inside a blocked folder. Add an Allow line to the same group, for example Allow: /admin/help/ under Disallow: /admin/.
• Rules for one crawler only. Start a new group with its own User-agent: line and put that crawler's rules under it. A crawler that finds a group with its own name follows that group and ignores the * group, so repeat any shared rules there too.

Keep the Sitemap line where it is. It isn't part of any group and applies to the whole file.

Before you add a path to keep a page out of search results, know what the rule does. Disallow asks crawlers not to fetch a URL. It doesn't remove the URL from the index. If other pages link to a blocked URL, a search engine can still list the address, usually with no description, because it knows the page exists but was asked not to read it.

To keep a page out of results, let it be crawled and add a noindex robots meta tag. The Meta Robots Tag Generator on this site writes one. Use the generator here for areas that have no value in search, such as carts, checkouts and admin screens.

After uploading, open the robots.txt report in Google Search Console to confirm Google fetched the new version without errors.