Robots.txt Generator

Decide what needs blocking

Start with a real crawl problem, not a long list of folders copied from another website. An internal search route or an endless set of filter combinations might need attention. A useful article, product page or stylesheet usually needs to remain available.

For a small site with no crawl restrictions, an empty Disallow: line is a reasonable starting point. Adding more rules does not make the file more effective. Every extra restriction is another thing to check when a page disappears from a crawler report.

Be careful with Disallow: /: it blocks all paths for the relevant crawler unless a more specific allowed exception applies. This generator asks you to acknowledge broad blocking rules. That confirmation does not make them suitable for your site.

Build the file

Enter the origin where the file will live, such as https://example.com. Add a crawler product token, such as Googlebot, or use * for the general group. A full browser user-agent header does not belong in that field. Put tokens on separate lines when they share the same rules.

Add an Allow or Disallow row for each path. Paths begin with /; use * for a wildcard and a final $ for an exact ending. Keep path case as it appears on your site. Encode non-ASCII characters and literal spaces before entering rule paths.

Sitemap addresses are optional. Use the complete URL of an existing sitemap or sitemap index, one per line. The generator does not create those files or check whether their addresses return successfully. Our XML Sitemap Generator handles reviewed URL lists separately.

Test both sides

A useful test includes a URL you mean to block and another you mean to keep accessible. With the WordPress starting point, test /wp-admin/, then /wp-admin/admin-ajax.php. The second URL has a more specific Allow rule. An ordinary public page, such as /about/, should remain allowed.

The result identifies the matching line in the generated file. It does not request the URL or inspect your current live robots.txt. An allowed result therefore says nothing about the page’s HTTP status, canonical tag or indexation.

Test crawler-specific groups separately. A specific group does not inherit the general group’s rules; repeated groups for the same token are combined. When equally specific path rules conflict, Allow takes precedence. These details follow Google’s published interpretation.

Crawling is not indexing

Blocking a URL here is not a reliable way to remove it from search results. Google may know about the address through links even when it cannot fetch the page. If the goal is exclusion from search, investigate a crawlable noindex directive instead. If the content is private, require authentication.

Robots rules are public instructions, not access controls. Do not put secret addresses in this file expecting them to become protected. Also avoid blocking scripts and styles needed to understand your pages. Google’s introduction explains those limitations, and RFC 9309 defines the protocol.

Check before replacing

Keep a copy of your existing file. If WordPress or an SEO plugin already manages robots.txt, review that configuration before uploading a physical replacement. The WordPress starting point here is a draft, not a recommendation to overwrite custom rules.

Place the reviewed file at the origin’s root: https://example.com/robots.txt. The HTTP version, another subdomain or a different port has its own scope. Open the exact address after uploading and confirm that it serves plain text, not a login page or an HTML error.

Use Search Console to investigate Google’s crawl and indexing evidence after a change. This tool does not submit files to Google or promise faster indexing. For a page-level review, the SEO Checker can inspect the server-returned HTML. Your draft rules are not retained in this plugin’s report cache or operational log.

Plan the next placement

Crawl settings and publisher selection solve different problems. Once your target page is ready, compare relevant publishers below or ask our outreach team to research a shortlist. Approved guest post orders retain the dofollow placement and Google indexing guarantee stated in their confirmed terms. That service commitment is separate from this file generator.

Common questions

Request a shortlist

Share your target page, audience and preferred market. We will confirm publisher fit and placement terms before you approve an order.