Indexability Checker

Start with the exact URL

A page opening in your browser does not settle whether a crawler can access it. Your browser may have a login cookie; a public request may receive a different response. Check the address you actually want in search, including its protocol, hostname and path.

This indexability checker makes fresh public requests. It records the page’s HTTP redirect chain, fetches robots.txt for each origin encountered and evaluates those files for the Googlebot web token. The request itself identifies as EduGuestPost’s diagnostic checker, not as Googlebot.

Separate two decisions

Crawlability is about access. Indexing directives concern what a search engine should do with the document it reads. A robots.txt block and a page-level noindex are not interchangeable. Google may know a blocked URL exists without being able to fetch its content.

Suppose a staging rule still blocks /guides/ after launch. Removing a noindex tag from one guide would not fix that crawl restriction. Conversely, opening the directory to crawling would not remove a noindex header added by the server. Check both layers. Google’s noindex guidance explains why a crawler needs access to see that instruction.

Read the matching rule

The report keeps the fetched robots text and, where available, the matching line. A broad restriction can have a more specific exception. For example, Disallow: /private/ and Allow: /private/public/ need different results for two URLs inside those paths. The checker uses the bundled robots-parser library for matching rather than checking whether a file merely contains the word Disallow.

Read HTTP findings carefully. A 404 at the robots address is not the same as a 404 at the page address. For robots.txt, Google’s handling treats most 4xx responses as no restrictions, except 429. A server failure or rate-limited robots response needs investigation: this tool cannot know Google’s cached rules or retry history. See Google’s robots.txt specification.

Check the final response

When the submitted URL redirects, the extracted page tags belong to the final response. Confirm that the destination is intentional. An old campaign page pointing to a general homepage may be technically reachable while still being the wrong destination for visitors.

The report separates robots meta tags and HTTP indexing headers from canonical declarations. It flags noindex, retains declaration evidence and calls out a different canonical or competing URLs. A declared canonical is not proof of Google’s choice. Use the Canonical Tag Checker for a separate check of the declared destination, or the Redirect Checker for more detailed redirect work.

Do not remove every directive simply to get a cleaner report. A private account page may intentionally be excluded. Likewise, nofollow is not a noindex instruction for the page itself. Conditional directives, including unavailable_after and indexifembedded, need context; their presence is flagged without resolving dates or embedded-page exceptions. Google’s robots directive reference covers those distinctions.

What a clean result means

No common blocker observed means exactly that: no common restriction was found in the responses inspected. It does not confirm Google indexing, content quality, rendered JavaScript, soft-404 handling or sitemap membership. Authentication, network location and bot-specific responses can also produce different evidence.

Requests have a 35-second total budget. Page inspection reads up to 256 KiB and follows up to three configured redirects. Each origin’s robots inspection follows up to five redirects and reads up to 500 KiB. Incomplete chains, oversized files and challenge pages are labeled for review, not passed. No page body or robots report is saved by this module.

After a deliberate fix, recheck the same URL and compare the evidence. For a property you control, continue in Search Console’s URL Inspection and Page indexing reports. That is the place to investigate Google’s own crawl and indexing information; repeating a third-party technical check cannot establish it.