Trafficfolio

Home / Blocked by robots.txt in Search Console: How to Fix It

Blocked by robots.txt: what it means and how to fix it

A rule in your robots.txt file tells Google not to crawl these URLs. Google respects it, so it cannot read the pages.

Updated

What it means

Before crawling, Google reads /robots.txt on your domain. A Disallow rule matching the URL stops the crawl. The page may still appear in search with no description if other sites link to it, which is a separate status: Indexed, though blocked by robots.txt.

Common causes

  • A Disallow: / left over from a staging or development site.
  • A broad rule like Disallow: /blog that also blocks pages you want indexed.
  • Rules blocking CSS or JavaScript folders that pages need to render.
  • A CMS or plugin setting such as “discourage search engines” turned on.

How to fix it

  1. Find the rule. Open https://yoursite.com/robots.txt and look for a Disallow line matching the URL path. The robots.txt report in Search Console Settings shows the version Google last fetched.
  2. Decide. If the URL should stay private or out of search, the block is working. Admin, cart and internal search pages are often blocked on purpose.
  3. Remove or narrow the rule if the page should be crawled.
  4. To keep a page out of search entirely, allow crawling and add a noindex tag. Google cannot see a noindex tag on a page it is not allowed to crawl.

Check that the fix worked

  1. Open the page in URL Inspection and choose Test live URL. This shows what Google sees now, not what it saw on its last visit.
  2. If the live test looks right, choose Request indexing for the pages that matter most.
  3. In the Pages report, open the issue and choose Validate fix. Google rechecks the listed URLs, which can take a few days to a few weeks.

After that, watch clicks and impressions for the affected pages. A free new site checkup also flags common indexing blockers on any URL.

Questions

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing. Use a noindex tag on a crawlable page to keep it out of results.

How fast does Google see a robots.txt change?

Google usually caches robots.txt for up to a day. You can ask for a recrawl from the robots.txt report in Search Console settings.

Related guides

See all your sites in one place

Connect Google and Bing once and your portfolio fills in.

Create account