Home / Blocked by robots.txt in Search Console: How to Fix It
Blocked by robots.txt: what it means and how to fix it
A rule in your robots.txt file tells Google not to crawl these URLs. Google respects it, so it cannot read the pages.
Updated
What it means
Before crawling, Google reads /robots.txt on your domain. A Disallow rule matching the URL stops the crawl. The page may still appear in search with no description if other sites link to it, which is a separate status: Indexed, though blocked by robots.txt.
Common causes
- A
Disallow: /left over from a staging or development site. - A broad rule like
Disallow: /blogthat also blocks pages you want indexed. - Rules blocking CSS or JavaScript folders that pages need to render.
- A CMS or plugin setting such as “discourage search engines” turned on.
How to fix it
- Find the rule. Open
https://yoursite.com/robots.txtand look for aDisallowline matching the URL path. The robots.txt report in Search Console Settings shows the version Google last fetched. - Decide. If the URL should stay private or out of search, the block is working. Admin, cart and internal search pages are often blocked on purpose.
- Remove or narrow the rule if the page should be crawled.
- To keep a page out of search entirely, allow crawling and add a noindex tag. Google cannot see a noindex tag on a page it is not allowed to crawl.
Check that the fix worked
- Open the page in URL Inspection and choose Test live URL. This shows what Google sees now, not what it saw on its last visit.
- If the live test looks right, choose Request indexing for the pages that matter most.
- In the Pages report, open the issue and choose Validate fix. Google rechecks the listed URLs, which can take a few days to a few weeks.
After that, watch clicks and impressions for the affected pages. A free new site checkup also flags common indexing blockers on any URL.
Questions
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. Use a noindex tag on a crawlable page to keep it out of results.
How fast does Google see a robots.txt change?
Google usually caches robots.txt for up to a day. You can ask for a recrawl from the robots.txt report in Search Console settings.
Related guides
See all your sites in one place
Connect Google and Bing once and your portfolio fills in.
Create account