Robots.txt Blocking Your Site From Google? How to Fix It
A client calls in a panic: their site was ranking fine last week, and now it's nowhere in Google. No downtime, no error pages, nothing broken in the browser. Almost every time this happens with zero site-side symptoms, the real cause is one of a handful of crawl-blocking settings - and they're all quick to check once you know where to look.
Symptom: The Site Loads Fine, But Google Can't See It
The pattern is usually one of these:
- Pages that used to rank have dropped out of search results entirely
- Google Search Console shows pages under "Excluded" with a reason like
Blocked by robots.txtorExcluded by noindex tag - A brand-new site or migrated site never shows up in search at all, even after weeks
- Only some pages are missing - often ones on a specific plugin, category, or URL pattern
The site itself works. Visitors don't see anything wrong. That's the giveaway - this isn't a hosting outage or a broken page, it's a crawler being told to stay away.
Cause: Four Places a Crawl Block Can Hide
1. A Disallow rule that's too broad
Someone (a developer, a migration script, or a plugin default) added a rule like this to robots.txt and it never got removed:
User-agent: *
Disallow: /That single line blocks the entire site from every well-behaved crawler. It's the single most common cause we see, especially right after a migration - staging sites are supposed to ship with this rule, and it's easy to forget to strip it out before going live.
2. A leftover noindex meta tag
Separate from robots.txt, a page can carry its own instruction in the HTML head:
<meta name="robots" content="noindex, nofollow">This is common when a staging copy gets cloned to production and the noindex setting comes along for the ride, or when an SEO plugin's per-page "Discourage search engines" toggle gets flipped by mistake.
3. WordPress's own search engine visibility switch
WordPress has a site-wide checkbox that writes a noindex directive into every page's <head>: Settings → Reading → Search Engine Visibility. It's meant for sites still under construction, and it's the classic "we launched three weeks ago and still aren't ranking" cause. It's on by default on fresh installs, which is exactly why it gets missed.
4. An X-Robots-Tag HTTP header
This one is invisible in the page source because it's not HTML at all - it's an HTTP response header, usually set in .htaccess, an Nginx config, or by a CDN/WAF rule:
Header set X-Robots-Tag "noindex, nofollow"We see this most often left behind by a security plugin or a Cloudflare Page Rule that someone set up temporarily and forgot about.
Fix: Check Each Layer in Order
Step 1: Read the live robots.txt
Don't guess - pull the actual file being served:
curl -s https://yourdomain.com/robots.txtLook for any Disallow: / under User-agent: *. If your site should be fully crawlable, the file should look something like this instead:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xmlStep 2: Fix it in cPanel's File Manager
Log in to cPanel → File Manager → navigate to public_html/ (or the document root for that domain) → open robots.txt and edit it directly. If the file doesn't exist yet, create it there - a missing robots.txt isn't a problem on its own, but it's worth adding a clean one with a Sitemap line for good measure.
Step 3: Check WordPress's visibility setting
In wp-admin, go to Settings → Reading and confirm "Discourage search engines from indexing this site" is unchecked. If you're managing this via WP-CLI on a staging-to-production deploy script, you can check and clear it in one line:
wp option get blog_public
wp option update blog_public 1Step 4: Check for page-level noindex tags
View the page source (Ctrl+U) and search for robots. If you're running Yoast SEO or Rank Math, check the plugin's per-page "Advanced" tab too - it can override the global setting for individual posts or the whole post type.
Step 5: Check the response headers
This is the one people forget, because it never shows up in the page source:
curl -I https://yourdomain.com/If you see X-Robots-Tag: noindex in the output, search your .htaccess file and any CDN/firewall rules (Cloudflare Page Rules, a WAF, a security plugin) for where it's being added, and remove it.
Step 6: Ask Google to recrawl
Once everything's clean, don't just wait it out. In Google Search Console, use URL Inspection on your key pages, confirm it now says "URL is available to Google," and click Request Indexing. For a whole-site fix, resubmitting your sitemap under Sitemaps nudges Google to recrawl faster than it would on its own.
Quick Reference: Common Robots.txt Lines
| Line | What it does |
|---|---|
Disallow: / | Blocks the entire site from that user-agent |
Disallow: /wp-admin/ | Blocks the admin area only (normal and expected) |
Disallow: /*?s= | Blocks internal search result URLs (usually fine) |
Allow: /wp-admin/admin-ajax.php | Carves out an exception inside a blocked folder |
Sitemap: https://yourdomain.com/sitemap.xml | Points crawlers to your XML sitemap |
Prevention
- Add a go-live checklist for every migration or relaunch: robots.txt, WordPress Reading settings, and header rules all get checked before DNS is cut over, not after.
- Keep staging environments behind HTTP authentication instead of relying on noindex/robots.txt alone - a password prompt can't accidentally leak into production the way a config file can.
- Check Search Console's Coverage report monthly. A sudden jump in "Excluded" pages is usually the first sign something changed before you'd otherwise notice.
- Version-control your robots.txt if you manage it outside WordPress, so a bad edit is one revert away from being fixed.
None of these checks take more than a couple of minutes individually - the trick is just knowing there are four separate places to look instead of assuming it's one setting.
Frequently asked questions
Does deleting robots.txt fix a crawl block?
Not by itself. A missing robots.txt is treated as no restrictions, which helps if that file was the only problem, but it won't undo a noindex meta tag, a WordPress visibility setting, or an X-Robots-Tag header. Check all four layers, don't just delete the file.
How long does it take Google to re-index after I fix it?
After you fix the block and request indexing through Search Console, individual pages are often recrawled within a few hours to a couple of days. Full re-ranking to previous positions can take longer, from several days to a few weeks, depending on how long the block was in place.
Can a CDN or firewall block crawlers even if my server config is clean?
Yes. Cloudflare Page Rules, bot-fight modes, and some WAF configurations can inject an X-Robots-Tag header or outright block the Googlebot user-agent before the request ever reaches your server. Always run the curl -I check against your live domain, not just your origin server.
Is it bad to block /wp-admin/ in robots.txt?
No, that's the normal, recommended setup. WordPress's default robots.txt already disallows /wp-admin/ while allowing admin-ajax.php through it. That's not a crawl-block problem - it's expected and doesn't hurt your SEO.
Why does only part of my site show this problem?
A narrow Disallow rule (like Disallow: /blog/) or a plugin's category-specific noindex setting will only affect the URLs it matches. Check the robots.txt path patterns and any SEO plugin category or taxonomy settings if the block isn't site-wide.