AI Crawlers Overloading Your Server? How to Block Them

Your bandwidth graph is spiking, your server load is up, and Google Analytics shows nothing unusual. Check your raw access logs instead of your analytics dashboard, and you'll usually find the real visitor: not a person, but an AI crawler indexing your site for a chatbot's training data or its live search answers.

Symptom: Traffic That Doesn't Match Your Analytics

This one trips people up because it hides from the usual tools. Google Analytics and most WordPress stats plugins run JavaScript in the visitor's browser, and bots almost never execute JavaScript. So your "real" traffic numbers look flat while your server-level metrics tell a different story:

• cPanel's bandwidth usage or AWStats shows a jump that analytics never recorded
• Your VPS load average climbs during odd hours with no matching human traffic
• cpanel-dcv-check or your hosting dashboard flags unusually high outbound data transfer
• Server response times get slow for real visitors during a crawl burst

Pull up your raw access log and you'll usually spot the pattern within a minute:

tail -5000 /home/username/access-logs/yourdomain.com | grep -Ei "GPTBot|ClaudeBot|Bytespider|CCBot|Amazonbot|PerplexityBot" | wc -l

If that number is in the thousands for a single day on a mid-size site, you've found your traffic spike.

Cause: Meet the AI Crawlers

Over the last couple of years, every major AI company has shipped its own crawler to pull content for model training or for live retrieval when their chatbot answers a question. Unlike Googlebot, which has spent 25 years learning to crawl politely and cache aggressively, several of these bots are new, poorly rate-limited, and in a few cases openly indifferent to robots.txt.

Most of them are well-behaved if you tell them what to skip. The trouble is that a lot of sites have never bothered to say anything, so every one of these crawlers happily walks the entire catalogue, every archive page, every tag and filter combination WooCommerce or a badly-paginated blog can generate. On a shared cPanel account with CloudLinux LVE limits, that's exactly the kind of sustained load that trips a "Resource Limit Is Reached" error for completely innocent reasons.

Fix 1: Set Boundaries in robots.txt First

This is the polite, low-effort option, and it works for every bot in the table above except Bytespider, which has a documented history of ignoring it. Edit (or create) robots.txt in your site's document root:

User-agent: GPTBot
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /*?add-to-cart=
Crawl-delay: 10

User-agent: ClaudeBot
Disallow: /wp-admin/
Crawl-delay: 10

User-agent: CCBot
Disallow: /wp-admin/
Crawl-delay: 10

User-agent: PerplexityBot
Disallow: /wp-admin/
Crawl-delay: 10

Note the Crawl-delay directive: it won't stop a bot, but a well-behaved one will space its requests out and stop hammering your server all at once. If you want to block a bot entirely rather than just steer it, use Disallow: / under its user-agent block. Save the file, then confirm it's actually reachable:

curl -sI https://yourdomain.com/robots.txt

You should get a 200 OK, not a redirect or a 404 - a broken robots.txt is worse than none, because some crawlers treat a failed fetch as "no restrictions apply."

Fix 2: Hard-Block With .htaccess When robots.txt Isn't Enough

For bots that ignore the polite request - Bytespider is the usual offender, along with generic scrapers spoofing a browser user-agent - you need a server-side block. On Apache/LiteSpeed (which covers virtually every cPanel account), add this near the top of your site's .htaccess, before the WordPress rewrite rules:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider|CCBot|Amazonbot|PerplexityBot|Google-Extended|Meta-ExternalAgent) [NC]
RewriteRule .* - [F,L]

The [F,L] flags return a 403 Forbidden and stop processing further rules for that request. Test with a spoofed user-agent before trusting it:

curl -A "GPTBot" -I https://yourdomain.com/

You should see 403 Forbidden. If you instead see a normal 200, check that the block sits above any existing RewriteRule ^index\.php$ line - WordPress's own rewrite rules will otherwise catch the request first.

Fix 3: Push the Block to Cloudflare (Cleaner, and Saves Origin Bandwidth)

If your domain is proxied through Cloudflare, blocking at the edge is more efficient - the request never reaches your cPanel or VPS server at all, so it doesn't count against bandwidth or CloudLinux LVE limits. Cloudflare's dashboard has a purpose-built option for this:

1. Go to Security → Bots in your Cloudflare dashboard
2. Under AI Scrapers and Crawlers, toggle blocking on (available on free plans as of 2025-2026)
3. For anything not covered by that toggle, add a custom WAF rule: field User Agent contains Bytespider, action Block

This is the option we point most Getwebup clients toward when a single bot is repeatedly triggering "under attack" style slowdowns, because it's a two-minute change with no risk of a stray character breaking your .htaccess.

Should You Block Everything?

Not necessarily, and this is the part worth pausing on before you copy-paste a block-everything rule. GPTBot and PerplexityBot crawling your site is also how your pages end up cited as sources in ChatGPT and Perplexity answers - a growing referral channel some site owners actively want. A reasonable middle ground:

• Allow GPTBot, ClaudeBot, and PerplexityBot if AI-driven referral traffic matters to your business, but add Crawl-delay and disallow heavy pages (search results, cart, filtered archives)
• Block Bytespider and any crawler you can't identify by a clean, documented user-agent string
• Rate-limit rather than block outright if you're on a VPS with headroom to spare - a WAF rule capping requests per IP per minute is gentler than a flat 403

Prevention: Watch This Going Forward

AI crawlers are still a moving target - new ones show up every few months. Two habits keep this from becoming a recurring fire drill:

• Check your access logs for new, unfamiliar user-agents monthly: awk -F'"' '{print $6}' access-logs/yourdomain.com | sort | uniq -c | sort -rn | head -30
• If you're on Getwebup VPS hosting, set a simple cron alert on bandwidth usage so a crawl spike gets flagged before it turns into an overage or a CloudLinux LVE throttle

A robots.txt file costs nothing to maintain and stops the well-behaved majority. The .htaccess or Cloudflare block is your backstop for the minority that doesn't play by the rules - keep both in place and you won't need to think about this again until the next bot shows up.