Why AI Doesn't Cite Your Website: Finding the Bot Blocking Behind It

What 'not being cited by AI' actually looks like from the server side

Most people diagnose this from the outside: they ask ChatGPT a question their page should answer, get a competitor's URL back, and conclude the content wasn't good enough. That's usually the wrong conclusion, and it sends you off rewriting copy for weeks.

Diagnose it from the inside instead. An AI answer engine can only cite a page whose text it actually received. If the crawler received a challenge page, a 403, or an HTTP 202 with a JavaScript puzzle, the model never saw your content — the quality of your writing is irrelevant to that outcome.

The tell is in your access logs, not your rankings. Look for these three patterns:

• The crawler is absent entirely. No GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot hits at all over 30 days. Something upstream is dropping them before they reach your application log.
• The crawler is present but every response is 403, 423, or 202. This is the classic anti-bot challenge signature. A 200-only filter on your log analysis will hide it completely.
• The crawler gets 200s, but only on the homepage. Rate limiting kicked in after the first few requests and everything deeper returned 429.

I ran into this from the opposite direction first. While building a research pipeline that fetched pages from client and competitor sites, I kept getting non-content responses from a subset of hosts. Fixing my own scraper taught me exactly what an AI crawler experiences on those same hosts — and it turned out the block had nothing to do with robots.txt.

The four layers where an AI crawler gets stopped

Requests die at different layers, and each layer requires a different fix. Working top-down saves you from editing a robots.txt file that was never the problem.

Note the ordering consequence: robots.txt is the last layer, and it is the only one most SEO guides talk about. A perfect robots.txt behind an anti-bot layer that returns 202 to non-browser clients accomplishes nothing, because the crawler never gets far enough to read it.

SiteGround sgcaptcha: the HTTP 202 that quietly de-indexes pages

SiteGround runs an in-house bot prevention system. SiteGround's own description of the Anti-Bot AI is that once the system flags an IP address or user agent as malicious, that client is challenged with a captcha page, and a human who solves it gets whitelisted.

The mechanics matter more than the description. When the system flags a request, the server does not return your page and does not return a clean error. It returns HTTP 202 Accepted, with an sg-captcha: challenge header and a redirect toward /.well-known/sgcaptcha/. A browser follows that redirect, executes the JavaScript proof-of-work, gets a cookie, and lands on the real page. Every non-browser client — including every AI crawler and every SEO audit tool — receives the challenge page and stops there.

Two things make this worse than an ordinary block.

The 202 is a success code, so your monitoring stays green

Uptime monitors and log dashboards treat 2xx as fine. You will not see an error spike. The site looks healthy in every dashboard you own while returning nothing usable to automated clients. If your uptime tool reports the site up and your crawl-based audit tool reports zero pages, that gap is the diagnosis.

The challenge response can carry a noindex header

This is the part that turns a crawl problem into a ranking problem. Security researchers at Sprocket Security documented that the challenge response carries an X-Robots-Tag: noindex header, and that when Googlebot triggers the captcha it receives that header too. Their write-up on bypassing JavaScript proof-of-work captchas cites a site that fell from 270 indexed pages to 62 over three months before anyone identified the cause, and concludes that the fix is to ask SiteGround support to disable the anti-bot system for the account.

They also tested the whitelisting claim and found it weaker than advertised: after solving the captcha from a flagged IP, requests carrying the session cookie returned 200s, but requests from the same IP without the cookie still returned 202s. The whitelist is effectively per-session, not per-IP.

There is a second-order trap here too. If you sit Cloudflare in front of SiteGround, Cloudflare's request routing can itself trip SiteGround's detection, and you end up chasing a redirect loop across two vendors who each point at the other.

What actually resolves it

1. Open a ticket with SiteGround and ask them to disable the Anti-Bot AI for your site, or to whitelist the crawler user-agents and IP ranges you care about. You cannot configure this yourself from Site Tools.
2. Re-test with a plain HTTP client (below) and confirm you get a 200 with real HTML, not a 202.
3. Check Search Console coverage for pages that dropped out during the affected window and request re-indexing.

From the automation side, I confirmed the same thing the hard way. Updating curl_cffi in my pipeline fixed TLS fingerprint rejection on several hosts — that's a genuinely useful fix for JA3-based filtering — but it did nothing at all for sgcaptcha, because the challenge is JavaScript, not fingerprinting. The only two clients that get through are a real browser engine (Playwright, Puppeteer) or a whitelisted IP. AI crawlers are neither, unless the host has explicitly allowed them.

Imunify360 WebShield: the 403 that has nothing to do with your robots.txt

Imunify360 is standard on a large share of cPanel and CloudLinux shared hosting. Its WebShield component sits in front of Apache or LiteSpeed and decides per request whether a visitor passes, gets a captcha, or is dropped. The default behaviour for a client that cannot execute JavaScript is a 403 or 423.

The important detail — and the one that wastes the most engineering time — is that changing the user-agent string does not help. WebShield's primary filter is the JavaScript challenge, not static UA matching. Send a perfect Chrome user-agent from a Python client and you still fail, because you still cannot run the challenge script.

For declared good bots, Imunify360's logic is a chain of checks rather than a single list. Per CloudLinux's documentation on allowing crawler bots from unknown IPs, an IP being in the ipset whitelist is not sufficient: the request must also carry the expected crawler user-agent, match an additional allowed-crawler IP list, and clear an RBL lookup against the good-bots zone. Fail any link and you get the incident log line IM360 WAF: Found crawler not in whitelist and a 403.

That chain is built around the crawlers Imunify360 knows: Googlebot, Bingbot, Yandex. Newer AI crawlers are not automatically in it. This is the structural reason AI bots get blocked on shared hosting far more often than Googlebot does — the allowlist was written for a search landscape that predates them.

The misconfiguration that blocks everything

If your server sits behind a CDN and the web server is not resolving the real client IP correctly — no remoteip module, or it isn't configured for Cloudflare — then every request appears to originate from the CDN edge IP. Reverse-DNS verification of a declared crawler then fails by definition, and ModSecurity blocks legitimate traffic wholesale. CloudLinux documents this wrong-IP-detection failure mode explicitly. Check this before you touch any rule.

Fixing it

On managed shared hosting, this is a support ticket: ask for the AI crawler user-agents and their published IP ranges to be added to the WebShield whitelist. With root access you can add a custom ModSecurity rule that bypasses the bot protection for verified crawler IPs — place it in a file outside the Imunify360 rules directory so a ruleset update doesn't overwrite it. Do not disable the bot protection rule wholesale; you will trade an AI visibility problem for a brute-force problem.

How to test whether ChatGPT, Claude, and Perplexity can actually reach your site

Do this before changing anything. It takes two minutes and it tells you which layer you are fighting.

Fetch your own page with a declared AI crawler user-agent and inspect the headers, not just the body:

curl -sSI -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' https://example.com/your-page/

curl -sSI -A 'Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])' https://example.com/your-page/

curl -sSI -A 'Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' https://example.com/your-page/

Read the result against this table:

Two follow-ups worth running. Compare the byte size of the crawler-UA response against a browser-UA response — a large gap means you are serving different content to bots, which is its own problem. And run the same test against a deep page, not just the homepage, because rate limits and path-scoped rules only show up past the first request.

The order to fix things in

Sequence matters because each step invalidates the measurement of the one after it. Confirming access before optimising content is not a preference, it's the only order that produces interpretable results.

1. Unblock the transport layer. CDN rules, then hosting anti-bot, then server rules. Verify with curl after each change.
2. Get robots.txt right. Vendors now run three separate user-agents each, and blocking the wrong one removes you from AI search while leaving training crawlers untouched. The specifics are in the companion piece on configuring robots.txt for AI crawlers.
3. State your usage preferences. Once crawlers can reach you, you can express what they may do with the content — that's what the Content-Signal directive in robots.txt is for, and it is a separate decision from access.
4. Make the content retrievable in structure, not just in bytes. Server-rendered HTML, one clear answer per section, semantic markup, stable heading anchors.
5. Measure again after 30 days. Crawl frequency changes take weeks to show up in citations; robots.txt changes alone can take about a day to be reflected.

What to do once the crawlers can actually get in

Unblocking restores eligibility. It does not by itself produce citations. The pages that get quoted are the ones that are cheap for a retrieval system to parse and easy to attribute.

On a client ecommerce build I work on, the agent-readiness layer we shipped after fixing access looked like this:

• An llms.txt file summarising what the site is and linking the highest-value entry points. If you don't have one, our free llms.txt generator will produce a first draft from your sitemap.
• Markdown content negotiation on product and article detail views, so a client sending Accept: text/markdown gets clean markdown instead of a template full of navigation chrome.
• A robots.txt with explicit AI crawler rules plus Content-Signal directives, so preferences are stated rather than inferred.
• A /.well-known/api-catalog Linkset endpoint pointing agents at the machine-readable interfaces, with all actual API access behind token auth.

The principle underneath all four: reduce the work an automated client has to do to extract a correct, attributable answer from your page. That is the same principle behind every technical fix above — a crawler that spends its budget on challenges, redirects, and template noise has none left for your content.

If you'd rather not run the diagnostic yourself, that's exactly what our $20 audit and roadmap covers: we check how Google and AI crawlers actually see your site, identify which layer is blocking them, and send back a prioritised fix list within 48 hours — yours to act on or hand to a developer. If you'd sooner talk it through first, the free AI-visibility consultation costs nothing and needs no account.