Back to blog
Web Scraping·October 7, 2026·15 min read

Cloudflare Now Blocks AI Agents by Default: What Builders Need to Know

Cloudflare now blocks AI agent traffic by default on ad-supported pages for new domains. See the traffic data driving the change, the Perplexity stealth-crawling case, the IETF standard forming around it, and how to tell a real block from a misclassification before building against a domain.

M

Mahendra Sreekumar

Anakin Team

Cloudflare's new default blocking AI Agent and Training bots behind a wall on ad-supported pages, while Search crawlers pass through, effective September 15, 2026

A RAG pipeline or research agent that fetched a page cleanly last month can now come back with nothing, on the same domain, with the same code. Often that isn't a bug. Since September 15, 2026, new ad-supported domains on Cloudflare start with a recommended setting that blocks Agent traffic, meaning automated fetches made for a person in real time, on pages that display ads. Search crawlers stay allowed. The change landed in the middle of the fastest growth in AI bot traffic the web has measured.

So the short answer for builders: check a domain's declared preference before writing fetch logic against it, and treat a block that only hits ad pages as policy, not as a rate limit. Below is exactly what changed and for whom, why Cloudflare made it the default, how the wider industry (IETF, Anthropic, OpenAI, and Cloudflare's own Pay Per Crawl program) is converging on the same idea, what a block looks like from an agent's side, and how to tell a deliberate opt-out from a misclassification.

Anyone pulling pages through Anakin's URL Scraper, or any comparable fetch layer, hits this exact wall first. The diagnosis below is written from that seat.

What changed, and when

Cloudflare's own documentation set out the plan in July: for new domains, bots classified as Training or as Agent would be blocked on pages that display ads, and Search would stay allowed, starting September 15, 2026. What shipped that day changed the Training half. Cloudflare's September 15 post lists the preset for ad-supported domains as Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads. The Agent block on ad pages, the part that hits RAG pipelines and research agents, went ahead as planned.

  • New domains onboarding to Cloudflare are offered these presets from day one, and the owner can change them during onboarding or at any time after.
  • New domains added by existing customers get these defaults, even if the account is years old.
  • Existing Free plan domains are less clear-cut. Cloudflare's July announcement said existing Free customers who hadn't changed their settings by September 15 would move to the new defaults too, but the September 15 post says existing domains carry their current settings over. Check the zone's settings rather than assume either way.
  • Existing paid customers who already had a domain live before September 15 keep whatever they had configured. Nothing changed under them without action.

That last point matters more than the headlines suggest. Cloudflare isn't blocking agents across the entire web overnight. It's a default for new and newly onboarded properties, plus whatever the Free plan migration did (see the bullet above).

Why Cloudflare moved: the traffic numbers behind the policy

Cloudflare didn't pick this default out of nowhere. It's answering a measurable shift in what hits origin servers.

  • Imperva's 2026 Bad Bot Report found automated traffic accounted for more than 53% of all web traffic in 2025, up from 51% the year before, with human activity continuing to decline. The report frames AI agents as a distinct new category: systems that retrieve data, execute workflows, and act on a user's behalf rather than just scanning pages.
  • Fastly's network data shows AI requests grew roughly 30% between January and May 2026, about 6.5x faster than human traffic in the same window, with more than 51% of AI requests requiring a trip back to origin infrastructure versus under 9% for human requests. AI traffic hits the expensive part of the stack far more often per request.

An agent that cannot be cached the way a browsing human can, multiplied across a 6.5x growth curve, is an infrastructure cost problem before it is a content-licensing problem. The default targets the traffic pattern driving that cost. A first fetch through Anakin's URL Scraper fits that profile too. Repeat requests for the same URL within 24 hours are served from Anakin's cache at zero credits and never reach the origin.

Bar stat comparison: 53 percent of all web traffic is automated per Imperva's 2026 Bad Bot Report, and AI traffic growing 6.5x faster than human traffic per Fastly's 2026 network data

The three categories Cloudflare actually checks

Cloudflare's bot classification splits automated traffic into named behaviors, and the new default only touches two of them.

  • Search - crawlers that index content to answer questions later, the kind that sends referral traffic back. Allowed by default.
  • Agent - automated activity acting in real time on a person's behalf: chat-fetch bots, browser-use agents, anything retrieving a page to answer one specific request right now. Blocked by default on ad pages.
  • Training - crawlers harvesting content to train or fine-tune a model. Set to Disallow AI Training in the ad-supported preset: training-only crawlers are blocked, while mixed-use crawlers Cloudflare rates as Accountable stay allowed for search.

A RAG pipeline that fetches a page at query time to ground an answer, whether through Anakin's URL Scraper or a hand-rolled fetch, is Agent traffic by Cloudflare's own definition, not Search. That is the category this change targets.

Why ad-supported pages specifically

Cloudflare's stated logic: an ad on a page is evidence that page was built for a person to look at it. An agent that fetches the page, reads the price or the review, and answers a user who never loads the page themselves never sees that ad. Search crawlers get a pass because they send a visitor back eventually. Agent and Training traffic, by Cloudflare's framing, consume the content without returning anything the ad model depends on.

Pages with no ads on the same domain are not covered by this default at all. A news site's article pages might block agents while its documentation pages do not.

Diagram of Search, Agent, and Training bot request lanes hitting a web page with an ad unit, Search passing through while Agent and Training traffic are blocked by Cloudflare's default

The catch for mixed-purpose crawlers

Googlebot, Bingbot, and Applebot each crawl for Search and for their own AI training under a single bot identity. Cloudflare's September 15 update handles them with a new Training setting, Disallow AI Training, which is what the ad-supported preset uses. It publishes a no-training preference in robots.txt, blocks training-only crawlers such as OpenAI's and Anthropic's, and lets these three keep crawling for search. The catch moved to the stricter settings. Since September 15, choosing Block or Block on pages with ads for Training also stops Googlebot, Bingbot, and Applebot, search included. A site owner who tightens the default by hand can end up suppressing search indexing it never meant to touch, on exactly the pages that monetize the site.

The real compliance problem robots.txt alone can't fix

A 2025 study out of Duke University ran a controlled, large-scale test of 130 self-declared bots over 40 days and found bots grow less compliant as robots.txt directives get stricter. The researchers' conclusion is blunt: certain categories of bots, including AI search crawlers, rarely check robots.txt at all. Relying on a voluntary text file as the only control was already shaky before this policy existed.

The clearest real-world example of that risk played out between Cloudflare and Perplexity in 2025.

Cloudflare's own investigation found that "Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring, or sometimes failing to even fetch, robots.txt files." On test domains Cloudflare built specifically to be unreachable by any declared crawler, Perplexity's assistant still returned answers describing the page content. Cloudflare responded by de-listing Perplexity as a verified bot and adding detection heuristics for the stealth pattern. Perplexity disputed the findings publicly.

Whichever side of that specific dispute turns out to be right, the incident is why Cloudflare now treats a declared, honest user agent as the baseline for a verified bot, and why Anthropic and OpenAI each publish named crawlers with documented robots.txt behavior.

The wider shift: standardizing how sites say yes or no

Cloudflare's Pay Per Crawl program repurposes the long-unused HTTP 402 Payment Required status code: a site owner sets a per-crawl price, and a crawler that doesn't pay gets a 402 instead of the page. Site owners choose, per crawler, to allow for free, charge, or block outright. Cloudflare's post still describes it as a private beta.

Cloudflare's Content Signals Policy works one layer up, as a declared-preference extension to robots.txt itself:

curl https://example.com/robots.txt

# User-Agent: *
# Content-Signal: search=yes, ai-train=no
# Allow: /
  • search - permission to index and return excerpts in search results.
  • ai-input - permission to use the content as live input for a generated answer, the exact thing a RAG fetch or agent fetch does.
  • ai-train - permission to use the content in model training or fine-tuning.

Cloudflare's own robots.txt is a live example of the format: checked on 26 Oct 2026, it carried the line Content-Signal: ai-train=yes, search=yes, ai-input=yes, so it allows all three uses.

Cloudflare's managed robots.txt set search=yes, ai-train=no across more than 3.8 million domains as of September 2025, and on September 15, 2026 Cloudflare began moving it to Bot Preference Sync, which publishes the no-training preference for sites that choose Disallow AI Training. None of this is Cloudflare acting alone, either.

The IETF's AI Preferences working group (AIPREF) is drafting a standard vocabulary for this kind of signal, with a Content-Usage HTTP header and a matching robots.txt directive, meant to work the same way across every site and every crawler rather than per-vendor. That standardization effort is the direction this is all heading: fewer one-off vendor rules, one shared way for a site to state what it allows.

Know the crawler before building against the domain

Each major AI company runs separate, named bots for separate jobs, documented directly by the companies that operate them.

OperatorBot nameJobRespects robots.txt
OpenAIGPTBotCrawls content for model trainingYes
OpenAIOAI-SearchBotSurfaces sites in ChatGPT searchYes
OpenAIChatGPT-UserFetches a page a user asked about, liveNot necessarily - user-initiated
AnthropicClaudeBotCrawls content for model trainingYes
AnthropicClaude-UserFetches a specific page a user asked Claude aboutYes
AnthropicClaude-SearchBotImproves Claude search result qualityYes
PerplexityPerplexityBot / Perplexity-UserDeclared crawler and user-fetch agentDisputed - see Cloudflare's findings above
GoogleGooglebot / Google-ExtendedSearch indexing / Gemini trainingYes

A site that disallows ClaudeBot but not Claude-User, for instance, is drawing the same Agent-versus-Training line Cloudflare's own default enforces. Checking which named bot a request will actually present as, before assuming a domain-wide block, is worth doing before writing retry logic that treats every one of these the same.

Anakin's URL Scraper identifies itself with a declared, checkable user agent, so a site owner can allow or block it by name like any bot in that table. A declared identity doesn't move a request into the Search category, though: under Cloudflare's definitions, a fetch made for one user right now is still Agent traffic.

Table of named AI crawlers including OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, Google's Googlebot, and Anakin's URL Scraper, with their role and robots.txt compliance status

What this looks like from the agent's side

Not a clean 403 on every request to a domain. It shows up as inconsistent: a product page with ad units returns a block or a challenge, while a docs page with no monetization on the same domain returns content normally. A scraper tuned to treat any non-200 as a retry-and-move-on case will often get this wrong, since it looks like a transient rate limit rather than a deliberate policy.

The fastest way to confirm it is the Agent-block default, not a rate limit or a one-off fingerprint flag, is to check the same URL from a plain Search-style request pattern versus a request that behaves like a live fetch-and-answer agent. If only the second consistently fails on ad-carrying pages, this is the default at work.

On Anakin's URL Scraper, that pattern shows up as a blocked or challenge response on ad-carrying pages specifically, with the same domain's non-ad pages returning clean markdown, no code change required to tell the two apart.

Check the site's declared preference before building against it

Content Signals in robots.txt are voluntary on the crawler's side. Nothing stops a request at the network layer, and the Duke study above is a direct warning about trusting that file alone. But checking it first is still a five-second request that tells a builder whether a block seen later is policy or something else entirely, like a plain rate limit or fingerprint mismatch, before any pipeline gets built against a domain that has already said no. For a domain a pipeline depends on, Anakin's Website Monitoring API can watch that robots.txt URL on a schedule and fire a webhook when it changes, so a new ai-input=no line shows up as an alert instead of a failed run.

Where Anakin's URL Scraper fits

None of this changes what Anakin's URL Scraper is built for: turning a URL into clean markdown, HTML, or structured JSON, for data access a team is authorized to pull. The more common problem most pipelines hit isn't a deliberate block. It's a plain HTTP client getting misclassified as automated traffic on a domain that has not restricted agent access at all, which looks identical to a real block until someone tests it.

URL Scraper's useBrowser: true parameter switches a request from a raw HTTP fetch to a full headless browser render, with fingerprint, locale, and timezone signals aligned to a real session, at no extra credit cost over the base scrape. That's the difference between a request that looks like a script and one that looks like a browsing session with a real execution context, and it's the distinction most generic bot detection keys on.

curl -X POST https://api.anakin.io/v1/url-scraper/scrape \
  --max-time 120 \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product-page",
    "formats": ["markdown"]
  }' 

A run of that request on 25 Oct 2026 against Cloudflare's own Block AI Bots documentation page returned the page as markdown, starting with the heading "# Block AI Bots", in about 1.6 seconds.

If that comes back blocked specifically on ad-carrying pages while docs pages on the same domain succeed, switch to a full browser render:

curl -X POST https://api.anakin.io/v1/url-scraper/scrape \
  --max-time 120 \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product-page",
    "useBrowser": true,
    "formats": ["markdown"]
  }' 
Toggle switch illustration showing Anakin URL Scraper's useBrowser parameter flipping a request outcome from flagged as bot to real session

It isn't a workaround for a site that has explicitly declared ai-input=no in its Content Signals, or disallowed the relevant bot by name, and shouldn't be treated as one. It's the right tool for the mismatch case: a domain that has not restricted agent access but flags a plain HTTP client as a bot anyway, which is the case most RAG pipelines and research agents hit day to day, long before they ever encounter a site that has deliberately opted out.

For a pipeline that needs the rank-and-describe step before it knows which URLs are even worth fetching, Anakin's Search API sits in front of URL Scraper for that job, and Anakin's Browser API is available when a pipeline needs direct Playwright or Puppeteer control over the same stealth browser infrastructure instead of the managed URL Scraper endpoint. URL Scraper remains the starting point for the most common job: turning one URL into clean content a pipeline can use.

What doesn't change

  • Search crawling is unaffected by this default. A domain's SEO visibility does not drop because of it, on its own.
  • Existing paid Cloudflare customers with domains already live before September 15 are not auto-switched. The new defaults apply to new domains and new customers; for unchanged Free plan domains, Cloudflare's own posts differ (see the first section).
  • A site owner can always override the default in either direction, in Security settings, whenever they choose.
  • Content Signals in robots.txt are a declared preference, not a technical barrier, and the Duke study and the Perplexity incident both show declared preferences get ignored in practice more than site owners assume.

This sits next to the wider Cloudflare anti-bot picture. A request failing for TLS fingerprint, IP reputation, or browser-rendering reasons unrelated to this specific default needs a different fix: see How to Fix Cloudflare Blocking Your Automation System and 6 Ways to Reduce Browser Fingerprint Detection. For the search layer above this one, Google Custom Search API Shutdown: What to Use Instead covers the migration path.

What to check before the next fetch

Cloudflare drew a real line between Search traffic and Agent traffic, and that line now defaults to closed on ad pages for new ad-supported domains, against a backdrop of AI traffic growing 6.5x faster than human traffic and bot traffic crossing 53% of the web. For anyone building agents or RAG pipelines, the practical move is to check a target domain's robots.txt Content Signals before writing a single line of fetch logic against it, know which named crawler the pipeline will actually present as, and reach for a real browser-rendering path when the problem turns out to be a mismatch rather than a deliberate opt-out.

Start with Anakin's URL Scraper and test against the domains that actually matter to the pipeline before assuming either way.

Frequently asked questions

When exactly did Cloudflare start blocking AI agents by default?

September 15, 2026. It applies to new domains onboarding to Cloudflare, including new sites added by existing customers. Cloudflare's July announcement also covered existing Free plan customers who hadn't changed their settings, while its September 15 post says existing domains carry their settings over, so check the zone's settings.

Does this affect every website on Cloudflare?

No. It changes the default for new domains, and possibly for unchanged Free plan domains (Cloudflare's July and September posts differ on that). Existing paid customers with domains already live before September 15 keep their prior configuration unless they change it. The Agent block also only applies to pages that display ads.

Will an agent get blocked on every page of an affected domain?

No. The block applies specifically to pages that carry ads. A documentation page or a login page on the same domain with no ad units is not covered by this default.

How can you tell if a specific block is this policy and not a rate limit or fingerprint issue?

Check the domain's robots.txt for a Content-Signal line. If ai-input is set to no, or the request fails consistently on ad-carrying pages specifically while succeeding elsewhere on the same domain, this default is the likely cause rather than a generic anti-bot trigger.

Does Googlebot get blocked too?

Not under the default. The ad-supported preset sets Training to Disallow AI Training, which lets Googlebot, Bingbot, and Applebot keep crawling for search while training-only crawlers are blocked. Since September 15, a site owner who picks Block or Block on pages with ads for Training does stop those three crawlers, search included.

Is robots.txt actually enough to keep an unwanted AI crawler out?

Not reliably. A 2025 Duke University study of 130 self-declared bots over 40 days found AI search crawlers in particular rarely check robots.txt at all, and Cloudflare's own investigation into Perplexity documented a named crawler rotating user agents and IPs specifically to get around declared no-crawl rules on test domains. Treat robots.txt as a stated preference, not an enforcement mechanism.

What is Cloudflare's Pay Per Crawl, and is it related to this default?

Pay Per Crawl is a separate but related Cloudflare feature that lets a site charge AI crawlers per request using the HTTP 402 status code, with allow, charge, or block set per crawler, and is still in private beta. It runs alongside the Agent/Training default covered here, as part of the same broader shift toward giving site owners explicit, enforceable control over AI access instead of relying on robots.txt alone.

Is there a legitimate way to keep accessing a site that has blocked Agent traffic?

Respect the site's declared Content Signals and named-bot rules first. If ai-input is set to no, that is the site owner's stated position, not a technical obstacle to route around. For domains that have not restricted agent access but misclassify a plain HTTP client as a bot, a real browser-rendering request through a tool like Anakin's URL Scraper resolves that specific mismatch.

Sources