Content-Signal in robots.txt: What It Does, and Whether to Use It

What the Content-Signal directive actually is

Every directive in a traditional robots.txt answers one question: may you fetch this? Allow and Disallow govern access and nothing else. Once a crawler has legitimately fetched a page, the protocol has no vocabulary for what happens next.

Content-Signal fills that gap. It is a machine-readable line, introduced by Cloudflare in September 2025 and based on the IETF AI preferences work, that states what a fetched page may be used for. It does not grant or revoke access — the Allow and Disallow lines still do all of that, unchanged.

The syntax is a directive inside a user-agent group:

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

There is also a path-scoped form, which takes a path before the category list:

User-agent: *
Content-Signal: search=yes, ai-train=no
Content-Signal: /docs/ ai-input=yes
Allow: /

Values are yes or no. Omitting a category entirely is meaningful and different from setting it to no: absence means no expressed preference, while no is an explicit refusal. That distinction is the whole reason the mechanism exists, and it matters legally more than it matters technically — see below.

The three categories, precisely

The categories are deliberately narrow so a compliant crawler can act on them without interpretation. Getting them confused is the main way sites set a preference they did not intend.

The line most sites get wrong is the boundary between search and ai-input. Setting search=yes, ai-input=no is a coherent position — index me and link to me, but do not use my text as raw material for an answer that replaces the click. It is also, in practice, a position that removes you from exactly the AI citation surface most businesses are currently trying to win. Decide it deliberately rather than inheriting it from a template.

Cloudflare has also been testing content-use, an optional extension expressing usage terms in more detail. Treat it as experimental and not something to build a policy on yet.

Content-Signal vs Disallow vs Content-Usage

Three mechanisms now overlap in the same file, and they are not interchangeable.

All three are requests, not controls. None of them stops anything at the network level. The only mechanism that actually prevents a fetch is a WAF rule, an IP block, or authentication — Cloudflare's own documentation is explicit on this point, recommending robots.txt to express preferences and AI Crawl Control to enforce them.

You can use Content-Signal and Content-Usage together in the same file; they do not conflict. Whether that's worth the maintenance burden is a separate question.

Does anything actually obey Content-Signal?

This is where you should be sceptical, and where most articles on the topic quietly aren't.

John Mueller of Google has stated that the Content Signals directive has, in his words, no effects whatsoever for any crawler or LLM — that it was created by a CDN, that using it adds bloat and future maintenance to robots.txt, and that crawlers simply use the directives they support and ignore the rest. That is a direct assessment from the search vendor with the largest crawler on the internet, and there is no public evidence contradicting it. No major AI vendor has published documentation committing to honour Content-Signal.

So on the technical question — will adding this line change what any crawler does today — the honest answer is: there is no evidence that it will.

Why it may still be worth adding

The argument for it is not technical. Cloudflare's framing is that robots.txt preferences can carry legal significance, and the policy text it ships states that restrictions expressed via content signals are express reservations of rights under Article 4 of the EU Copyright Directive 2019/790.

Article 4 permits text and data mining of lawfully accessible works unless the rightsholder has expressly reserved that right in a machine-readable form. Content-Signal is an attempt to be exactly that machine-readable reservation. Whether a court agrees is untested. But the asymmetry is worth noting: if you never expressed a reservation, you cannot later argue you did.

That gives a clean decision rule. If your content is a licensable asset and you may one day want to assert rights over its use in training, the two-line cost of an explicit reservation is rational insurance. If your content is marketing and you want maximum AI visibility, the directive buys you nothing and adds a file to maintain.

Cloudflare's managed robots.txt and the defaults changing on 15 September

If your site sits behind Cloudflare — roughly a fifth of all websites do — you may already be publishing Content-Signal directives you never wrote.

With the managed robots.txt feature enabled, Cloudflare prepends its policy comments and sets Content-signal: search=yes, ai-train=no on your behalf, together with Disallow rules for a list of AI crawler user-agents. Notably it does not set ai-input for you, on the grounds that it should not assume your preference on the most consequential category.

There is also a passive case: free zones with no robots.txt of their own and no managed robots.txt enabled will still have the Content Signals Policy served when a crawler requests the file. That version outlines the framework only and expresses no preference.

The change to actually diarise: Cloudflare has set 15 September 2026 as the date it applies new defaults across its three classifications. For domains newly onboarding, the Training and Agent categories are to be blocked by default on pages displaying ads, while Search remains allowed by default. Because these defaults are applied at the platform level, check your zone settings rather than assuming your current posture survives the change — and confirm the current state in the dashboard, since platform rollouts of this kind shift.

Practical audit step, whatever your CDN: fetch your own robots.txt from outside your network and read the whole file. Managed features, security plugins, and CMS defaults all write to it, and the version you remember writing is frequently not the version being served.

curl -sS https://example.com/robots.txt

If that returns a challenge page or a non-200 status rather than plain text, you have a more urgent problem than usage signals — the file itself isn't reaching crawlers. That case is covered in the piece on why AI doesn't cite your website.

Implementing it without breaking anything

Content-Signal is additive and low-risk, but there are three things worth knowing before you deploy it.

Google Search Console will report a syntax error

The robots.txt report may flag Syntax not understood for Content-Signal and other newer directives. This is expected — an unknown directive is ignored by design under the protocol. Cloudflare reports observing no impact on crawl rates or SEO from these warnings. Do not delete the directive to make the warning disappear; if you do want a clean report, that is a legitimate reason not to add it in the first place.

It belongs inside a user-agent group

Content-Signal is a group member directive, so place it under a User-agent line, not floating at the top of the file. And remember the grouping rule that governs the rest of robots.txt: a crawler obeys only its most specific matching group and does not inherit from User-agent: *. A Content-Signal line placed only in the wildcard group does not apply to any crawler that has its own named group elsewhere in your file. This is the same trap that causes protected paths to leak, covered in the guide to configuring robots.txt for AI crawlers.

A signal is not a block

If a crawler is already ignoring your Disallow lines, it will ignore your Content-Signal line too. Pair preferences with enforcement: WAF rules or bot management for crawlers you actually want stopped, and monitoring so you can see who is complying. Cloudflare surfaces robots.txt violations per crawler in its dashboard, which is the cleanest way to find out empirically who honours what on your own traffic.

So should you use it?

A position rather than a hedge, since the topic invites hedging.

• Publisher, archive owner, or anyone with licensable content: yes. Set ai-train=no explicitly. The technical effect today is likely zero; the value is holding a documented, machine-readable reservation of rights from a specific date.
• Business whose content exists to be found: set search=yes, ai-input=yes if you add it at all, and understand you are adding a statement of intent, not a mechanism. Do not set ai-input=no reflexively — that's the category that governs being cited in answers.
• Already on Cloudflare with managed robots.txt on: read the file today. You may be publishing ai-train=no plus a list of AI crawler disallows you never chose, and the platform defaults are scheduled to shift again.

The larger point is one of sequencing. Usage signals are the last layer of an agent-readiness stack, and they only matter once the layers beneath them work: the crawler can reach your server, robots.txt permits the fetch, and the page returns parseable server-rendered content. Sites tuning Content-Signal while returning 403s to ClaudeBot are optimising a preference nobody can read.

Getting that whole stack right — crawler access, robots.txt policy, usage signals, llms.txt, structured data, and an api-catalog endpoint — is the agent-readiness work in our SEO, AEO and GEO service, implemented directly on your platform. You can also start with a $20 audit and roadmap to see which of those layers your site is currently missing.