Should You Serve Markdown to AI Agents? Only 3 of 7 Ask for It

What Accept: text/markdown actually does

Content negotiation is not new. It has been part of HTTP since 1.1, and it is how browsers have long asked for WebP over JPEG or French over English. The client states a preference in the Accept request header; the server picks a representation and answers.

What changed is that agents started using it for document format:

curl https://example.com/docs/getting-started \
  -H 'Accept: text/markdown'

One URL, two representations. A browser sends Accept: text/html,... and gets your page. An agent sends text/markdown and gets the same content with the markup stripped to headings, lists, links and prose. Nothing is cloaked, nothing is duplicated at a second URL, and clients that do not know about it are unaffected because the fallback is your normal HTML.

Two things pushed this from idea to infrastructure in early 2026. Cloudflare shipped Markdown for Agents on 12 February 2026, a zone-level toggle that converts HTML to Markdown at the edge on the fly. Vercel documented the same pattern natively for Next.js. Neither requires you to author content twice.

Which AI agents actually send the header?

This is the question the vendor announcements skipped, and the answer decides whether the work is worth doing. Checkly tested seven common agents in February 2026 by pointing each one's native fetch tool at an endpoint that echoes request headers. The result:

Three out of seven. That is the honest state of play, and it shapes the decision: this is a coding-agent feature today, not an AI-search feature. Cloudflare's own announcement names the same two — Claude Code and OpenCode — as the agents it already sees sending the header.

Two details in that table matter when you implement:

• Claude Code expresses preference by order, not by q-value. It sends Markdown and HTML at the same implicit weight of 1.0. A naive parser that requires Markdown to outrank HTML will serve Claude Code the HTML it did not want.
• */* is not a Markdown request. It means give me anything. Treating a wildcard as consent to send Markdown will break plain HTTP clients, monitoring probes and anything else that shrugs at content types.

How much does Markdown actually save?

The savings are not marginal, and three organisations measured them independently on their own content:

The spread between 80% and 99.7% is not measurement error — it is the difference between a lean content page and a documentation page carrying a large navigation tree. The heavier your chrome, the bigger the win.

Why it compounds: a heading like ## About Us costs roughly three tokens. Its HTML equivalent with a class and an id costs twelve to fifteen, before any wrapper divs. Multiply across a page and most of what you send an agent is structure it will throw away.

There is a second-order effect worth naming. Agents that receive HTML do not give up — they convert it themselves. Claude Code, by public accounts, runs fetched pages through a conversion step and a smaller summarising model before injecting the result into the conversation. Every one of those steps is a place your content can be truncated or paraphrased before anyone reads it. Serving Markdown removes the lossy middle stage. That, more than the token count, is the argument.

Does Google care whether you serve Markdown?

No, and it says so directly. The Search Central AI optimization guide lists llms.txt files and Markdown together under the tactics you can ignore for Google Search: you do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI features, because Search does not use them.

So the payoff here is not rankings, AI Overviews or AI Mode. It is the quality of what a coding agent walks away with when it reads your documentation. Keep those two ledgers separate — most of the confusion in this space comes from merging them. If you want the full picture on the search side, we covered whether llms.txt helps SEO and what Google actually says about the whole category.

Three ways to ship it, in increasing order of effort

1. Let your CDN do it

If you are on Cloudflare, this is a toggle. Enable Markdown for Agents on the zone; the network fetches your HTML from origin and converts it in flight. It is in Beta at no cost for Pro, Business and Enterprise plans plus SSL for SaaS. Converted responses carry an x-markdown-tokens header with an estimated token count, which is useful if you are the one building the agent.

2. Use your framework's middleware

Next.js sites can wrap the request in middleware that negotiates the header and strips non-content elements. This is the right call when your pages are already component-structured and a converter can find the main content reliably.

3. Render it yourself

Worth the effort when your content already exists in a structured form — a CMS field, a product record, a docs source file — because then you are not converting HTML back into text you already had. This is the route we took on a client eCommerce build, where product and article bodies live in the database and the HTML was always the derived artifact, not the source.

The negotiation itself is short. The part people get wrong is the header parsing, so here is a version tested against all seven agent strings above:

MARKDOWN_TYPES = ('text/markdown', 'text/x-markdown')


def _accept_weights(header):
    weights = {}
    for part in header.split(','):
        bits = [b.strip() for b in part.split(';') if b.strip()]
        if not bits:
            continue
        media = bits[0].lower()
        q = 1.0
        for param in bits[1:]:
            if param.lower().startswith('q='):
                try:
                    q = float(param[2:])
                except ValueError:
                    pass
        weights[media] = max(weights.get(media, 0.0), q)
    return weights


def wants_markdown(request):
    weights = _accept_weights(request.META.get('HTTP_ACCEPT', ''))
    markdown = max((weights.get(t, 0.0) for t in MARKDOWN_TYPES), default=0.0)
    html = max(weights.get('text/html', 0.0), weights.get('*/*', 0.0))
    return markdown > 0.0 and markdown >= html

The >= is deliberate, not sloppy. It is what makes Claude Code's order-based, all-equal-weight header resolve to Markdown. Requiring a strict > would silently exclude the single most active agent in the category.

Wiring it into a Django class-based view is then a mixin over render_to_response:

from django.http import HttpResponse
from django.utils.cache import patch_vary_headers


class AgentMarkdownMixin:
    '''Serve a Markdown representation when an agent asks for one.'''

    def get_markdown(self, context):
        raise NotImplementedError

    def render_to_response(self, context, **kwargs):
        if wants_markdown(self.request):
            response = HttpResponse(
                self.get_markdown(context),
                content_type='text/markdown; charset=utf-8',
            )
        else:
            response = super().render_to_response(context, **kwargs)
        patch_vary_headers(response, ('Accept',))
        return response

Use patch_vary_headers rather than assigning response['Vary'] directly — Django and its middleware set Vary elsewhere (Cookie, Accept-Encoding), and a straight assignment silently drops them.

Four things that break it in production

1. A missing Vary: Accept header. This is the one that causes real damage. Without it, any cache between you and the client — your CDN, a reverse proxy, a corporate proxy — may store the Markdown response under the URL alone and then serve it to a browser, or serve cached HTML to an agent that explicitly asked otherwise. Cloudflare's own converted responses set vary: accept. Yours must too.
2. A cache key that ignores the header. Vary tells caches what to do; some edge configurations still need the Accept header added to the cache key explicitly. Test it: request the URL as a browser, then as an agent, then as a browser again, and confirm you get HTML, Markdown, HTML — not HTML, Markdown, Markdown.
3. Applying noindex to a same-URL Markdown response. Tempting, and wrong. If any crawler ever receives that variant, you have applied noindex to the canonical URL of a page you want indexed. Same-URL negotiation needs no robots directive at all — Googlebot asks for HTML and gets HTML. Reserve X-Robots-Tag: noindex for a separate .md URL, where a second indexable copy is a genuine duplicate-content risk, and pair it with a canonical pointing back to the HTML.
4. Serving different substance, not a different format. The cloaking question comes up constantly and the answer is clean: format negotiation is not cloaking, because both representations carry the same content. The moment your Markdown variant contains claims, keywords or offers that the HTML does not, it stops being a representation and becomes a second page written for machines. Generate the Markdown from the same source as the HTML and the question never arises.

Verifying is a two-line job. Confirm the negotiation, then confirm the caching:

curl -sI https://yoursite.com/some-page -H 'Accept: text/markdown' \
  | grep -iE 'content-type|vary|cf-cache-status'

You want content-type: text/markdown and vary: accept in that output. If Vary is absent, stop and fix that before you enable anything else.

How agents discover your Markdown in the first place

Negotiation only helps an agent that already has your URL and already thinks to ask. Discovery is a separate problem, and it is the one the llms.txt v2 spec — published 10 August 2026 — set out to solve.

v2 adds two standard link relations. rel='alternate' type='text/markdown' points at a page's Markdown version; rel='describedby' points at the llms.txt file covering it. Both can be HTML <link> elements or an HTTP Link: response header:

Link: </docs/page.md>; rel='alternate'; type='text/markdown',
      </docs/llms.txt>; rel='describedby'

The header form is the interesting one. It works for non-HTML resources, and it can be added in web-server or CDN configuration without editing a single page — which for most sites is the difference between a half-day and a sprint. v2 also accepts both Markdown URL shapes, page.html.md and page.md, so whichever your build tool already produces is now spec-compliant.

Because the two mechanisms are complementary — negotiation optimises how content is served, llms.txt tells an agent what exists — most sites shipping one should ship both. Our free llms.txt generator will scaffold the index side from your existing pages, and our guide to generating an llms.txt file covers automating the refresh so it does not rot.

So does your site need a Markdown version?

Answer it from your logs, not from a blog post — including this one. Before building anything, count how many requests to your site carry text/markdown in the Accept header over the last thirty days. For most marketing sites the answer is approximately zero, and that ends the discussion.

The honest summary for the rest of 2026: this is cheap and correct infrastructure for anyone whose audience includes agents, it is a distraction for everyone else, and it does nothing for Google rankings either way. The hard part is not the header — it is knowing whether agents are reaching your pages at all, which is what server logs and crawl access tell you. If you would rather have that read for you, our one-time site audit and roadmap covers exactly which bots are hitting your site, what they are getting, and what to fix first.