How to Generate an llms.txt File (And What v2 Changed)

Four ways to generate an llms.txt file

The rule that decides it: if a human has to remember to update it, it will be wrong within a quarter. Pick whichever route ends in a scheduled job. A generator or plugin is a fine starting point; it should not be the steady state for a site that publishes regularly.

What the v2 spec changed, and what generators still get wrong

Jeremy Howard published v2 of the proposal on 10 August 2026, the first substantive revision since the original in September 2024. If you are generating a file this month, generate the v2 shape — and check whatever tool you use against these four points, because most were built against v1:

• The file no longer has to sit at the domain root. An llms.txt covers the pages under its own path, so /docs/llms.txt describes everything in /docs/, and where several files apply an agent uses the most specific one. If your documentation is the only part agents care about, scope the file to it rather than describing your whole marketing site.
• Link relations are the new discovery mechanism. v2 adds rel='alternate' type='text/markdown' to point at a page's Markdown version and rel='describedby' to point at the llms.txt that covers it, delivered as HTML <link> elements or an HTTP Link: header. The header form can be set in your web server or CDN config without editing pages, which makes this the cheapest part of the whole exercise.
• Both Markdown URL shapes are valid now. v1 specified page.html.md; v2 also allows page.md, matching what most static-site generators already produce.
• The Optional section lost its machine meaning. v1 tooling used it to decide what to drop when context was tight. v2 removed that tooling from the proposal, so Optional is now purely a human convention for secondary links. Any guide still describing it as a machine-readable instruction is describing v1.

The structural rules did not change, and they are short. One H1 with your site or project name — the only required element. A blockquote summary directly under it. Optional prose with no headings. Then zero or more H2 sections, each a Markdown list of - [name](url): notes. That is the whole grammar; our annotated llms.txt examples walk through real files element by element if you want to see it applied.

Generate a first draft from your sitemap

Your sitemap already knows every URL you publish, which makes it the obvious input. It is a poor substitute for llms.txt — it is exhaustive where this file must be curated — but an excellent starting point. This script walks a sitemap or sitemap index, groups URLs by their first path segment, drops the sections nobody wants an agent reading, and caps each group so the output stays inside a context window:

#!/usr/bin/env python3
'''Generate a first-draft llms.txt from a sitemap. Curation is still your job.'''

import sys
import urllib.request
from collections import defaultdict
from urllib.parse import urlparse
from xml.etree import ElementTree

NS = {'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
SECTION_TITLES = {
    'docs': 'Documentation',
    'blog': 'Articles',
    'products': 'Products',
    'services': 'Services',
}
SKIP_SEGMENTS = {'tag', 'author', 'page', 'search', 'cart', 'checkout'}
MAX_PER_SECTION = 8


def fetch(url):
    request = urllib.request.Request(url, headers={'User-Agent': 'llms-txt-builder/1.0'})
    with urllib.request.urlopen(request, timeout=20) as response:
        return response.read()


def collect_urls(sitemap_url, seen=None):
    '''Walk a sitemap or sitemap index and return every loc found.'''
    seen = seen if seen is not None else set()
    if sitemap_url in seen:
        return []
    seen.add(sitemap_url)

    root = ElementTree.fromstring(fetch(sitemap_url))
    if root.tag.endswith('sitemapindex'):
        urls = []
        for child in root.findall('sm:sitemap/sm:loc', NS):
            urls.extend(collect_urls(child.text.strip(), seen))
        return urls
    return [loc.text.strip() for loc in root.findall('sm:url/sm:loc', NS)]


def section_for(url):
    path = urlparse(url).path.strip('/')
    if not path:
        return 'Core Pages'
    segments = path.split('/')
    if segments[0] in SKIP_SEGMENTS:
        return None
    if len(segments) == 1:
        return 'Core Pages'
    return SECTION_TITLES.get(segments[0], segments[0].replace('-', ' ').title())


def label_for(url):
    path = urlparse(url).path.strip('/')
    if not path:
        return 'Home'
    return path.split('/')[-1].replace('-', ' ').replace('_', ' ').title()


def build(sitemap_url, site_name, summary):
    grouped = defaultdict(list)
    for url in collect_urls(sitemap_url):
        section = section_for(url)
        if section is None:
            continue
        grouped[section].append(url)

    lines = [f'# {site_name}', '', f'> {summary}', '']
    for section in sorted(grouped, key=lambda s: (s != 'Core Pages', s)):
        urls = sorted(set(grouped[section]), key=len)[:MAX_PER_SECTION]
        lines.append(f'## {section}')
        lines.append('')
        for url in urls:
            lines.append(f'- [{label_for(url)}]({url}): TODO describe what this page answers.')
        lines.append('')
    return '\n'.join(lines).rstrip() + '\n'


if __name__ == '__main__':
    if len(sys.argv) != 4:
        sys.exit('usage: gen_llms_txt.py SITEMAP_URL SITE_NAME SUMMARY')
    print(build(sys.argv[1], sys.argv[2], sys.argv[3]))

Run it against a sitemap and you get a valid skeleton in seconds:

$ python3 gen_llms_txt.py https://example.com/sitemap.xml \
    'Example Co' 'Example Co builds monitoring tools for engineering teams.'

# Example Co

> Example Co builds monitoring tools for engineering teams.

## Core Pages

- [Home](https://example.com/): TODO describe what this page answers.
- [Pricing](https://example.com/pricing): TODO describe what this page answers.

## Documentation

- [Quickstart](https://example.com/docs/quickstart): TODO describe what this page answers.
- [Api Reference](https://example.com/docs/api-reference): TODO describe what this page answers.

Note what the output is telling you. Api Reference is wrong — the slug-to-title guess mangles acronyms — and every description says TODO. That is deliberate. A script can find and group your URLs; it cannot judge which of them an agent should read first or what each page actually answers. Treat the output as scaffolding, not as a file.

Two knobs worth tuning for your own site: extend SKIP_SEGMENTS with anything paginated, faceted or gated, and lower MAX_PER_SECTION if your sections are large. The practical target is roughly three to five sections and ten to twenty links total — an index, not an inventory.

If you are not writing a script: platform routes

If you want a spec-correct draft without installing anything, our free llms.txt generator crawls a domain and returns a structured file you can edit and download — useful for judging whether the file is worth keeping before you wire it into a pipeline.

The two parts no generator can do for you

The blockquote is the load-bearing line

The summary under your H1 is the single sentence an AI system is most likely to lift verbatim as the definition of what you are. Whatever you write there tends to become what the model treats as true about you. Which means:

• Write what you do, who you serve, and what is specific about it. No leading, no best-in-class, no innovative.
• Make it verifiable from your own pages. A claim the rest of the file does not support is a claim an agent will drop or contradict.
• One or two sentences. This is a definition, not a pitch.

Compare: Acme is a leading provider of innovative analytics solutions tells a model nothing. Acme Analytics is a product analytics platform for SaaS teams, covering event tracking, funnels and retention gives it three concrete facts it can use in an answer.

Descriptions should answer, not label

The generator emits page titles. What earns the fetch is the note after the colon telling an agent what question that page settles — Endpoints, auth and rate limits with examples rather than API documentation. On a commerce site this is where you put the facts an agent needs to answer a real query: categories, price bands, fitment, returns terms. Done well, one cheap fetch replaces crawling forty product pages.

Validate before you ship

Four checks, in order. The first two take seconds and catch the most common failures:

1. It resolves and is plain text. A file served as text/html because your framework wrapped it in a template is the most common mistake, and it is invisible in a browser.

   curl -sI https://yoursite.com/llms.txt | grep -iE 'HTTP/|content-type'

   You want a 200 and text/plain. A 200 with text/html means you are serving a page, not a file.
2. Every link returns 200. Dead links are worse than a missing file, because an agent that does fetch it now has confidently wrong context.

   grep -oE 'https?://[^)]+' llms.txt \
     | xargs -P8 -I{} sh -c 'printf "%s %s\n" "$(curl -s -o /dev/null -w "%{http_code}" {})" "{}"' \
     | grep -v '^200'

   Silence means everything passed.
3. Structure is valid. Exactly one H1, absolute https:// URLs only, no headings inside the intro prose, every list item a real Markdown link.
4. The spec's own test: ask an agent. Give a model nothing but your llms.txt and ask it questions about your business. If it cannot tell what you sell or which page to open next, the problem is your curation, not the format. This is the check most people skip and the only one that measures whether the file works.

Automate the refresh so it does not rot

Ahrefs' study of 137,210 domains found that 97% of published llms.txt files got zero requests in a month — but the ones that are fetched are mostly on documentation and developer sites, where content changes constantly. Those are exactly the files most likely to drift out of date. So wire the regeneration into the pipeline that already runs:

• Build-time is best. If you deploy from a repository, generate the file during the build. It cannot be stale, because it is rebuilt whenever the content is.
• Scheduled is fine for CMS-driven sites. A weekly job that regenerates and opens a pull request keeps a human in the loop for the descriptions without making them remember.
• Fail the build on dead links. Run the link check above in CI and exit non-zero. This single step is what separates a file that stays useful from one nobody trusts after six months.
• Regenerate on structural change, not on every post. A new blog article rarely belongs in a curated index. A new product line, docs section or pricing page does.

Finally, while you are in there: if you are shipping llms.txt for agents, the companion move under v2 is making your pages available as Markdown so the links actually point at agent-friendly content. That is a separate build — we covered the header negotiation, the token savings and the caching trap in serving Markdown to AI agents.

If you would rather have someone look at whether any of this is worth doing on your specific site before you build it, we run a free AI-visibility consultation — no cost, no account, and an honest answer if the files are not your bottleneck.