What is llms.txt, and should you have one?
A proposed file that points language models at your best content in plain Markdown. It is not a standard, it is not robots.txt, and it does not control access — here is what it does and does not do.
By Sapun Lamichhane · Arcetis
llms.txt is a proposed convention, introduced by Jeremy Howard in 2024, for a Markdown file at the root of your domain that points language models at your most useful content. The reasoning behind it is sound: a model working with a limited context window cannot read your whole site, and a crawler landing on your homepage has to guess which of four hundred pages matter.
Before anything else, two clarifications, because both are widely misunderstood and one of them wastes real effort.
It is not robots.txt, and it grants nothing
robots.txt controls access. llms.txt offers guidance. If your robots.txt disallows an AI crawler, adding llms.txt changes nothing whatsoever — the crawler is not permitted to fetch the file that would have told it where to look. Access first, guidance second, and in that order.
The second clarification: it is a proposal, not a standard. There is no RFC, no W3C specification, and no major AI operator has publicly documented honouring it. Anyone telling you that llms.txt is required for AI visibility is describing a hope rather than a documented behaviour.
So is it worth adding?
For most content-heavy sites, yes — with clear expectations. It takes about half an hour, it cannot break anything, and it costs nothing to maintain if you keep it short. The realistic case for it is that the convention gets adopted and you already have one. The realistic case against is that it never does, and you spent half an hour.
What it should not be is your response to discovering you have an AI visibility problem. If assistants are not citing you, the cause is almost always one of two things that llms.txt does not touch: a robots.txt token blocking the crawler, or page structure that makes your content hard to extract. Fix those first.
What the file looks like
Plain Markdown at https://example.com/llms.txt. An H1 with the site name, an optional blockquote summarising what you do, then H2 sections of annotated links. The annotations matter more than the links — they are what lets a model decide whether a page is worth spending context on.
# Example Co
> Accounting software for independent contractors in the UK.
## Docs
- [Getting started](https://example.com/docs/start): Account setup and first invoice, ten minutes.
- [VAT handling](https://example.com/docs/vat): How VAT is calculated, including the flat rate scheme.
## Policies
- [Pricing](https://example.com/pricing): Current plans and limits.
- [Security](https://example.com/security): Data handling and retention.
## Optional
- [Changelog](https://example.com/changelog): Release history.An `## Optional` section at the end is part of the convention: it marks links a model may skip when context is tight. It is the one part of the format that directly acknowledges the constraint the file exists to address.
Writing one that is actually useful
- Describe each link in terms of the question it answers, not its title. 'How VAT is calculated, including the flat rate scheme' is useful; 'VAT page' is not.
- Link the pages that answer questions, not the pages you most want traffic to. These are frequently different, and the file is worthless if you treat it as a promotional surface.
- Keep it short. Twenty well-chosen links beat two hundred, and two hundred defeats the purpose of the file entirely.
- Serve it as text/markdown or text/plain. A file that returns your HTML error page is worse than no file.
- Update it when your documentation moves. A stale llms.txt pointing at 404s is an active liability.
The order that matters
- 1Confirm your robots.txt permits the AI crawlers you actually want, token by token. Nothing else matters until this is right.
- 2Make sure your substantive content is in the initial HTML rather than rendered client-side.
- 3Give every answer a heading that states its question, and every page a named author and date.
- 4Then, if you like, add llms.txt as a cheap bet on a convention that may or may not be adopted.
Our AI crawler checker reports whether you serve an llms.txt, whether it is served with a sensible content type, and — far more importantly — exactly which AI crawlers your robots.txt permits.