llms.txt: What It Is and How to Create One

llms.txt is a plain-text Markdown file you place at the root of your website (at /llms.txt) that gives large language models a clean, curated map of your most important pages. Think of it as a robots.txt for the AI era: instead of telling crawlers what to avoid, it tells AI models what matters and where to find it.

The format was proposed in 2024 by Jeremy Howard of Answer.AI and is documented at llmstxt.org. The pitch is simple: web pages are full of navigation, scripts and markup that waste an AI model's limited context window, so a curated Markdown index helps models ingest your canonical content efficiently.

What is llms.txt?

llms.txt is a single Markdown file that lists your site's key pages as titled links, often with one-line descriptions, so an AI system can find and read your best content without crawling the whole site. It lives at a fixed location, https://yourdomain.com/llms.txt, the same way robots.txt and sitemap.xml do.

The standard (llmstxt.org)

There is no official, search-engine-backed standard for llms.txt. It is a community proposal published at llmstxt.org and adopted voluntarily by some documentation platforms and tools. Treat it as a useful convention, not a rule Google or OpenAI has endorsed.

llms.txt vs robots.txt vs sitemap.xml

What an llms.txt file looks like

An llms.txt file starts with an H1 (your site name), an optional blockquote summary, then H2 sections containing Markdown link lists. Here is a real, annotated example:

# AI Ranking

> Community and course teaching business owners AI search
> engine optimization (SEO and GEO).

## Core guides
- [Learn GEO](https://airankingskool.com/learn/geo/): Get cited by AI search
- [Learn AI SEO](https://airankingskool.com/learn/ai-seo/): Automate SEO with AI
- [llms.txt guide](https://airankingskool.com/learn/geo/llms-txt/): This page

## Optional
- [About](https://airankingskool.com/about/): Founder story and authority

File structure and syntax

• H1: the site or project name (required, one only).
• Blockquote: a short summary of what the site is.
• H2 sections: grouped Markdown link lists (for example "Core guides", "Docs").
• Optional section: links an AI can skip if context is tight.

llms.txt vs llms-full.txt

Some sites also publish llms-full.txt, which inlines the full Markdown content of every page rather than just linking to it. It is heavier but lets a model ingest everything in one request. Use llms.txt as the index and llms-full.txt only if your content is small enough to fit.

How to create an llms.txt file

1. List the 10 to 30 pages you most want AI models to read (pillars, key guides, product pages).
2. Write a one-line description for each, focused on what the page answers.
3. Format them as Markdown: an H1 title, a blockquote summary, then H2 sections with link lists.
4. Save the file as llms.txt and upload it to your site root so it resolves at /llms.txt.
5. Validate it, then keep it updated as you publish important new pages.

Generate it (free generators)

You do not have to write it by hand. Free generators like Firecrawl (llmstxt.firecrawl.dev) and Mintlify's auto-generation will crawl your site and produce a draft. Inside the community we use DataWise to generate and maintain it, then prune it down to your money pages.

Platform notes (WordPress, Next.js, Wix)

• WordPress: the easiest route. Both Yoast SEO and Rank Math now include a free llms.txt option: flip the toggle on (Yoast generates and refreshes it automatically and lets you preview it; in Rank Math it is the "LLMS Txt" module) and the file is served at /llms.txt with no code. A plugin-generated file will not be as curated as one you write by hand, but it is a solid one-click start.
• Next.js: serve it from /public/llms.txt or a route handler.
• Wix / Squarespace: use a file or redirect that exposes /llms.txt at the root.

Validate it

Run your file through an llms.txt validator (low effort, easy win) to confirm the Markdown structure is correct and every link resolves. Broken links in your llms.txt defeat the purpose.

Does llms.txt actually work?

Is anyone actually using it?

Yes, on the publishing side. Semrush counted roughly 951 domains with an llms.txt file by mid-2025, and directories like directory.llmstxt.cloud track adoption. Many of those are documentation sites (Mintlify, Anthropic docs and similar) where a clean Markdown index is a natural fit.

The reality check: do AI crawlers fetch it?

This is where most guides hedge and we will not. If you check your bot analytics or raw server logs, you will usually see plenty of hits to your normal HTML pages from AI crawlers and almost none to /llms.txt. The file is a forward-looking convention, not a switch that turns on AI traffic.

Why "it's useless" is now the outdated take

For most of 2024 and 2025 the loudest verdict on llms.txt was that it is useless, and the proof offered was always the same: check your server logs and you will see almost no AI crawlers requesting the file. That was a fair observation. It also measured the wrong thing.

In its experimental agentic browsing audit, Google's Lighthouse (the site-quality tool built into Chrome DevTools) now grades how ready your site is for AI agents. One of its checks sits under a section called Stability and Discoverability, is named simply llms.txt, and "checks for the presence of a machine-readable summary at the domain root." In plain English: Google's own tooling now looks for an llms.txt file and counts its presence in your favor.

What actually wins AI citations

AI search cites pages it can read and trust. That comes from a direct answer in your first two sentences, clean heading structure, real expertise and being mentioned across the web, not from a text file alone. That is the core of generative engine optimization, and it is exactly what we teach. Add llms.txt because it is cheap and future-friendly, but put your real effort into answering questions directly and getting cited by ChatGPT.