llms.txt and llms-full.txt: What is it?

If you've been following SEO news lately, you may have stumbled across two file names: llms.txt and llms-full.txt. Sounds technical, sounds important, sounds like something you should immediately add to your website. But what's really behind it?

In short: llms.txt is a text file that lives at the root of a website, as in example.com/llms.txt. It's designed to give large language models like ChatGPT, Claude or Gemini a compact, machine readable summary of the website. Think of it as a robots.txt, but not for traditional search engine crawlers, rather for AI systems.

The idea comes from Jeremy Howard, the founder of fast.ai. His proposal: Instead of making an LLM wade through hundreds of HTML pages, cookie banners and JavaScript frameworks, you give it a cleanly formatted text file with the key information. The website name, what it's about, which pages are relevant, what the core topics are.

llms.txt is the short version. Compact, reduced to the essentials, a few hundred lines at most. llms-full.txt is the detailed version with more depth, longer descriptions and deeper insights into the content.

Now the exciting question: Is it actually being crawled?

Honestly: not much as of today. OpenAI has GPTBot as a crawler, Anthropic has ClaudeBot, Google has its own. But none of these crawlers currently look specifically for an llms.txt file the way Googlebot looks for robots.txt or sitemap.xml. The format is a proposal, not a standard. There is no official specification from Google, OpenAI or Anthropic that says: Yes, we read this file.

Still, the idea isn't bad. For several reasons.

First: If you work with AI tools that analyse websites, for example Perplexity, ChatGPT with browsing or Claude with MCP, then an llms.txt can serve as a quick reference. The tool doesn't have to parse the entire site but gets a clean summary. That saves tokens and delivers better results.

Second: If LLM crawlers actually start looking for such a file in the future, you're prepared. Early adopter advantage. Similar to how sitemap.xml took years to become a real standard.

Third: It forces you to think about your website structure. What are the most important pages? What is the core message? That's a valuable exercise in itself.

What does an llms.txt look like? Pretty simple: The first section describes the website. A title, a short description in one or two sentences. Then a list of the most important pages with URLs and a brief explanation of what can be found there. Optionally you can add contact information, main topics or a summary of services. The format is deliberately kept simple, plain text, no HTML tags, no JSON, no XML.

The llms-full.txt goes deeper. Here you can write full paragraphs about individual pages, summarise blog posts, describe portfolio projects. Everything that could help an LLM truly understand the website.

Can you improve your crawling with it? Not directly. The major search engine crawlers currently ignore the file. But you can benefit indirectly. If AI powered search systems like Perplexity or Google SGE understand your content better, you're more likely to be cited as a source. And being cited in AI answers is the new ranking.

My advice: If you're building a new website or already working on the SEO structure, create an llms.txt. Takes 20 minutes, doesn't hurt and could become relevant in the future. If you have an existing website and other priorities right now, there's no rush. It's not a ranking factor and probably won't be anytime soon.

What you definitely should not do: keyword stuffing in the llms.txt. The file is meant for machines that understand natural language. They notice immediately when you try to manipulate them. Just write honestly and clearly what your website does and who it's for. That's enough.

In that spirit: llms.txt is a neat experiment on the edge of the SEO world. Not a must have, but an interesting glimpse into the future. And who knows, maybe in two years every AI crawler will read this file first. Then you were already prepared.