Everyone is adding llms.txt to their website. Plugins generate it automatically. Checklists include it. SEO tools flag its absence as an issue. There is one problem with all of this: the file almost certainly does nothing, and the reasons why go right to the heart of how AI systems actually work.
Three things changed after this article was published. First, Google released its official AI optimization guide explicitly dismissing LLMs.txt as unnecessary for Google Search. Then two contradictions surfaced on the same day. Google’s own developer documentation at ai.google.dev runs a standard LLMs.txt file as the discovery layer for a complete set of .md.txt companion pages built for AI systems to read cleanly. And Chrome’s Lighthouse tool added a dedicated LLMs.txt audit to its new Agentic Browsing category on May 5, 2026, flagging sites where the file returns a server error.
The picture is now more specific than “useless” or “essential.” LLMs.txt does nothing for Google Search. Google’s own systems do not use it. But for ChatGPT, Claude, Perplexity, and other AI systems that crawl the web independently, it remains the only reliable discovery mechanism for machine-readable content that is not internally linked from HTML pages. Google knows this, which is why it built the system for its own documentation while telling publishers they do not need it for Search. The scope distinction is everything. Full analysis here.
What is llms.txt, and where did it come from?
In September 2024, Jeremy Howard, a researcher and lecturer at the Universities of Queensland and Stanford and co-founder of the AI labs Answer.AI and fast.ai, published a proposal for a new file called llms.txt. The idea was simple: place a plain text file at the root of your website, written in Markdown format, that summarises your most important content. AI systems, the proposal argued, could then read this file and get a clean, structured overview of your site without having to crawl dozens of pages full of ads, navigation menus, and JavaScript clutter.
On paper, the comparison to robots.txt seems obvious. robots.txt has been around since 1994 and tells search engine crawlers which parts of a site they are allowed to visit. sitemap.xml lists all the URLs on a site so search engines can find them efficiently. Why not have a similar file for AI systems?
The answer is that robots.txt and sitemap.xml work because search engines agreed to honour them. No such agreement exists for llms.txt.
The numbers: a study of 300,000 domains
In November 2025, analytics company SE Ranking published an analysis of approximately 300,000 domains to test whether having an llms.txt file had any measurable relationship with how often a site was cited in AI-generated answers.
The conclusion was clear: adding llms.txt is low effort, but the data suggests you should not expect any measurable improvement in how often AI systems cite your content.
Why the whole idea misunderstands how LLMs work
To understand why llms.txt cannot do what it promises, you need to understand the two very different ways an AI system uses the web: training and inference.
Training is when an AI system learns from data. It happens offline, in bulk, before the model is ever released to the public. The training data for most large language models comes primarily from a dataset called Common Crawl, a massive archive of web pages that has been collected since 2008. A peer-reviewed study presented at the 2024 ACM Conference on Fairness, Accountability, and Transparency found that over 80% of GPT-3’s training tokens came from Common Crawl, and at least 64% of 47 major language models published between 2019 and 2023 used it.
Inference is when a user asks the AI a question and it generates a response. Some AI systems with web search tools retrieve live pages at this stage using a process called Retrieval-Augmented Generation (RAG). This is what happens when ChatGPT or Perplexity pulls in a recent news article to answer a question about something that happened last week.
The llms.txt file has no role in either of these processes. Training datasets are assembled in bulk from Common Crawl snapshots, not by visiting individual websites and checking for hint files. RAG retrieval systems query search indexes and fetch pages directly, not proprietary markdown guides that website owners wrote about themselves.
The assumption behind llms.txt is that an AI system visits your website, sees the file, reads it, and then understands your site better as a result. That is not how any of this works. The analogy would be leaving a note on your front door asking a satellite to take better photos of your house.
What Google and the major AI companies actually say
Google’s Search Advocate John Mueller has been one of the clearest public voices on this. In April 2025, he compared llms.txt directly to the keywords meta tag, a feature that search engines abandoned years ago because it was trivially easy to abuse and provided no reliable signal. In June 2025, Mueller stated publicly that no AI system was using the file, pointing to server log data as evidence.
At Google’s Search Central Live event in July 2025, Google’s Gary Illyes confirmed that Google does not support llms.txt and has no plans to do so.
| Claim about llms.txt | What the evidence shows | Verdict |
|---|---|---|
| AI systems read it at crawl time | Server log audits show AI bots are not checking for the file. Google’s desktop crawler accounted for 95% of observed hits in one 1,000-domain audit, with GPTBot absent entirely. | FALSE |
| It improves AI citation frequency | An analysis of 300,000 domains found no statistical relationship between having llms.txt and how often a domain is cited in AI-generated answers. | Not supported |
| It works like robots.txt for AI | robots.txt works because search engines agreed to honour it as a standard. No major AI company has made that commitment for llms.txt. | MISLEADING |
| It affects training data inclusion | LLM training uses pre-assembled datasets like Common Crawl. It does not involve visiting individual websites and checking for hint files during training. | FALSE |
| Google is building support for it | A brief appearance in Google’s developer docs on December 3, 2025 was quickly removed. Mueller confirmed it was not an endorsement. | FALSE |
| It is harmless to add | Low direct harm, but it can expose your full content structure to competitors and wastes developer time that could go toward things that actually work. | Debatable |
The one situation where it actually makes sense
The original proposal was not aimed at marketing teams trying to rank better. Jeremy Howard designed it for developer documentation: technical sites where AI coding tools like Cursor, GitHub Copilot, or Claude in an IDE context need to understand an API’s structure quickly. When an AI assistant is helping a developer use your library, having a well-structured llms.txt pointing to your API reference pages has a clear, practical value.
Vercel, Stripe, Cloudflare, and similar developer-focused platforms have implemented it for exactly this reason, and the use case is legitimate. The problem is that this specific use case has been repackaged and sold to every website owner as a general AI visibility strategy, which it was never meant to be.
Why the myth keeps spreading
The SEO tool industry has a financial interest in new things to implement. SE Ranking, RankMath, and other tools began flagging missing llms.txt as a site issue, which created demand from site owners who saw the warning and wanted to fix it. Plugins were built. Guides were published. The feedback loop created the appearance of an established best practice where none actually exists.
There is also a very human tendency to draw parallels between new things and familiar ones. robots.txt and sitemap.xml are simple files that made a real difference. The idea that a similar file could give you an edge in AI search sounds logical. It just does not match how the technology actually works.
In December 2025, an llms.txt briefly appeared in Google’s own developer documentation. The SEO community interpreted it as a sign of endorsement. It was removed the same day. Mueller’s response on Bluesky was: “hmmn :-/”
The file had been added by a team inside Google that was experimenting with the format for internal documentation purposes. It had nothing to do with Google Search adopting the standard.
What actually works for AI visibility
The things that get your content cited in AI-generated answers are the same things that have always determined content quality in the eyes of search systems: authority, clarity, structure, and trust.
Write in clear, self-contained chunks. RAG systems retrieve passages, not full pages. Each section of your content should make sense on its own, answer a specific question, and avoid relying on context from other parts of the page.
Use semantic HTML and structured data. Schema.org markup in JSON-LD format tells AI retrieval systems exactly what type of content they are reading, who wrote it, and what it refers to. This is the actual signal layer that search-based RAG uses.
Build entity authority across the web. AI systems trained on Common Crawl and similar datasets learn about your brand through consistent mentions across many authoritative sources. One llms.txt file cannot replace that signal.
Let AI crawlers access your content. Make sure GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are not blocked in your robots.txt. If AI systems cannot crawl your pages at all, no hint file can compensate for that.
Write factually and directly. AI answer systems favour content that answers questions in the first sentence, uses precise language, and does not pad answers with marketing copy. The format that works best for AI is also the format that works best for humans.
The short version
llms.txt was proposed as a tool for developer documentation, not as an SEO tactic. No major AI company officially supports it. An analysis of 300,000 domains found no measurable connection between having the file and appearing in AI-generated answers. Google’s own representatives have compared it to the keywords meta tag, which was abandoned for exactly this kind of reason. Server logs show AI bots are not checking for it.
If you run a developer-facing product and want to help AI coding assistants navigate your documentation, implementing llms.txt is a reasonable experiment. For everyone else, the time is better spent on structured data, clean content architecture, and building the kind of broad entity authority that AI systems learn from during training.
You do not need to become a GEO expert overnight. What you do need is someone who already is one, so the technical side gets handled properly while you focus on your business. That is exactly what we do.
Sources
- Howard, J. (2024). The llms.txt file proposal. Answer.AI / llmstxt.org. llmstxt.org
- SE Ranking. (November 2025). LLMs.txt shows no clear effect on AI citations, based on 300k domains. SE Ranking Research. seranking.com
- Baack, S. (2024). A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl. ACM FAccT ’24, Rio de Janeiro. doi.org/10.1145/3630106.3659033
- Mueller, J. (April 2025). Public statement on llms.txt. Reported via Search Engine Journal, April 17, 2025.
- Mueller, J. (June 2025). “No AI system currently uses llms.txt.” Public statement. Reported via Omnius, December 2025.
- Illyes, G. (July 2025). Statement at Google Search Central Live. Reported via The SEO Community. theseocommunity.com
- Longato, F. (August 2025). Why AI Crawlers Ignore llms.txt: 2025 Audit. longato.ch
- getpublii.com. (January 2026). The Complete Guide to llms.txt. Noting that SEO tools flag its absence as a site issue. getpublii.com
- Mozilla Foundation / Baack, S. (February 2024). Training Data for the Price of a Sandwich: Common Crawl’s Impact on Generative AI. mozillafoundation.org