Google's official AI optimization guide says publishers do not need LLMs.txt or machine-readable files for AI search. Google's own developer documentation uses LLMs.txt as a discovery layer for a complete set of Markdown companion files built for AI systems to consume. Both things are true. Understanding why tells you more about the future of content than either statement does on its own.
On May 19, 2026, Google published its official guide to optimizing for generative AI search. It contains this line:
“You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search.”
The guide is published at developers.google.com/search/docs/fundamentals/ai-optimization-guide.md.txt. That extension is not a typo. It is a Markdown file. And when you trace how AI systems are supposed to find it, you arrive at ai.google.dev/api/llms.txt: a standard LLMs.txt index file listing every page in Google’s developer documentation alongside a link to its clean Markdown companion.
Google built exactly what it told you not to build. The question worth asking is why.
Google’s developer documentation runs a three-layer system: a LLMs.txt file for AI crawlers to discover pages, individual .md.txt files per page for AI systems to read content cleanly, and standard HTML for human visitors. None of the HTML pages link to their .md.txt companions. The only way an AI system finds the Markdown files is through the LLMs.txt index. Google built a complete machine-readable content layer and used the convention it publicly dismissed to make it discoverable.
How the Architecture Actually Works
To understand what Google built, you need to follow the three layers in sequence, because each one only makes sense in relation to the others.
Layer 1: LLMs.txt as the discovery index
At ai.google.dev/api/llms.txt, Google maintains a standard LLMs.txt file. It lists every page in the Gemini API documentation as a Markdown link pointing to that page’s .md.txt companion. An AI crawler that checks for LLMs.txt at the site root finds this file, reads the list, and knows exactly where to find clean machine-readable content for every page on the site. Without this file, the individual .md.txt pages are invisible to crawlers. There is no link from any HTML page to its Markdown version, no rel="alternate" in the page head, nothing. The LLMs.txt is the only entry point.
Layer 2: .md.txt files as the content layer
Each page in Google’s developer documentation has a .md.txt companion at a predictable URL: the HTML page URL with .md.txt appended. These files are pure Markdown served as plain text. No navigation. No cookie banners. No footer links. No JavaScript dependencies. Just the content, in a hierarchy an AI system can parse without any preprocessing. A page about billing becomes a clean structured document. A page about OAuth becomes a step-by-step guide with no surrounding noise.
Layer 3: HTML for humans
The HTML pages remain unchanged and serve human visitors normally. They are fully indexed by Google Search through the standard pipeline. They rank, they appear in results, they get crawled by Googlebot exactly as any other page would. The machine-readable layer sits alongside them, completely parallel, invisible to human visitors and to any system that does not specifically look for it.
Why Google’s Guide and Google’s Behavior Are Both Correct
This is not a contradiction once you read the guide carefully. Google’s statement that you do not need machine-readable files is specifically about appearing in Google Search’s generative AI features. That claim is true. Google’s own crawler reads HTML. Its RAG system pulls from its standard index. A LLMs.txt file does nothing to improve how Googlebot crawls your site or how AI Mode cites your content.
But Google’s developer documentation is not trying to rank better in Google Search. It is trying to be read accurately by every AI system that exists: ChatGPT, Claude, Perplexity, Gemini itself, and any agent or tool that browses the web to answer technical questions. Those systems have different crawlers, different content pipelines, and different preferences. Some of them check for LLMs.txt. Some of them handle HTML poorly when it contains heavy navigation or JavaScript-loaded content. For all of them, a clean Markdown file with a predictable URL is a better input than the HTML equivalent.
Google’s AI optimization guide is scoped to Google Search. It makes accurate claims about what helps you appear in AI Mode and AI Overviews. It says nothing about what helps you appear in ChatGPT answers, Claude responses, Perplexity citations, or any other AI system. Those systems are outside Google’s jurisdiction and outside the guide’s scope. Publishing the guide while simultaneously maintaining LLMs.txt and .md.txt files for the benefit of those other systems is entirely consistent once you understand the scope distinction. Most people reading the guide missed it.
How Different AI Systems Actually Use This
To make this concrete, here is how the three-layer system plays out differently depending on which AI system is reading Google’s documentation.
A developer asks Perplexity how to use the Gemini API
Perplexity’s crawler checks for LLMs.txt at ai.google.dev, finds the index, and retrieves the relevant .md.txt pages directly. The content it reads has no navigation chrome, no tab-switched code examples, no cookie consent interruptions. It extracts the exact authentication steps, the correct endpoint, and the current pricing tier in one clean pass. The answer it produces for the developer is accurate, complete, and correctly attributed to Google’s documentation. Perplexity cites the source. Google’s documentation appears in the answer.
If the .md.txt files did not exist and Perplexity had to read the HTML, it would still get most of the content. But “most” is not the same as “all,” and in technical documentation the parts that get lost in HTML parsing are often the most important: code examples in inactive tabs, version-specific notes in collapsed sections, parameters buried in long JavaScript-rendered tables.
A user asks Claude to explain a Gemini API endpoint
Suppose a developer pastes a link to a Gemini API reference page and asks Claude to explain how a specific method works. If Claude fetches the HTML page, it receives the full page including navigation, related links, feedback widgets, and the actual documentation content mixed together. Claude has to identify which parts are the documentation and which are page furniture, and it does this reasonably well but not perfectly.
If Claude instead fetches the .md.txt companion, it receives a file that is entirely documentation with zero surrounding noise. Every parameter, every code example, every note about deprecation or version requirements is present and clearly structured. The explanation Claude produces is more precise, and the developer is less likely to encounter a subtle error caused by a parameter Claude misread from a cluttered HTML table.
A ChatGPT agent browses Google’s documentation to complete a coding task
Suppose someone uses a ChatGPT operator setup to automate a task that involves calling the Gemini API. The agent is given a goal and needs to read Google’s documentation to understand the correct request format. It browses to the relevant page. The HTML version of the page loads a navigation sidebar with dozens of links, a feedback module at the bottom, and code examples that require the user to click a language tab to see the Python version.
The agent reads what is in the DOM at load time. The Python code example is in an inactive tab and not present in the initial HTML. The agent writes the API call using the JavaScript example it did find, which uses a different syntax. The task fails. The developer spends an hour debugging a problem that the documentation would have prevented if the agent had read the .md.txt version, where all code examples are present simultaneously with no tab switching required.
Google’s own AI Mode cites its developer documentation
This is the one case where the .md.txt files do not help and Google’s guide is straightforwardly accurate. When someone asks AI Mode a question about the Gemini API, Google’s RAG system retrieves from its standard Search index. The HTML pages are indexed, ranked, and cited. The .md.txt files may also be indexed, but they are not adding anything the HTML pages do not already provide for Google’s own systems. Here, the guide’s advice is correct: you do not need machine-readable files to appear in Google’s generative AI features, because Google’s systems are built to handle HTML at scale.
Google built the three-layer system for the AI systems it does not control. Its own systems do not need it. Every other system benefits from it. The LLMs.txt and .md.txt investment is not about ranking in Google Search. It is about being cited accurately everywhere else.
What This Means for Publishers and SEO Professionals
Google’s behavior is the clearest signal yet about where the AI content ecosystem is heading. A company with more resources than any publisher will ever have, and with the deepest possible knowledge of how AI systems consume web content, chose to build a parallel machine-readable layer for its documentation. That is a strong signal about what works, regardless of what any guide says about what is strictly required.
The practical question is not whether to copy Google’s exact implementation. Most publishers do not have documentation sites with hundreds of pages and a platform that can auto-generate .md.txt companions. The practical question is what the principle behind the implementation tells you about your own content strategy.
For sites targeting Google Search primarily
Google’s guide is accurate for your use case. Focus on clean HTML structure, explicit heading hierarchies, content that answers specific questions directly, and technical SEO fundamentals. LLMs.txt adds nothing to how Googlebot crawls your site or how AI Mode selects citations. Your effort is better spent on content quality and entity-level authority signals.
For sites where citations in ChatGPT, Claude, or Perplexity matter
The calculus changes. If your content is the kind that gets referenced in AI-assisted research, technical questions, product comparisons, or any context where accuracy and attribution matter across multiple AI systems, then the three-layer approach Google uses for its developer documentation is worth studying seriously. You do not necessarily need the full implementation. Even a well-maintained LLMs.txt pointing to your most important pages, written in clean prose with no surrounding noise, is a meaningful step toward being cited correctly by systems outside Google’s ecosystem.
For documentation and reference content sites
This is where Google’s approach is most directly applicable. If you run a documentation site, an API reference, a knowledge base, or any content that developers or researchers use to complete technical tasks, the risk of being partially consumed by an AI system is highest. Tab-switched code, collapsed sections, and JavaScript-loaded content are exactly the patterns that cause AI systems to produce subtly wrong answers. A clean Markdown companion removes all of that risk. The LLMs.txt index makes it discoverable. The combination is small in implementation cost and potentially significant in citation accuracy.
Frequently Asked Questions
Yes. Google maintains a LLMs.txt file at ai.google.dev/api/llms.txt that serves as a discovery index for its Gemini API documentation. The file lists every page in the documentation alongside a link to that page’s .md.txt companion file. This is a standard implementation of the LLMs.txt convention, built specifically so that AI crawlers from other companies (ChatGPT, Claude, Perplexity, and others) can find and read Google’s documentation in clean Markdown format.
Google’s official guide is specifically about appearing in Google Search’s generative AI features, such as AI Mode and AI Overviews. For that purpose, LLMs.txt genuinely adds nothing: Google’s own crawler reads HTML through its standard indexing pipeline. Google uses LLMs.txt for a different purpose entirely: to help AI systems from other companies read its developer documentation accurately. The guide’s scope is Google Search. The behavior’s scope is the entire AI ecosystem. Both statements are accurate within their respective scopes.
A .md.txt file is a Markdown version of a specific page, accessible at that page’s URL with .md.txt appended. It contains only the page’s content in clean structured text, with no HTML tags, no navigation, no JavaScript, and no surrounding page elements. LLMs.txt is a site-level index file, placed at the domain root, that lists all pages an AI system should know about. The two work together: LLMs.txt tells crawlers which pages exist and where their .md.txt companions are. Without LLMs.txt, .md.txt files are orphan pages with no discovery path.
It depends on your content type and which AI systems matter to your audience. For documentation, technical reference, research, and data-heavy content where accuracy in AI citations matters, a LLMs.txt pointing to your most important pages is a low-cost investment with a meaningful upside in how accurately non-Google AI systems represent your content. For local businesses, ecommerce, and content primarily targeting Google Search rankings, the effort is better spent elsewhere. Google’s behavior tells you that the convention has real value for the right use cases, regardless of what its official guide says.
Only if they know the URL convention and actively check for .md.txt variants of pages they discover. Google’s HTML pages do not link to their .md.txt companions, and there is no rel=”alternate” tag in the HTML head pointing to the Markdown version. Without LLMs.txt, a crawler would have to independently guess that appending .md.txt to a URL produces a useful file. Some crawlers may do this. Most do not. LLMs.txt is the only reliable discovery mechanism for .md.txt files that are not internally linked from the HTML version of the page.
For Google, GEO means producing content that is accurate, well-structured, clearly attributed, and technically accessible through standard crawling. Google’s RAG system selects citations from its Search index, so anything that helps you rank also helps you get cited in AI Mode. For non-Google AI systems, GEO means reducing the friction those systems encounter when reading your content: avoiding JavaScript-loaded text, minimising surrounding page noise, providing clean hierarchical structure, and making content discoverable through conventions like LLMs.txt. The two approaches overlap significantly but are not identical, and the gap between them is where Google’s own three-layer documentation architecture sits.
Sources: Google Search Central: AI optimization guide (.md.txt). Google AI developer docs: LLMs.txt index. Wikipedia: Markdown.