AI and GEO

llms.txt Explained: What It Is and Whether You Need It

Automation Services

An llms.txt file is a proposed plain-text, Markdown-formatted file placed at the root of a website to point AI tools to your most useful pages. It is not an official web standard. Google has said it does not use llms.txt for Search, and no major AI platform has confirmed that it ranks or cites sites based on it. You can add one cheaply, but do not expect it to move rankings.

This guide explains what the file is, where it came from, what it contains, and how to decide whether to make one. It is written to be honest about the evidence, because this is a topic where hype runs well ahead of fact. For the work that does affect AI citations, see our generative engine optimization services.

What is llms.txt?

llms.txt is a file you host at yourdomain.com/llms.txt. It is written in Markdown and gives a short, curated summary of your site with links to the pages that matter most. The idea is that a language model or AI agent could read this one file to understand your site quickly, instead of crawling dozens of pages with navigation and ads.

The proposal was published at llmstxt.org in 2024 by Jeremy Howard of Answer.AI. It is a community proposal, similar in spirit to robots.txt and sitemap.xml but not governed by any standards body. No search engine is required to read it, and each AI company decides for itself whether to use it.

That last point is the heart of the matter. A file only helps if something reads it. Whether anything reads llms.txt in a way that affects your visibility is the real question, and we answer it below.

Does Google use llms.txt?

No. Google has said publicly that it does not use llms.txt for Search. Google’s own documentation on AI features also says you do not need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. You can read that in Google’s AI features and your website page.

So for Google Search, including AI Overviews, an llms.txt file is not a ranking or inclusion factor. Adding one will not hurt, but it will not help you appear there either. Anyone who tells you otherwise is not following Google’s own statements.

For other platforms, be careful. OpenAI, Perplexity, Anthropic and others have not published a commitment that they read llms.txt to rank or cite pages in their search products. Some developer tools and AI coding assistants do read these files, especially for documentation sites. For a normal business website, there is no confirmed benefit in ChatGPT or Perplexity answers.

What does an llms.txt file look like?

The proposal describes a simple structure. It starts with a title, followed by a short summary, then sections with lists of links and a brief note about each link. Here is the layout in words:

  • A single top-level heading with the name of the site or business.
  • A short blockquote summarizing what the site is about.
  • Optional paragraphs with key context.
  • Sections with headings, each containing a list of links with short descriptions.
  • An optional section for lower-priority links.

Here is a small example for a fictional business:

# Example Plumbing Co.

> Licensed residential plumber serving the Denver area. Services include repairs, water heaters and drain cleaning.

## Services
- [Water heater repair](https://example.com/water-heaters/): Repair and replacement options
- [Drain cleaning](https://example.com/drains/): Clogs, camera inspection and clearing

## About
- [About us](https://example.com/about/): Who we are and service area

Notice what it is not. It is not a list of every URL on the site. It is a curated map of the pages you most want understood.

How is llms.txt different from robots.txt and sitemap.xml?

Each file does a different job. Mixing them up causes confusion, so here is a quick comparison.

FilePurposeStatus
robots.txtTells crawlers what they may or may not fetchLong-standing standard, widely honored
sitemap.xmlLists URLs you want search engines to discoverWidely supported by search engines
llms.txtOffers a curated, readable summary for AI toolsProposal, no confirmed use by Google

The important one for AI access is robots.txt. If you want to control which AI crawlers can fetch your pages, that is where the rules live. You can read how Google handles it in its robots.txt introduction. An llms.txt file cannot block or allow anything. It only suggests where to look.

Do you need an llms.txt file?

For most businesses, no. It is optional, low effort and unproven. If your goal is more visibility in Google, ChatGPT or Perplexity, your time is better spent on content, technical health and authority. If you have spare time and a clean site, creating one is harmless.

It makes more sense in a few cases:

  • You run a documentation site or developer product, where AI coding tools may read it.
  • You want a tidy, human-readable index of your key pages for any agent or person who looks.
  • You manage a large site and want to practice curating its most important pages.

It makes less sense if you plan to treat it as a ranking tactic. The evidence does not support that, and it may distract from work that does.

How do you create an llms.txt file step by step?

If you decide to add one, keep it simple and accurate. Here is the process we use.

  1. Pick your key pages. Choose 10 to 30 pages that best describe the business: main services, about, contact, pricing and your strongest guides.
  2. Write a plain summary. In two or three sentences, say who you are, what you do, who you serve and where. Keep it factual and consistent with your About page.
  3. Group the links. Organize them under clear headings such as Services, Guides and Company.
  4. Describe each link. Add a short, specific note after each URL. Do not copy meta descriptions blindly.
  5. Save as llms.txt. Create a plain text file with Markdown content and upload it to the root of your domain.
  6. Check that it loads. Visit yourdomain.com/llms.txt and confirm it returns a normal page, not an error or a redirect loop.
  7. Keep it current. Update it when pages are added, moved or removed.

Should you let a plugin generate it?

You can, but check the output. Some WordPress SEO plugins can generate an llms.txt automatically. Automatic files often list every post, page and product, including cart, checkout and account pages that have no value to an AI tool. That defeats the point of a curated file.

When we reviewed an auto-generated file on our own site, it contained well over a hundred links and no description of the company. The fix was to curate it by hand: remove utility pages, add a clear company summary and keep only the pages that matter. If you use a plugin, treat its output as a first draft.

What matters more than llms.txt for AI visibility?

The fundamentals do the real work. If you want to be cited in AI answers, the following matter far more than any single file:

  • Letting the right crawlers in, through robots.txt and your firewall.
  • Pages that answer specific questions directly and clearly.
  • Facts you can source, with a named author and real business details.
  • Consistent information about your business across the web.
  • Technical health: fast pages, clean HTML, working internal links.

Our generative engine optimization service focuses on exactly these factors. If you want to know what is holding your site back first, an SEO audit is a good start.

Can llms.txt cause any harm?

Rarely, but there are small risks. A file that lists private or outdated URLs can point people to the wrong place. A file that makes claims you cannot support creates a trust problem if anyone reads it. And a badly configured server might return errors for the file, which is untidy.

It will not cause a Google penalty simply because it exists. Still, do not stuff it with keywords or promotional claims. Write it the way you would write a short, honest company overview.

How should you talk about llms.txt with an agency?

Ask for evidence. If an agency says llms.txt will boost your AI rankings, ask which platform confirmed it reads the file, and where. As of Google’s current documentation, the answer for Google is that it does not. A trustworthy provider will tell you it is optional and low priority, and will show you what else it plans to do.

That is how we approach it. We are happy to create a clean llms.txt as a small extra, but we will not sell it as the reason your traffic grows.

How can you check whether AI tools are visiting your site?

Your server logs record every request, including those from AI crawlers. Look for user agents such as OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and Googlebot. If you see requests for /llms.txt from a particular bot, that tells you something real. If you see none, the file is likely unused by that bot.

Many hosts give you access to raw access logs, and some CDNs show bot traffic in a dashboard. Check a few weeks of data rather than a single day. This is the honest way to learn whether anything is reading the file on your site, rather than relying on claims from articles, including this one.

Also look at which pages bots fetch most. If an AI crawler is hitting your guides and service pages, your content is being read. If it is hitting error pages or being blocked, that is the problem to fix first. Our technical SEO team does this kind of log and crawl review.

What is the difference between llms.txt and llms-full.txt?

The proposal also describes an optional companion file, llms-full.txt, that holds the full text of key pages in a single Markdown document, so a tool can load everything in one request. This is more relevant to documentation sites than to a typical business site.

The same caution applies. There is no confirmed ranking benefit from either file, and Google has said it does not use them. If you publish a full text file, make sure it is accurate and does not expose anything you would not publish on your public pages. Keep it in sync with your site, or leave it out.

The bottom line is simple. Treat both files as optional extras. Do the fundamentals first, and add the files later if you have time and a clear reason.

Is it worth the time for a small business?

Probably only a little. Creating the file takes under an hour if you already know your key pages. The risk is low, but so is the confirmed reward. If you have a backlog of real SEO work, such as thin service pages, slow templates or missing author details, do that first. Those changes help every search system, and they have clear evidence behind them.

Once the basics are strong, a clean llms.txt becomes a reasonable finishing touch. Treat it like a tidy sign on the door: harmless, helpful to anyone who looks, and not the reason customers arrive. Revisit the question whenever Google or another major platform publishes new guidance.

Frequently asked questions

Is llms.txt an official standard?

No. It is a community proposal published at llmstxt.org. No standards body governs it, and no search engine is required to support it. Each AI company decides whether to read it, so treat it as optional.

Does ChatGPT or Perplexity read llms.txt?

Neither company has published a commitment that llms.txt affects how it finds or cites sites. Their documentation focuses on crawlers and robots.txt controls. Check the latest official pages from each company, because policies can change.

Where do I put the llms.txt file?

Place it at the root of your domain so it loads at yourdomain.com/llms.txt. Use plain text with Markdown formatting. Confirm it returns a normal page and that it is not blocked by your firewall or caching rules.

Will llms.txt block AI bots from my site?

No. It cannot block or allow anything. To control which bots may crawl your pages, use robots.txt and your server or firewall settings. The llms.txt file is only a suggestion about where to look.

Should I add llms.txt to my WordPress site?

It is optional. If you add one, curate it by hand rather than listing every page. Keep it short, accurate and consistent with your About page, and update it when your site changes.

If you want to improve how AI tools and search engines understand your business, start with a review of the basics. See our SEO packages or request a quote and we will look at your site with you.

Own-ur-Rehman Sheikh

Own-ur-Rehman is the Founder and CEO of Rank Ray, where he leads the charge in technical SEO, digital growth strategies, and generative engine optimization. A self-proclaimed tech nerd and lifelong gamer, he spends his time breaking down complex search algorithms, building autonomous AI agent workflows, and exploring emerging tech. When he isn't engineering scalable digital marketing systems or diving into code, you’ll find him testing out the latest tech hardware or immersed in an open-world RPG.

About the founder

Recent Articles

Get a Free Digital Marketing Proposal for Your Business