llms.txt explained: should your website have one?

llms.txt is a proposed Markdown file placed at the root of a website to give AI tools a short, curated map of its most useful content. It was proposed in September 2024 and is not an adopted standard. Google has said Search ignores it, and studies of server logs and citations show little evidence that AI crawlers use it or that it improves visibility. It can still help developer documentation used by coding assistants.

What is llms.txt?

llms.txt is a plain Markdown file, served at yourdomain.com/llms.txt, that summarises a website and links to its most important pages. The idea is that a language model reading the site at the moment a user asks a question could use the file as a shortcut, instead of crawling and parsing many HTML pages. Some sites also publish an llms-full.txt containing the full text of key pages.

Jeremy Howard of Answer.AI proposed the format in September 2024. It is a community proposal, not a standard adopted by search engines or AI companies.

How is it different from robots.txt?

If you want to control how AI companies crawl your site, robots.txt is the tool that works. Our guide to getting your brand mentioned in ChatGPT explains the difference between OpenAI’s search and training crawlers.

What does Google say?

Google has been consistent. Gary Illyes said in July 2025 that Google does not support llms.txt and has no plans to, and John Mueller compared it to the old meta keywords tag. In June 2026 Google added a note to its guidance on generative AI features stating that you do not need new machine-readable files, AI text files or Markdown to appear in Google Search, including its AI features, as reported by Baseline Labs.

What does the data show?

A summary of 2026 studies by Geoly found AI crawlers rarely request the file, and SE Ranking’s study of about 300,000 domains found no relationship between having one and being cited by AI. Some providers’ crawlers do fetch it occasionally, but there is no evidence it changes what gets cited.

When is llms.txt still worth adding?

The strongest case is developer documentation. Coding assistants such as Claude Code, Cursor and Copilot can be pointed at an llms.txt file to load an API’s docs quickly, and several documentation platforms generate one automatically. If your audience includes developers, adding it is cheap and sensible. For most marketing and service sites, time is better spent on clear, crawlable pages.

Why did llms.txt get so much attention?

As AI search grew, site owners looked for a simple technical switch that would improve their visibility, much as robots.txt and sitemaps once did for search engines. llms.txt looked like that switch, and many tools and plugins started generating it automatically. The appeal is understandable: it is quick to add and feels proactive. The problem is that a file only matters if the systems you care about read it, and so far the major ones have either said they do not or show little sign of doing so in practice.

How do you create one, if you decide to?

Start with a title and summary: A heading with your organisation name and a one-line description.
Group your key pages: Short sections, such as Services, Guides or Docs, each with a list of links.
Describe each link: A few words on what each page covers.
Publish at the root: Serve it as plain text at /llms.txt.
Keep it current: Update it when key pages change, or generate it automatically.

Do: Keep it short, accurate and focused on your most useful pages.
Do: Use the same facts and names as your main pages.
Don’t: Expect it to improve rankings or AI citations.
Don’t: Use it instead of robots.txt for access control.

What should you do instead for AI visibility?

Make sure important pages are crawlable, indexed and fast
Answer questions directly under clear headings, as covered in our AEO guide
Add sources, statistics and named experts
Earn mentions on credible third-party sites
Track AI mentions and AI referral traffic monthly

These are the practices behind generative engine optimization, and they work regardless of which AI system reads your site.

For the full picture of preparing developer docs and APIs for AI agents, see how to make your product agent-ready.

How UIUX Studio helps

We audit sites for AI visibility and focus effort on the changes that evidence supports: structure, content quality and technical health. Talk to our product engineering team, or to Social Mafia by UIUX Studio for ongoing SEO.

Frequently asked questions

Is llms.txt an official standard?

No. It is a 2024 proposal that has not been adopted by major search engines or AI providers.

Does Google use llms.txt?

No. Google says Search, including its AI features, ignores the file.

Does llms.txt improve ChatGPT visibility?

There is no published evidence that it does.

Can llms.txt block AI crawlers?

No. Use robots.txt to control crawler access.

Should developer docs have one?

It can help, because coding assistants and agents may use it to load documentation.

Related articles

{{ p.category }}

{{ t }}

{{ p.title }}

{{ p.excerpt }}

U{{ author.name }} {{ p.readMins }} min read

{{ p.category }}

{{ p.title }}

{{ mo.sub }}

{{ m.tag }}

{{ m.t }}

{{ m.d }}

{{ m.left.t }}

✕{{ x.t }}

{{ m.right.t }}

✓{{ x.t }}

{{ m.big }}{{ m.label }}

{{ x.t }}

{{ m.label }}

✓{{ x.t }}

{{ x.t }}

{{ m.label }}
{{ x.t }}
{{ m.status }}

{{ m.label }}

{{ x.t }}{{ x.n }}

{{ x.t }}

{{ m.label }}{{ m.stepTxt }}

{{ x.t }}

{{ m.btn }}

{{ x.t }}

{{ m.q }}

{{ m.a }}

{{ x }}

{{ k.n }}

{{ k.v }}

{{ m.label }}

{{ m.status }}

{{ m.label }}

{{ st.n }}{{ st.t }}

{{ m.hub }}

{{ nd.t }}

AaAaAaAa

ButtonInput fieldTag

THE SHORT ANSWER

{{ p.answer }}

KEY TAKEAWAYS

✓{{ k }}

RELATED SERVICES

{{ b.t }}

{{ g.t }}{{ g.t }}

{{ g.t }}

{{ b.label }}:{{ b.t }}

{{ c.ini }}

{{ c.t }}

{{ c.d }}

{{ s.n }}

{{ s.t }}

{{ s.d }}

{{ h }}

{{ c.t }}

{{ b.alt }}

{{ g.alt }}

{{ b.title }}

{{ c.t }}

{{ c.s }}

{{ i }}

{{ c.t }}{{ c.s }}

{{ b.caption }}

{{ s.v }}

{{ s.l }}

{{ s.s }}

{{ b.caption }}

{{ b.title }}

{{ b.unit }}

{{ r.l }}{{ r.d }}

{{ b.caption }}

{{ b.title }}

A{{ b.a.t }}

{{ i }}

B{{ b.b.t }}

✓{{ i }}

{{ b.caption }}

{{ b.title }}

{{ t.d }}

{{ t.t }}

{{ b.caption }}

“

{{ b.t }}

{{ b.by }}

{{ b.title }}

{{ r.d }}

{{ r.l }}

{{ b.caption }}

{{ b.title }}

{{ m.n }}

{{ b.lo }}{{ b.hi }}

{{ m.n }}

{{ m.l }}

{{ m.d }}

{{ b.caption }}

{{ b.cs.client }} case study

Case study · {{ b.cs.client }} {{ b.cs.title }} {{ b.cs.text }}

{{ f.v }}{{ f.l }}

View case study →

SOURCES AND FURTHER READING

Frequently asked questions

{{ f.a }}

U

Written by {{ author.name }}

{{ author.role }}

Keep reading

More articles from across design, AI and marketing.

All articles

{{ r.category }} · {{ r.readMins }} min{{ r.title }}

For Ambitious Founders

Startup Strategy Session Discuss Your Project with Our Senior Experts

No Obligation
Expert Insights
Tailored Guidance