Does llms.txt Actually Work? What 290,000+ AI Bot Requests Show

We wrote the guide on adding llms.txt to WordPress back when the standard was picking up momentum, and imagewize.com has hosted one since. So when a client asked us to keep it automatically in sync — the way The SEO Framework keeps sitemap.xml current — we assumed that was the right next step. It sounded like an obvious parity feature: sitemap.xml stays live, llms.txt should too.

Before building it, we went looking for evidence that AI crawlers actually request the file. We didn’t find any — not in our own logs, and not in two other independent datasets that measure exactly this. Here’s what the data says, and what we did instead.

In This Article

• The Claim Behind llms.txt
• Three Independent Datasets, One Result
• What We Found in Our Own Logs
• So Who Is Requesting llms.txt?
• Is llms.txt Useless, Then?
• What We Did on imagewize.com
• Frequently Asked Questions

The Claim Behind llms.txt

llms.txt is a plain-text file at your site root — modeled on robots.txt — that’s supposed to give AI assistants a structured summary of what your site covers, so they can understand it without crawling every page. The pitch, repeated across dozens of WordPress plugins and SEO guides (including our own), is that adding one helps AI systems find, understand, and cite your content faster.

That’s a reasonable hypothesis. It’s also, as far as we can tell from real traffic data, not happening yet.

Three Independent Datasets, One Result

The most detailed public data comes from an unexpected place: a GitHub feature request asking The SEO Framework (a major WordPress SEO plugin) to add llms.txt support. Instead of shipping it on faith, the plugin’s author pulled real numbers from The SEO Framework’s own network, and a Roots.io contributor followed up with their own site’s logs. We added a third data point by pulling imagewize.com’s own logs the same way. Here’s how the three compare:

Source	Period	Total AI-bot hits	Hits to /llms.txt from citation crawlers
The SEO Framework network	~5 months	180,219	0
Roots.io	57 days	79,947	0
imagewize.com	~7 weeks	28,987	0
Total		289,153	0

“Citation crawlers” here means the bots that would actually matter for AI recommendations: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Amazonbot, and meta-externalagent — the crawlers behind ChatGPT, Claude, Perplexity, Amazon’s Rufus, and Meta AI. Not one of them, across nearly 290,000 combined requests on three different WordPress-adjacent sites, requested /llms.txt.

What We Found in Our Own Logs

We pulled seven weeks of imagewize.com’s own Nginx access logs (late June through mid-August) to check this ourselves rather than take the other two datasets on faith. The first pass actually gave a false positive worth mentioning: an unescaped grep 'llms.txt' matches the “.” as “any character,” so it also matched every crawl of /how-to-add-llms-txt-to-wordpress/ — our own guide about the file, which gets crawled heavily. That inflated the count to over 1,700 hits. Once corrected to match the literal path (/llms\.txt), the real number was 25 requests in seven weeks.

Of those 25, none came from GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, or Amazonbot — all of which are active and heavy crawlers on imagewize.com, generating thousands of requests each to our actual pages over the same window, as we’ve tracked since our first AI-bot traffic analysis. Only one AI-related bot touched /llms.txt at all: CCBot, the crawler behind Common Crawl, which is a training-data archive, not a live citation source. Everything else in that list of 25 was curl (likely our own testing), ordinary browsers, and marketing/SEO tools — Semrush’s site-audit bot, Dataprovider.com, and a couple of niche crawlers. One hit was even our own site-auditor tool scanning imagewize.com during development.

So Who Is Requesting llms.txt?

Across all three datasets, the requests that do land on /llms.txt come from a consistent, narrow set of visitors — and it’s the same handful of categories every time:

• “AI readiness” scanners — tools like AgentReadinessScanner and LLMS-Txt-Scanner that exist specifically to check whether a site has adopted the standard, not to read its content.
• SEO and marketing data tools — Dataprovider.com and Semrush’s SiteAuditBot showed up independently in both Roots.io’s data and ours, cataloguing what exists on a domain.
• Curious humans — a meaningful share in every dataset is a plain browser user-agent, someone manually checking whether a site has one (developers, competitors, or the site owners themselves).
• Spoofed or opportunistic bots — The SEO Framework’s data flagged a spoofed Googlebot/2.1 hitting only llms.txt from a non-Google IP, alongside generic security/recon scanners.

None of these are the AI shopping assistants or search products the standard was built for.

Is llms.txt Useless, Then?

Not necessarily — but the honest answer today is “not yet, and maybe not the way it was pitched.” Two credible people in the WordPress SEO space landed on opposite conclusions from the same data, and both arguments are worth hearing.

The SEO Framework’s author argues the plugin shouldn’t ship a feature that visibly does nothing yet, and that plugins already offering it risk misleading users about what it actually accomplishes today. Yoast founder Joost de Valk pushed back in the same thread: with an AI coding agent, he added llms.txt support to two of his own products and a blog post in roughly 10–15 minutes each. His argument is that the cost of shipping is so low it’s worth doing regardless of proof, and you can always remove it later if the data stays flat.

We land closer to “cheap to have, not worth engineering around.” A static file costs nothing to keep. Building infrastructure to keep it dynamically fresh — the original ask that sent us looking at this data — is a different calculation, because that’s real engineering time spent on a channel with zero measured traffic from the audience it’s meant to serve.

What We Did on imagewize.com

Three concrete changes came out of this:

• Kept the static file, skipped the automation. We fixed the one stale fact it had (an out-of-date blog post count) and redeployed. We’re not building dynamic generation or a scheduled regeneration job — the data doesn’t justify the engineering time right now.
• Adjusted our own site-auditor tool. Our WooCommerce store checkup grades stores on AI-shopping readiness and includes an llms.txt check. It was already weighted lightly (5 out of 100 possible points, and never a blocking factor) — but the copy explaining a missing file implied adding one would help AI assistants find and recommend the store. We rewrote it to say what the data actually shows: cheap to add, low priority, not a proven citation lever.
• Corrected our own earlier guide. Our original llms.txt how-to guide claimed AI crawlers “actively look for it.” This data said otherwise, so we rewrote that sentence to match the evidence.

If you’re deciding what to prioritize for AI visibility on your own WordPress or WooCommerce site, the data points elsewhere: making sure AI crawlers aren’t blocked in robots.txt, serving content that renders without JavaScript, and adding real structured data all show measurable crawler activity today. If getting recommended by AI shopping assistants specifically is the goal, we’ve covered what that actually takes in more depth. llms.txt is a cheap five minutes, not a strategy.

Frequently Asked Questions

• Do AI crawlers like ChatGPT and Claude actually check for llms.txt? Not based on the data available today. Across three independent datasets covering nearly 290,000 combined AI-bot requests — The SEO Framework’s plugin network, Roots.io, and imagewize.com — zero requests to /llms.txt came from a citation-focused crawler like GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, or Amazonbot.
• Should I still add an llms.txt file to my WordPress site? It’s cheap to add and there’s no downside, so it’s reasonable to have one. Just don’t expect it to move AI citations or traffic today, and don’t spend engineering time keeping it automatically in sync the way you would a sitemap — the audience it’s meant for isn’t checking it yet.
• Is llms.txt an SEO ranking factor? No. It’s unrelated to search engine ranking and isn’t equivalent to robots.txt or sitemap.xml, both of which have decades of proven crawler behavior behind them. llms.txt is a proposed standard that, so far, has no measured adoption by the AI crawlers it targets.
• What should I prioritize instead of llms.txt for AI visibility? Confirm your robots.txt doesn’t block AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Amazonbot, and similar), make sure your content renders without relying on JavaScript, and add real structured data — Product, Organization, and FAQ schema — to the pages you want AI systems to understand and cite. All three show measurable AI-crawler activity today; llms.txt doesn’t yet.