We scanned the websites of 541 AI tools in November 2026. 79.7% publish an llms.txt file, 92.1% carry structured data, and only 0.4% block ChatGPT’s search bot. Most AI tool sites are technically open to AI search. The gaps are outside mentions and product schema, not crawler access.
Key findings
- 79.7% of AI tool sites have an llms.txt file. Ahrefs found 28% across a general web sample.
- 7.0% block at least one of six major AI bots. 0.4% block OAI-SearchBot and 0.4% block PerplexityBot.
- Only 34.6% name an AI bot in robots.txt at all, so most sites rely on defaults.
- 61.7% mark up the product as software. Just 17.6% use Product schema.
- Only 21.6% show a “featured on” section, the weakest signal in the scan.
How did we run this study?
The sample is every AI tool listed in the AI SEO Gazette tools directory that has a website link. That was 553 sites. On 21 November 2026, 541 answered with a normal homepage and 12 did not. All percentages below use the 541.
The script sent one request per second to each site and identified itself as AISEOGazetteResearch/1.0. For each site it fetched three files: the homepage, robots.txt and llms.txt.
- robots.txt: we tested whether each of six bots may fetch the homepage: OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended.
- llms.txt: counted only when the server returned a non-HTML file that starts with a markdown heading. Pages that return a styled 404 as HTTP 200 do not count.
- Schema: JSON-LD types read from the homepage HTML.
- Other checks: title, meta description, viewport, HTTPS, a pricing link, a blog link and “featured on” style text.
We ran the analysis twice on the saved data and got identical numbers. Limits you should know about: these are tools that chose to list in a directory, so they skew young and AI-aware. We checked homepages only, and we measured presence, not quality. We did not test what ChatGPT or Perplexity say about these tools. A robots.txt rule shows intent, not proof that a bot was stopped.
How many AI tool sites publish llms.txt?
79.7% do. That is nearly three times the rate Ahrefs measured in its 2026 study of 137,210 domains, where 28% published the file. Ahrefs also noted its sample skews technical, so treat 28% as an upper bound for the whole web.
Share of sites with a valid llms.txt file
AI tool sites in this scan against a general web sample
Source: AI SEO Gazette scan, 21 November 2026. Percentages are of reachable sites. Ahrefs figure: 137,210 domains using Ahrefs Web Analytics.
High adoption does not prove it works. The same Ahrefs study found that 97% of llms.txt files received no requests in May 2026. Google’s guide to generative AI features says you do not need llms.txt or other special files to appear in AI Overviews or AI Mode. AI tool builders add the file because it is cheap, not because it is proven.
Do AI tool sites block AI crawlers?
Rarely. 7.0% of sites block at least one of the six bots, and nearly all of them block training or opt-out bots, not search bots.
Share of sites whose robots.txt blocks each bot
Lower is more open to AI search
Source: AI SEO Gazette scan, 21 November 2026. Percentages are of reachable sites.
The difference matters. OpenAI documents that OAI-SearchBot surfaces sites in ChatGPT search, while GPTBot collects training data and does not affect search appearance. Blocking GPTBot is a content-training choice. Blocking OAI-SearchBot removes you from ChatGPT search answers. In this scan, 6.7% block a training or opt-out bot while leaving both search bots open. That is a reasonable setup.
One more detail: only 34.6% of robots.txt files name any of the six bots. Most sites never decided. They are open by default.
How common is structured data on AI tool homepages?
92.1% of homepages carry some JSON-LD. 80.0% use Organization, 61.7% mark the product as a software application and 60.8% include FAQPage. Only 17.6% use Product.
Share of homepages with each JSON-LD schema type
Homepage only
Source: AI SEO Gazette scan, 21 November 2026. Percentages are of reachable sites.
Google says special markup is not required for its AI features. Schema still helps search engines read what a page is, and 7.9% of the sites in this scan have none at all. Our AI search and GEO tool category lists tools that audit this kind of gap.
What about pricing pages, blogs and outside mentions?
Basics are in place. 99.4% have a meta description (median length 153 characters) and 81.0% link to a pricing page. Nearly half have a blog link. Only 21.6% show a “featured on”, “as seen on” or “where we are listed” section.
Share of homepages with each trust signal
Text and link match on the homepage
Source: AI SEO Gazette scan, 21 November 2026. Percentages are of reachable sites.
That last number is the opportunity. In Ahrefs’ May 2025 study of 75,000 brands, web mentions correlated with AI Overview visibility at 0.664, against 0.218 for backlinks. That is a correlation, not proof of cause. Mentions come from other sites, so a tool owner cannot fix this with a settings change. A good first step is a “Where we are listed” page that shows every directory and publication that features you.
What does the same scan say about AI SEO Gazette?
We ran the identical scan on aiseogazette.com. It is a publication, not a software product, so software schema and a pricing page do not apply.
| Check | aiseogazette.com |
|---|---|
| robots.txt allows all six bots | Pass |
| llms.txt present | Pass |
| Organization and FAQPage schema | Pass |
| Meta description (139 characters) | Pass |
| Software schema | Not applicable |
| Pricing page link | Not applicable |
| “Featured on” section | Pass |
AI visibility scorecard for tool owners
Use this table to check your own site. The third column shows how many AI tool sites in the scan already pass.
| Check | What passing looks like | Sites passing |
|---|---|---|
| Search bots allowed | robots.txt does not disallow OAI-SearchBot or PerplexityBot | 99.6% |
| Meta description | One clear sentence, about 150 characters | 99.4% |
| Pricing page | A linked page with plain prices | 81.0% |
| llms.txt (optional) | A markdown file at /llms.txt, not an HTML error page | 79.7% |
| Software schema | SoftwareApplication or WebApplication JSON-LD | 61.7% |
| Outside mentions | A page listing where your tool is featured | 21.6% |
What should AI tool owners do next?
- Check robots.txt and make sure OAI-SearchBot and PerplexityBot are allowed. Block GPTBot only if you want to opt out of training.
- Add SoftwareApplication schema to your homepage and one clear sentence about what the tool does.
- Publish a page that lists where you are featured, and start with a free listing in the directory.
- Run our free AI search readiness checker, then a deeper audit. Tools such as Beket AI and SEOnautik check how AI search sees a site.
- Read the rest of our guides for tool owners and Google’s own position in our breakdown of its unified SEO, AEO and GEO approach.
Cite this study
AI SEO Gazette (2026). AI Tool Visibility Study: how 541 AI tool websites prepare for AI search. https://aiseogazette.com/ai-tool-visibility-study/
The charts and numbers are free to use with credit and a link to this page. Download the data table (CSV).
Frequently asked questions
How many AI tool sites have an llms.txt file?
79.7% of the 541 AI tool sites we scanned on 21 November 2026. Ahrefs found 28% across a general sample of 137,210 domains in May 2026.
Should I block GPTBot?
Only if you do not want your content used to train OpenAI models. OpenAI says GPTBot does not affect search appearance. OAI-SearchBot is the bot that matters for ChatGPT search.
Do I need llms.txt to appear in AI answers?
Google says no for AI Overviews and AI Mode. Ahrefs found that 97% of llms.txt files got no requests in May 2026. It is cheap to add but not proven to help.
Which sites did you include?
Every AI tool listed in the AI SEO Gazette directory with a website link. We publish aggregate numbers only and do not name individual sites.
How often will you update this study?
We plan to rerun the same scan every six months and show what changed.
Sources
- Ahrefs, “97% of llms.txt Files Never Get Read (137,000 Sites Analyzed)”, 2026. ahrefs.com/blog/llmstxt-study
- Google Search Central, “Optimizing your website for generative AI features on Google Search”, May 2026. developers.google.com
- OpenAI, “Overview of OpenAI crawlers”. developers.openai.com/api/docs/bots
- Ahrefs, “An Analysis of AI Overview Brand Visibility Factors (75K Brands Studied)”, May 2025. ahrefs.com/blog/ai-overview-brand-correlation
- AI SEO Gazette scan data, 21 November 2026 (this study).
Related guide: how to check your LLM visibility.
Related guide: AI search statistics and the AEO checklist.
Part of our SaaS SEO guide.
