Original research, fielded 2026-08-09
The AI Discovery Technical Hygiene Benchmark 2026
We crawled 160 live software company websites and scored the 10 technical hygiene signals AI crawlers and parsers depend on. Median score: 7 out of 10. Only 8 percent publish FAQ schema. This study measures access and legibility on a company's own domain, not whether assistants cite it. Methodology and raw data below, free to cite.
160
Software sites analyzed
175 attempted, 15 excluded
7/10
Median technical hygiene score
Mean 6.33
52%
Serve a real llms.txt
Verified as plain text, not a soft 404
81%
Have no AI crawler policy
Zero of 11 AI agents named in robots.txt
The short answer
Software websites are well optimized for humans and poorly optimized for the machines that now summarize them. Across 160 live sites, the median technical hygiene score was 7 out of 10, 34 percent scored 8 or higher, and 18 percent scored 4 or lower. The gap is concentrated in structured data and explicit crawler policy, which are the two cheapest things on the list to fix.
- Classic on-page hygiene is largely solved: 95% have a meta description and 90% a canonical tag
- Machine-facing hygiene is not: 38% publish no JSON-LD and only 8% publish FAQPage schema
- 81% of sites do not address a single AI crawler in robots.txt, and only 5% block any
- Hygiene tracks category maturity, from a mean of 7.1 in B2B horizontal SaaS to 4.5 in healthtech
The data
What we found
Every figure is the share of the 160 analyzed sites where the signal was present in the raw HTTP response on 2026-08-09.
Technical hygiene score out of 10 on the horizontal axis, number of sites on the vertical axis. n = 160.
Technical hygiene by category
| Category | Sites | Mean score | llms.txt | JSON-LD | FAQPage |
|---|---|---|---|---|---|
| B2B horizontal SaaS | 36 | 7.1 | 67% | 83% | 11% |
| Developer tools and infrastructure | 39 | 6.7 | 67% | 72% | 3% |
| Fintech and payments | 27 | 6.6 | 59% | 63% | 11% |
| Martech and analytics | 19 | 6.6 | 53% | 58% | 5% |
| Consumer software | 26 | 5.1 | 27% | 38% | 15% |
| Healthtech | 13 | 4.5 | 8% | 31% | 0% |
Median homepage HTML weight was 446.1 KB, mean 556.8 KB, 90th percentile 1116.6 KB. Median title length was 53 characters and median meta description 148 characters.
Analysis
Six findings that matter
Structured data is the biggest single gap
38 percent of software homepages carry no JSON-LD at all, and only 8 percent publish FAQPage schema. Structured data removes ambiguity for any machine parsing the page and makes a site eligible for certain search features. Google states that no special AI schema is required, so we do not present it as a citation lever. We present it as the cheapest way to stop asking machines to infer facts you could simply declare.
llms.txt adoption is widespread and unproven
52 percent of the full sample serves a real llms.txt, with a stark split: 67 percent in B2B horizontal SaaS and developer tools versus 27 percent in consumer software and 8 percent in healthtech. Adoption is not evidence of effect. Google has said publicly that it does not use llms.txt, and no major provider documents it as a ranking or citation input, so the honest reading is that teams in AI-influenced categories are hedging cheaply. Treat it as an experiment to measure, not a foundation.
Almost nobody has an explicit AI crawler policy
81 percent of sites do not mention a single one of the 11 AI crawlers we checked in robots.txt, and only 5 percent block any of them. Non-blocking is the right default for a company that wants to be cited, but silence is not a decision. Teams inherit whatever each vendor's default behavior is, then discover the consequences after the fact.
Human-facing hygiene is solved, machine-facing hygiene is not
The human-facing basics are nearly universal: 95 percent have a meta description, 92 percent a valid robots.txt, 90 percent a canonical. The machine-facing layer collapses right after that: 56 percent Organization schema, 17 percent product schema, 14 percent breadcrumbs. Two decades of SEO discipline produced pages that read well and describe themselves poorly.
Homepages are heavy enough to hurt retrieval
Median homepage HTML is 446 KB, mean 557 KB, and the 90th percentile clears 1.1 MB. Heavy client-rendered shells raise the odds that a crawler with a modest budget or no JavaScript execution sees a fraction of what a browser does. Content that never reaches the parser cannot be cited.
Category maturity, not company size, predicts hygiene
Mean hygiene score runs 7.1 in B2B horizontal SaaS down to 4.5 in healthtech, and several of the lowest scores in the sample belong to very large, very well-funded companies. Hygiene tracks whether a category's buyers already research through AI assistants, not headcount or revenue.
Methodology
How the study was run
Published in full so the results can be audited, reproduced, or argued with.
Collection
- Fielded on 2026-08-09 from a single crawler identifying itself as MomentenceBenchmarkBot.
- 175 real, currently operating B2B software and SaaS domains were attempted across 6 categories. 160 returned a usable homepage response and 15 were excluded for timeouts, blocks, or non-200 responses.
- For each domain we fetched the homepage, /robots.txt, /llms.txt, and /sitemap.xml.
- We parsed the raw HTML response and did not execute JavaScript, because retrieval crawlers frequently do not either.
- robots.txt was parsed into user-agent groups, and each of 11 AI agents was classified as blocked by a blanket disallow, explicitly named and allowed, or not addressed.
- llms.txt results were verified a second time to reject soft 404s, HTML fallbacks, and empty stub files. Three apparent hits failed verification and were reclassified as absent.
Scoring
One point per signal present, out of ten, unweighted:
- 01Any valid JSON-LD block on the homepage
- 02Organization or equivalent entity schema
- 03FAQPage schema
- 04Product, Service, or SoftwareApplication schema
- 05llms.txt served as plain text at the root (included to measure adoption; Google has stated it does not use the file, so read this component as a habit signal, not a requirement)
- 06sitemap.xml resolving to a valid urlset or sitemapindex
- 07Meta description present
- 08Exactly one H1 element
- 09Canonical tag present
- 10No blanket disallow against any of the 11 AI crawlers checked
Limitations, stated plainly
This is a purposive judgment sample of recognizable software companies, not a random sample of the software industry, so the figures describe the visible end of the market and are not statistically projectable to all software companies. Only the homepage was scored, so a site with excellent schema on its product and comparison pages can score low here. The crawl is a single point in time and reflects the response served to one non-browser user agent. Most importantly, technical hygiene is a precondition for being used as a source, not proof of citation: nothing in this dataset measures whether an assistant cited, mentioned, or accurately described anyone. Measuring that requires querying the assistants directly against a fixed prompt set. llms.txt is included as an adoption measure only; Google has stated it does not use the file, so its presence in the score should be read as a habit signal rather than a requirement.
Data, licence, and citation
The full per-domain dataset, including every signal and the computed score, is published as CSV. It is free to reuse, quote, and republish under CC BY 4.0 with attribution to Momentence and a link to this page.
Momentence. AI Discovery Technical Hygiene Benchmark 2026. Fielded 2026-08-09. https://www.momentence.com/reports/ai-discovery-technical-hygiene-benchmark
What to do about it
Five ways teams get this wrong
- Treating llms.txt as an AI discovery program. Google has said it does not use the file, and no provider documents it as an input. Publish it as an experiment if you like, then measure rather than assume.
- Publishing FAQPage schema that does not match visible page content. Mismatched markup is a trust liability, not a shortcut.
- Blocking training crawlers and retrieval crawlers with the same rule. GPTBot and OAI-SearchBot do different jobs, and blanket blocks quietly remove you from answers.
- Shipping entity schema on the homepage only. Product, pricing, and comparison pages are where vendor questions actually get answered.
- Auditing a rendered browser DOM instead of the raw HTML response. The two differ, and crawlers see the response.
If you want the fix list rather than the failure list, the definitive SEO, AEO, and GEO guide has the checklists, and answer engine optimization is where this work sits inside a Momentence program. Our methodology page shows the month-by-month sequence we run it in.
FAQ
Questions about this benchmark
Score your own site against the benchmark
We will run the same ten-signal audit on your domain, send you the raw output alongside the benchmark medians, and tell you which two fixes actually move you. No obligation.