llms.txt and AI-crawler access: giving agents a straight answer instead of a guess
What it is
llms.txt is a plain-text file, published at your domain root the same way robots.txt is, that tells AI agents and crawlers which content is meant for their consumption and where to find the clearest version of it. OneTrust's integration guide frames this as a governance question, not just a discoverability one: which content is fair game for third-party AI systems, and whether AI-agent traffic needs the same consent and logging treatment as human visitor traffic under your existing privacy program.
When you need it
You're publishing content you want AI answer engines and agents to read and cite accurately, and want to point them at the pages meant for that rather than letting them guess from a full crawl.
Your privacy or security team is asking whether AI-agent fetches of your pages should be logged or gated the same way other automated collection is under your existing consent architecture.
You want to keep certain content (behind-login pages, internal tools, anything not meant for public AI training or retrieval) explicitly out of scope without relying on robots.txt alone.
How to implement it
- 1
Decide what you actually want AI agents to read
Separate content meant for public retrieval (marketing pages, documentation, support content) from content that isn't (anything behind authentication, internal exports, anything you wouldn't want quoted back by an AI system).
- 2
Publish an llms.txt at your domain root listing the sanctioned entry points
Keep it a plain, short index - a link list with brief context, not a full site dump. This site's own /llms.txt is a working reference for the format.
- 3
Confirm your CMP correctly classifies known AI-agent user agents
A cookie-consent gate designed for human visitors can misclassify or block a bot's request outright - confirm known AI-agent user agents (the ones your own robots.ts / robots policy already documents allowing) aren't being served a cookie wall meant for people, and aren't being logged as a consenting individual either.
- 4
Route the logging question through your privacy/legal review, not just engineering
Whether AI-agent fetches need to appear in your data-processing records alongside other automated collection is a governance decision, reviewed alongside legal/privacy, not a default your CMS or consent platform picks for you.
- 5
Add machine-readable mirrors of the pages you want agents to read accurately
A markdown or plain-text mirror per page (this site generates one per route) gives an agent a clean, unambiguous version of the content instead of one it has to extract from rendered HTML.
How to verify it worked
Fetch /llms.txt directly in a browser or with curl and confirm it returns 200 and lists exactly the paths you intended - nothing you meant to exclude.
Check server logs for known AI-agent user agents and confirm their requests aren't being served a consent/cookie gate meant for human visitors.
Spot-check one of the markdown mirrors listed in llms.txt and confirm it matches the live page's actual current content, not a stale generated copy.
Related
Want this scoped against your specific OneTrust tenant?
We scope every engagement in discovery, before implementation - no assumptions about your stack.