Agent permissions and your data: llms.txt, Web Bot Auth, and letting agents act safely

When you need it

How to implement it

1. 1

   Keep llms.txt scoped to what it actually governs

   llms.txt, including this site's own /llms.txt referenced on the AI-crawler integration guide, tells a crawler which content is meant for retrieval or training. It says nothing about whether a different kind of request, an agent trying to submit a form or complete a checkout, should be allowed to act. Don't extend llms.txt's syntax to try to express action permissions; it wasn't built for that, and a crawler-focused parser won't apply it that way regardless.

2. 2

   Add request-identity verification for anything beyond serving a page

   A user-agent string is trivially spoofed - the exact debugging problem this site's sibling page on spoofed agent signatures covers. If a verified agent gets to skip a friction step (a form, a rate limit) that a human or unverified bot still faces, that decision needs an actual cryptographic signal, not a string match. Web Bot Auth is the current candidate, still an individual IETF Internet-Draft as of August 2026 even with a working group in place and real production deployments at Cloudflare, AWS WAF, and Akamai.

   // conceptual - verifying a signed request per the Web Bot Auth draft.
   // Exact header names and the verification flow are still moving; confirm
   // against the current draft and your edge/CDN vendor's implementation
   // before shipping a hard dependency on it.
   if (verifyWebBotAuthSignature(request)) {
     // treat as a verified, identified agent - not the same as "authorized to act"
   }

3. 3

   Separate "verified identity" from "authorized to act" explicitly in your logic

   A signed Web Bot Auth request tells you which agent this claims to be, cryptographically - not what that agent's operator, or the customer behind it, has actually authorized. Don't collapse the two checks into one boolean. An agent can be genuinely who it says it is and still be attempting something nobody authorized.

4. 4

   For anything that reads as a transaction or a customer-authorized action, require the AP2-style authorization chain, not just identity

   See the companion guide on the AP2 mandate model for what that chain actually proves. The short version here: identity verification (Web Bot Auth) and action authorization (AP2 Mandates) are two separate gates, and an implementation that only covers one has left the other half of the permission question unanswered.

5. 5

   Route the "should we allow this at all" policy decision to privacy and security, not to whichever team ships the integration first

   Which agents get elevated access - skip a rate limit, complete a transaction, read gated content - is a policy decision with real exposure if it's wrong, reviewed alongside privacy and security, not a default that falls out of whichever SDK happened to be easiest to integrate.

How to verify it worked

Related

Elsewhere in the agentic-traffic cluster