Policy and infrastructure disagree
robots.txt declares one thing; the CDN, WAF or reverse proxy does another. Both are working as configured, and the combination was never reviewed as one system.
Enterprise · Multi-team · Remote delivery
Enterprise AI Crawler Access Audit.
A fixed-price diagnostic for Heads of Digital, CMOs, SEO leads and platform owners who need to know what AI crawlers actually meet at the edge — and hand engineering a change list the same week.
An enterprise AI crawler access audit grades your declared robots.txt policy against a frozen 21-identity panel of crawler, fetcher and control identities, then separately records what Digital Dominator’s identified audit crawler received across your CDN, WAF, reverse proxy and origin layers. You get evidence, the affected paths, and prioritised remediation written for a developer. It is a diagnostic, not a platform subscription.
Does your edge answer AI crawlers the way your robots.txt says it does?
Fixed price. AUD $495 / USD $349 AI Access Check, or the AUD $1,195 / USD $799 Full AI Access Audit for deeper CDN, WAF, TLS and redirect analysis and a re-check after your fixes.
Book your AI Access Audit ›
01Why enterprise sites need this separately
On a large site, nobody owns crawler access end to end. Marketing owns the content, an agency may own robots.txt, platform engineering owns the CDN, and security owns the WAF. Each team makes reasonable decisions inside its own boundary, and the combined behaviour is something no one specified and no one is watching.
That is why declared policy and observed behaviour drift apart on enterprise estates more than anywhere else. A robots.txt can invite an agent that a bot-management rule quietly challenges. A redirect chain added for a locale launch can change what a non-rendering fetcher receives. None of it appears in a dashboard, because it is not a content problem.
This audit is scoped to answer that question and stop. It is a fixed-price diagnostic with a defined deliverable, not another platform to buy, integrate and renew, and not a project to reinvent internally.
02What can go wrong
Ownership sits across teams
The person who can change robots.txt is not the person who can change an edge rule. Findings without a named layer and an exact change stall between teams.
Redirect and locale chains
Apex to www, HTTP to HTTPS, country or language redirects. Each hop is a chance for a non-rendering fetcher to receive something other than the page.
Inherited defaults
Template robots files and platform defaults carried through migrations, still making decisions nobody made deliberately.
Staging rules reaching production
Access restrictions intended for a preview environment that were promoted along with everything else.
Identity classes treated as one
Training crawlers, search and discovery crawlers, user-triggered fetchers and data-use controls restricted together, when the commercial consequences differ sharply.
03What the audit examines
Two separate evidence classes, never merged, so a reader can always tell what was declared from what was observed.
Declared policy, 21 identities
Your robots.txt graded across a frozen panel of crawler, fetcher and control identities, with the exact source line behind every verdict.
Observed access, our crawler
What Digital Dominator’s identified audit crawler received: status, redirects followed, response characteristics and timing.
Edge and origin layers
Where a difference appears between declared policy and observed response — CDN, WAF, reverse proxy or origin — reported as what was observed at each hop.
Affected paths
Whether a restriction touches primary public content or only secondary and operational paths. The commercial difference between the two is large.
Redirects and status codes
The chain as an identified automated client experiences it, including where a hop changes the response.
Machine readability
Structured data, canonical hygiene and what appears in server-rendered HTML, because that is what a non-rendering fetcher receives.
04What you receive
An executive verdict
One page a CMO or Head of Digital can act on, stating what is restricted, where, and what it plausibly affects.
A developer-ready change list
Exact robots.txt lines, header and status-code changes, and edge settings, each tied to the evidence it rests on and the layer that owns it.
A security handoff
Findings that belong to the WAF or bot-management layer, described so a security reviewer can assess them without a marketing translation.
Prioritised remediation
Ordered by likely commercial effect, not by count of findings, so the first three items are the ones worth doing.
A re-check
The Full AI Access Audit includes a free re-check after your fixes, so the change is verified rather than assumed.
Multi-domain and agency scope
Portfolios, multi-brand estates and agency engagements are quoted per domain with consolidated reporting. Enquire with the domain list.
05Fixed-price tiers
AI Access Check
AUD $495
includes Australian GST · USD $349
- Executive access verdict
- Frozen 21-identity declared-policy matrix
- Exact robots.txt evidence per finding
- Separate Digital Dominator crawler access observation
- Prioritised developer-ready remediation
- Human-reviewed before delivery
Recommended
Full AI Access Audit
AUD $1,195
includes Australian GST · USD $799
- Everything in the AI Access Check
- Deeper CDN, WAF, TLS and redirect analysis
- Expanded technical access analysis
- Structured data and machine-readability review
- Walkthrough call
- Free re-check after your fixes
Fix & Verify
Quoted
- We implement the agreed changes
- Bounded scope, quoted from your audit
- Re-tested and verified after changes
- Separate from any ongoing retainer
Enterprise, agency and multi-domain engagements are quoted on the domain list.
06What this evidence does not show
Access is necessary for retrieval and citation but does not guarantee either. We report what Digital Dominator’s identified audit crawler observed; that is evidence about our crawler and is never presented as proof of another provider’s crawler behaviour, and we do not request from a vendor network or impersonate a vendor crawler. Training crawlers, search and discovery crawlers, user-triggered fetchers and data-use controls are not interchangeable and are graded separately. llms.txt is optional and emerging, not a ranking requirement. On an enterprise estate one further limit matters: a single observation is a point in time from one vantage. An edge can answer two requesters differently, so we record what was requested, what came back and when, rather than generalising from it.
07Common questions
Is this a dashboard or a subscription?
Neither. It is a fixed-price diagnostic with a defined deliverable and an end. You receive evidence and a change list, and the engagement finishes. The Full AI Access Audit includes one re-check after your fixes.
Who inside our organisation is this for?
The executive verdict is written for a Head of Digital, CMO or SEO lead. The change list is written for the developer or platform engineer who will apply it, and findings that belong to the WAF or bot-management layer are separated so a security reviewer can assess them directly.
Can you tell us which vendor crawler is being blocked?
We report what your robots.txt declares for each identity in the panel, and separately what our own identified crawler received. We do not request from a vendor network and we never impersonate a vendor crawler, so we do not claim to observe another provider on your infrastructure.
Can you audit several domains or brands?
Yes. Multi-domain and agency engagements are quoted on the domain list, with consolidated reporting across the estate. Enquire with the domains you want covered.
08Related reading
For the wider programme this sits inside, read the enterprise AI visibility audit guide. For the underlying method, read how an AI access audit actually works. For the handoff itself, writing a developer brief that gets acted on covers how these change lists are structured. The central AI Access Audit page has pricing and the full method.
Does your edge answer AI crawlers the way your robots.txt says it does?
Start with the central AI Access Audit, or book directly below.
Book your AI Access Audit ›