AI Access Audit
How crawler policy and access infrastructure affect retrieval of your website
Declared policy and observed access disagree
The origin-root robots policy permits some discovery identities and disallows others. Separately, Digital Dominator’s identified audit crawler received an edge challenge. The report does not attribute that response to OpenAI, Anthropic, Perplexity or any other provider.
Declared policy
SELECTIVE12 identities DISALLOWED; 9 ALLOWED at path /.
Observed DD access
CHALLENGE OBSERVEDThe exact requested URL, final URL, status and retained response signals are recorded separately.
What this Tier 1 result does—and does not—say
Robots.txt was retrieved from the origin and its rules were evaluated against the root path (/). If a subpath was submitted, it is retained as provenance only; the submitted page was not fetched or assessed.
This audit establishes declared policy for the frozen panel and the response received by our own identified crawler. Access is a prerequisite for retrieval and citation, not a guarantee. Where evidence cannot establish a state, the report says COULD_NOT_ASSESS rather than converting uncertainty into a clean result.
Selected identities from the 21-identity panel
| Identity | Purpose | State at / | Basis |
|---|---|---|---|
| OAI-SearchBot | Search/discovery | ALLOWED | Applicable group permits root |
| GPTBot | Training-associated | DISALLOWED | Matched Disallow directive |
| Claude-SearchBot | Search/discovery | DISALLOWED | Matched Disallow directive |
| Claude-User | User-triggered | ALLOWED | Applicable group permits root |
| Google-Extended | Data-use control | DISALLOWED | Control directive; not an HTTP crawler |
| Applebot | Search/discovery | ALLOWED | Applicable group permits root |
Six representative rows shown. A paid report includes all 21 identities, their classifications, evaluated path and retained evidence basis.
Every finding points back to an observation
One problem, made straightforward for a developer
Why it matters: selected search/discovery identities are declared disallowed, while the separate DD observation shows an infrastructure challenge. These are different controls and require different checks.
Owner: website developer or infrastructure administrator.
Automation gathers evidence; a person signs off the report
Before delivery, a human reviewer checks target attribution, scope, state vocabulary, evidence links, contradictions and overclaims. Confirmed findings remain distinct from assessment limits. Paid reports are delivered within five business days.
Need the complete answer for your domain?
Start with the focused AI Access Check. If the evidence points to deeper CDN, WAF, TLS or redirect behaviour, the Full AI Access Audit expands the infrastructure investigation.
Request an audit →