Know what is blocking AI access, give your developer the fix, and move on—without buying another dashboard or reinventing the audit internally. Digital Dominator separates declared crawler policy from observed access, identifies the technical barriers and provides a prioritised remediation brief your team can act on.
The AI Access Check evaluates declared policy across a frozen 21-identity crawler, fetcher and control panel, then records what our own identified audit crawler actually received. We never present our crawler’s response as proof about another provider’s crawler.
Built for Heads of Digital, CMOs, SEO leads and the developers responsible for implementation. Every paid report is human-reviewed and delivered within five business days. Access is a prerequisite for retrieval and citation, never a guarantee of either.
More customers are now asking AI assistants directly for recommendations, comparisons and answers instead of relying only on a traditional search results page. The assistant answers by citing a handful of sites it can access and trust. If your site restricts the search and discovery agents - either in declared policy or at the edge - it can lose direct retrieval and citation opportunities in that channel.
"Why isn't my website showing in ChatGPT?" can be an access question before it is a content question. Ranking well in Google does not automatically carry over: robots directives, firewall rules and CDN defaults are set separately, and any of them can create a barrier to discovery or retrieval. Where a search and discovery agent is restricted, it materially reduces eligibility to be retrieved and cited - access is a prerequisite, not a citation guarantee. That is why the AI Access Check is the first diagnostic to run before asking why competitors appear in AI answers and you do not.
Your robots.txt is graded across the frozen DD-AI-CRAWLERS-v1.0 panel of 21 crawler, fetcher and control identities, spanning OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl. Search and indexing, user-triggered fetching, training-associated crawling and data-use controls are kept distinct, because they are not the same thing.
What our identified audit crawler received when it connected: clean response, challenge page, 403, redirect or timeout. Reported separately from declared policy, and never attributed to a vendor's crawler - we do not impersonate any other provider's crawler identity.
A deeper investigation of how your edge actually behaves, and where enforcement diverges from what robots.txt declares. This tests the infrastructure layer that robots.txt alone cannot describe, without inferring the site owner’s intention.
Sitemap and crawl-delay observations, plus an optional llms.txt observation - whether one is published, whether it is valid, and whether it matches your declared policy. llms.txt is optional and emerging; its absence is not a technical failure.
Deeper investigation of redirect chains, certificate and TLS failures, response classifications, challenge pages and conflicting access configuration.
A prioritised, plain-English list of exactly what to change, with the precise robots.txt lines, header changes, or CDN settings - ready to hand to your developer. Every report is human-QA'd before delivery.
The audit applies research-informed crawler-access analysis informed by the independently published PTODA C01 AI Crawler Access Study - the research that showed why simple “AI blocked” statistics can mislead. Declared policy, path impact and observed access are different measurements. We measure them separately. Want to explore the deliverable? Read a focused sample report, or run the free DD Diagnostic Suite for the basics - the audit is the full evidence-grade version.
Not sure what you get? See the focused sample report ›
The Digital Dominator audit answers a focused technical question: what does your site declare about crawler access, what does our identified crawler observe, and what should your developer fix?
Once access is understood, OG01 provides the broader AI visibility audit: authority, content, entities, trust, competitor visibility, scoring and ongoing improvement. DD findings feed into that larger assessment. The products are sequential and deliberately do not duplicate one another.
The method is the same everywhere; what differs is where a restriction lives and who can change it. These pages cover the situations that come up most.
Multi-team estates where robots.txt, the CDN and the WAF are owned by different people. Includes the security handoff and multi-domain scope.
Where declared policy and edge enforcement disagree, what can be established from outside, and what needs your own dashboard.
Platform-generated robots behaviour, app and theme effects, and remediation that stays inside Shopify’s constraints.
Generated robots.txt, SEO and security plugins, caching, hosting and CDN — and which layer is actually deciding.
The audit is delivered remotely with an identified audit crawler, so location is never a barrier. We have local pages for the markets we work in most:
Corporate, finance, legal and multi-location brands behind enterprise firewalls.
Tourism, hospitality, property and local service businesses.
OpenAI uses distinct identities for distinct purposes: GPTBot is associated with potential model training, OAI-SearchBot handles ChatGPT Search discovery, and ChatGPT-User is a user-triggered fetcher rather than an ordinary autonomous crawler. Your robots.txt states a declared policy for each; your firewall or CDN decides what actually gets through, and the two often disagree. The audit reports the declared policy for every identity in the panel and, separately, what our own identified audit crawler received.
Access can be one reason: a policy or infrastructure response may prevent retrieval of your pages. The independently published PTODA C01 study analysed 2,699 sampled rows across five national cohorts, representing 2,652 distinct domains. It found whole-site exclusion of AI retrieval crawlers was uncommon in every market, while restrictions reaching primary public content remained a minority position. Digital Dominator applies research-informed analysis commercially to determine the evidence for your site.
The panel separates four classes. Training-associated crawlers (GPTBot, ClaudeBot) are associated with potential model training. Search and discovery agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so assistants can find and cite you today. User-triggered fetchers (ChatGPT-User and equivalents) act only when a person asks, and vendor-specific robots behaviour applies to them. Data-use controls (Google-Extended, Applebot-Extended) are not crawlers at all. Restricting training-associated crawlers is a legitimate choice and does not directly control current search/citation access; restricting the search and discovery agents can reduce direct retrieval and citation opportunities. The audit grades all four separately.
No - it tells you what you have declared. Declared policy does not prove actual firewall, CDN or WAF access; a site can allow an agent in robots.txt and still block it at the edge, or the reverse. That is why declared policy and observed access are reported as separate evidence classes, and why the deeper edge enforcement investigation sits in the Full AI Access Audit.
Not necessarily. llms.txt is optional and still emerging, and its absence is not a technical failure. Where one exists we observe whether it is valid and whether it agrees with your declared robots policy.
Because they ask different questions. Tools differ in which identities they test, whether they read robots.txt only or also make a request, how they treat redirects and challenge pages, and where they connect from. An edge can also answer two requesters differently. We record what was requested, what came back and when, so a result can be compared rather than argued about.
No, and we will not claim it. We report two things: what your robots.txt declares for each identity in the panel, and what Digital Dominator’s own identified audit crawler received when it requested your site. We do not send requests from a vendor network and we never impersonate a vendor crawler, so our observation is evidence about our crawler, not proof of theirs.
We say so plainly and issue no verdict we cannot support. If robots.txt cannot be read, or our crawler is refused, or the page cannot be retrieved, the report records what was observed and states which checks were therefore not performed. An absence of measurement is never reported as a clean result.
Yes. The audit observes HTTP behaviour rather than your admin panel, so it is platform-neutral. Platform specifics matter for remediation, which is why there are dedicated pages for WordPress, Shopify and Cloudflare-fronted sites. For JavaScript-heavy sites we report what is present in the server-rendered HTML, because that is what a non-rendering fetcher receives.
Re-check after any change to robots.txt, hosting, CDN or security configuration, after a platform or theme migration, and otherwise periodically. Quarterly suits most sites. Access is a point-in-time observation, and a plugin update or an edge rule can change it without anyone deciding to.
The report contains the exact changes - precise robots.txt lines, header changes, CDN settings - ready to hand to your developer. The report separates confirmed findings from assessment limits and is human-reviewed before delivery within five business days. The Full AI Access Audit includes a free re-check after your fixes.
Choose a fixed-price audit and pay securely by card. Stripe collects your website address at checkout, and we will confirm scope by email. Every paid report is human-reviewed and delivered within five business days.
Agency, multi-site or need to confirm scope first? Use the enquiry form below.
Live access observations are performed using Digital Dominator's identified audit crawler. It reads robots.txt first and only performs additional automated retrieval where that policy permits our crawler. Observed responses are reported separately from vendor-specific crawler policy.