Generated or cached robots.txt
The file is produced at request time and may be cached. What you edited and what is served can differ.
WordPress · Plugins, cache & hosting · Remote delivery
WordPress AI Crawler Access Audit.
On WordPress, crawler access is decided by a stack: a generated robots.txt, an SEO plugin, a security plugin, a cache, a host and often a CDN. Any layer can override the one above it, and usually nobody chose the result.
A WordPress AI crawler access audit grades the site’s declared robots policy across a frozen 21-identity panel and separately records what Digital Dominator’s identified crawler received. It identifies which layer a restriction appears at — generated file, plugin, cache, host or CDN — and returns remediation your developer or hosting company can apply directly.
Which layer of your WordPress stack is deciding what AI can read?
Fixed price. AUD $495 / USD $349 AI Access Check, or the AUD $1,195 / USD $799 Full AI Access Audit for deeper CDN, WAF, TLS and redirect analysis and a re-check after your fixes.
Book your AI Access Audit ›
01Why WordPress access problems are layered
WordPress rarely serves a static robots.txt. It is generated, and an SEO plugin usually manages it. A security plugin may add its own rules on top. A caching layer can serve a stored copy of any of it. The host may add rules of its own, and a CDN may add more.
The result is that the same site can be restrictive at one layer and permissive at another, and the layer that wins is not obvious from any admin screen. It is also why plugin settings appear to revert: another layer is producing the output.
This audit works from the outside in. It records what an identified automated client actually receives, then identifies which layer the behaviour appears to originate at, so the fix is applied where it will hold.
02What can go wrong
SEO plugin management
SEO plugins commonly own robots output and directive tags. Their settings can be overwritten by an update, an import or another plugin.
Security plugin rules
Security plugins may restrict automated clients as a category, which can affect AI retrieval agents alongside the traffic they were aimed at.
Caching layers
A cache can serve an older policy or an older page to an automated client long after the source changed.
Hosting and CDN layers
Managed hosts and CDNs add their own automated-traffic handling, independent of anything in WordPress.
noindex and nosnippet
Directives applied per template or per post type restrict use separately from crawler access, and are easy to apply more broadly than intended.
03What the audit examines
Observed behaviour first, then the layer it appears to originate at.
Declared policy, 21 identities
The served robots policy graded across the frozen panel, with the exact source line behind every verdict.
Observed access
What our identified crawler received on the homepage and a representative set of templates.
Layer attribution
Whether an observed restriction appears to originate at the generated file, a plugin, the cache, the host or the CDN.
Server-rendered content
What is present before JavaScript runs, since that is what a non-rendering fetcher receives.
Directive observations
noindex and nosnippet signals observed on the templates tested.
Redirects and status codes
The chain an identified automated client experiences, including the URL finally reached.
04What you receive
A layered verdict
What is restricted and at which layer it appears, so the change is made where it will actually hold.
Exact remediation
The precise robots lines, plugin settings, header or cache changes, written for your developer or hosting company.
A hosting-company brief
Findings that belong to the host or CDN, described so you can raise them as a support request without translation.
Prioritised order
Primary content restrictions first; secondary and operational paths after.
A re-check
The Full AI Access Audit includes a free re-check after your fixes, which matters most where settings have reverted before.
Evidence for each finding
Every item is tied to what was observed, so a plugin is never blamed on a hunch.
05Fixed-price tiers
AI Access Check
AUD $495
includes Australian GST · USD $349
- Executive access verdict
- Frozen 21-identity declared-policy matrix
- Exact robots.txt evidence per finding
- Separate Digital Dominator crawler access observation
- Prioritised developer-ready remediation
- Human-reviewed before delivery
Recommended
Full AI Access Audit
AUD $1,195
includes Australian GST · USD $799
- Everything in the AI Access Check
- Deeper CDN, WAF, TLS and redirect analysis
- Expanded technical access analysis
- Structured data and machine-readability review
- Walkthrough call
- Free re-check after your fixes
Fix & Verify
Quoted
- We implement the agreed changes
- Bounded scope, quoted from your audit
- Re-tested and verified after changes
- Separate from any ongoing retainer
Delivered remotely; no admin login is required for the audit itself.
06What this evidence does not show
Access is necessary for retrieval and citation but does not guarantee either. We report what Digital Dominator’s identified audit crawler observed; that is evidence about our crawler and is never presented as proof of another provider’s crawler behaviour, and we do not request from a vendor network or impersonate a vendor crawler. Training crawlers, search and discovery crawlers, user-triggered fetchers and data-use controls are not interchangeable and are graded separately. llms.txt is optional and emerging, not a ranking requirement. One discipline is worth stating for WordPress specifically: we do not name an individual plugin as the cause of a restriction without evidence. Where behaviour is consistent with a plugin or host layer, that is reported as an observation for your team to confirm.
07Common questions
My SEO plugin says AI crawlers are allowed. Why would the audit disagree?
Because the plugin describes its own intent, not the served result. A cache, a security plugin, the host or a CDN can produce different output. We report what an identified crawler actually received, and where that behaviour appears to originate.
Our robots.txt settings keep reverting. Why?
Commonly another layer is producing the served file, so the edit is being made in a place that does not decide the output. The audit identifies which layer appears to be winning so the change is applied where it holds.
Can a security plugin block AI crawlers?
A security plugin can restrict automated clients as a category, which may affect AI retrieval agents. We report the observed behaviour and the layer it appears to come from, without naming a specific plugin as the cause unless the evidence supports it.
Will you tell my host exactly what to change?
Yes. Findings that belong to the host or CDN are written as a brief you can send as a support request, with the observed evidence attached.
08Related reading
Start with how to check whether your site is blocking AI, then the 21 AI crawlers explained for which agents actually matter. Pricing and method are on the central AI Access Audit page.
Which layer of your WordPress stack is deciding what AI can read?
Start with the central AI Access Audit, or book directly below.
Book your AI Access Audit ›