Platform-generated robots behaviour
Much of the file is produced by Shopify. Knowing which lines are yours to influence prevents wasted effort and broken edits.
Shopify storefronts · Products & collections · Remote delivery
Shopify AI Crawler Access Audit.
Shopify generates a lot of your crawler policy for you. That is mostly helpful, and it means the parts you can change — and the parts you cannot — are worth knowing precisely before anyone starts editing.
A Shopify AI crawler access audit grades the storefront’s declared robots policy across a frozen 21-identity panel and separately records what Digital Dominator’s identified crawler received on representative product, collection and content pages. Remediation is written to stay inside Shopify’s platform constraints, so you are never handed a fix the platform will not accept.
Can AI assistants actually retrieve your product and collection pages?
Fixed price. AUD $495 / USD $349 AI Access Check, or the AUD $1,195 / USD $799 Full AI Access Audit for deeper CDN, WAF, TLS and redirect analysis and a re-check after your fixes.
Book your AI Access Audit ›
01What is yours to change on Shopify, and what is not
Shopify manages robots.txt generation, the storefront CDN and much of the redirect behaviour for you. Merchants can influence some of it, apps can influence more of it than most merchants expect, and a theme change can alter what reaches a non-rendering client without touching a single setting.
The practical consequence is that a Shopify access problem is rarely one file. It is usually an interaction: an app that adds rules, a theme that renders key content client-side, or a redirect that behaves differently for an automated client than it does in a browser.
This audit establishes what is actually happening, then writes remediation that respects the platform’s boundaries — because advice that requires control Shopify does not give you is not remediation.
02What can go wrong
Apps that change access
Installed apps can add rules or headers. The effect is real and often invisible in the theme editor.
Theme changes and client rendering
Content assembled in the browser is not present in the server-rendered HTML that a non-rendering fetcher receives.
Canonical and redirect behaviour
Duplicate collection paths, tag and filter URLs, and locale redirects can change what an automated client finally reaches.
noindex and nosnippet directives
Directives applied to templates or by an app can restrict how content is used, separately from crawler access.
Storefront CDN responses
The storefront edge answers automated clients on its own terms, which may differ from what a browser sees.
03What the audit examines
What an identified automated client actually receives across a representative set of storefront pages.
Declared policy, 21 identities
The storefront robots policy graded across the frozen panel, with the exact source line behind every verdict.
Observed access
What our identified crawler received on the homepage and representative product, collection and content pages.
Server-rendered content
What is present in the HTML before JavaScript runs, because that is what a non-rendering fetcher reads.
Canonical and redirect handling
Where automated clients are redirected and what they finally receive.
Directive observations
noindex and nosnippet signals observed on the pages tested, reported where relevant to retrieval and use.
App and theme effects
Where an observed behaviour is consistent with an app or theme influence, described as an observation for your team to confirm.
04What you receive
A storefront access verdict
What is restricted, on which page types, and whether it touches primary commercial content.
Platform-safe remediation
Changes that can actually be made inside Shopify, separated from anything that would need the platform or an app vendor.
A developer or agency brief
Written so your Shopify developer can apply it without a translation step.
Prioritised order
Product and collection access ahead of secondary paths, because that is where the commercial effect sits.
A re-check
The Full AI Access Audit includes a free re-check after your fixes.
Honest platform boundaries
Where Shopify controls the behaviour, we say so rather than promising control we do not have.
05Fixed-price tiers
AI Access Check
AUD $495
includes Australian GST · USD $349
- Executive access verdict
- Frozen 21-identity declared-policy matrix
- Exact robots.txt evidence per finding
- Separate Digital Dominator crawler access observation
- Prioritised developer-ready remediation
- Human-reviewed before delivery
Recommended
Full AI Access Audit
AUD $1,195
includes Australian GST · USD $799
- Everything in the AI Access Check
- Deeper CDN, WAF, TLS and redirect analysis
- Expanded technical access analysis
- Structured data and machine-readability review
- Walkthrough call
- Free re-check after your fixes
Fix & Verify
Quoted
- We implement the agreed changes
- Bounded scope, quoted from your audit
- Re-tested and verified after changes
- Separate from any ongoing retainer
Delivered remotely; no store admin access is required for the audit itself.
06What this evidence does not show
Access is necessary for retrieval and citation but does not guarantee either. We report what Digital Dominator’s identified audit crawler observed; that is evidence about our crawler and is never presented as proof of another provider’s crawler behaviour, and we do not request from a vendor network or impersonate a vendor crawler. Training crawlers, search and discovery crawlers, user-triggered fetchers and data-use controls are not interchangeable and are graded separately. llms.txt is optional and emerging, not a ranking requirement. On a hosted platform one limit is structural: we do not promise direct control over platform-managed infrastructure, and we do not name an app as the cause of a restriction without evidence. Where behaviour is consistent with an app or theme, that is reported as an observation to confirm, not a conclusion.
07Common questions
Can I even change robots.txt on Shopify?
Shopify generates much of it and gives merchants a defined way to influence parts of it. The audit identifies which findings are actually yours to change and which are platform-managed, so effort goes where it can have an effect.
Our theme renders content with JavaScript. Does that matter?
It can. We report what is present in the server-rendered HTML, because that is what a non-rendering fetcher receives. Content that only appears after JavaScript runs may not reach such a client at all.
Can an app be blocking AI crawlers?
An app can add rules or headers that affect automated access. We report the observed behaviour and note where it is consistent with an app influence, but we do not name an app as the cause without evidence.
Do you test product and collection pages or just the homepage?
A representative set, including product and collection pages, because storefront templates can behave differently from the homepage and those are the pages with commercial value.
08Related reading
To understand which agents matter for a storefront, read the 21 AI crawlers explained. For a practical first look at your own file, how to check whether your site is blocking AI. Pricing sits on the central AI Access Audit page.
Can AI assistants actually retrieve your product and collection pages?
Start with the central AI Access Audit, or book directly below.
Book your AI Access Audit ›