Declared crawler policy
Document robots.txt access rules for the audited set of crawler identities and relevant public paths. Policy declarations are not proof that a vendor crawler can retrieve content.
Public-sector websites often involve multiple teams responsible for publishing, security, infrastructure and procurement. Our review separates the crawler rules a website declares from what our own identified audit crawler actually receives, helping your teams establish evidence and decide what to investigate next.
Discuss a public-sector audit →Document robots.txt access rules for the audited set of crawler identities and relevant public paths. Policy declarations are not proof that a vendor crawler can retrieve content.
Record the HTTP responses received by our identified audit crawler, including restrictions or challenges at CDN, WAF or hosting layers.
Give digital, CMS, security and infrastructure owners an evidence-based handover, separating changes they can implement from questions needing privileged dashboard access.
Scope representative domains and page paths, define a pilot, then coordinate human-reviewed findings and priorities across the agreed estate.
Some government and higher-education organisations use Drupal; others use WordPress, enterprise content platforms or custom systems. The audit does not assume a particular CMS. We identify where robots rules, publishing behaviour and edge enforcement may be controlled—and where additional access or technical owners are required.
Evidence boundary: External scanning cannot establish internal firewall settings, privileged CDN rules, every vendor crawler's real-world access or procurement compliance. Results must be interpreted within the scope actually tested.
Confirm domains, representative paths, responsible contacts, safety and testing constraints.
Review declared policy and observed responses using a documented set of checks.
Distinguish confirmed issues, unresolved restrictions and evidence requiring owner validation.
Provide a practical plan for digital, development and security teams, with re-check scope agreed separately.
For a single site, see our AI Access Audit. For multiple domains or departments, see our Enterprise AI Crawler Access Audit.
Enquire about government scope →