Does your website allow AI crawlers?
AI services use different crawlers and content controls for search, user-requested retrieval and model development. This checker examines your robots.txt rules and compares homepage responses across selected user-agent profiles.
We check your public robots.txt and homepage. No login or form submission is performed. Genuine provider IP ranges are not tested.
How this check works
Robots policy. We fetch the site's robots.txt file and evaluate its rules for selected search crawlers, user-triggered fetchers, model-training crawlers and content-use controls. Restrictions affecting only private or utility routes are separated from restrictions affecting public content.
User-agent response testing. We request the public homepage using a normal browser user agent and selected crawler user-agent strings. We compare the responses to identify obvious user-agent-based blocking.
What this test cannot prove. The requests originate from AudFlo's infrastructure, not the providers' published crawler IP ranges. A CDN or firewall may apply additional IP-based rules. A successful result means no blocking was detected by the checks performed. It is not a guarantee that every genuine crawler can reach every page.