Free tool
Can AI engines read your site?
Check your robots.txt against 20 AI crawlers and assistants, from GPTBot to Perplexity-User, and see the exact rule that allows or blocks each one.
Questions
- Should I block GPTBot?
- GPTBot only collects training data. Blocking it keeps your pages out of future OpenAI model training, but ChatGPT search still uses OAI-SearchBot and ChatGPT-User. If you want ChatGPT to recommend you, keep those two allowed.
- What does Google-Extended do?
- It isn't a crawler. It's a robots.txt token that tells Google whether Gemini may use your content for training and grounding. Googlebot keeps crawling for Search and AI Overviews either way.
- What if my robots.txt returns an error?
- Under the robots.txt standard (RFC 9309), a server error means crawlers should assume the whole site is disallowed. A missing robots.txt (404) means everything is allowed.
- Does allowed mean AI engines use my site?
- No. Robots.txt only says what crawlers may fetch. Whether they visit, and whether answers recommend you, is what Vedlora's AI crawler analytics and visibility tracking measure.
See which AI crawlers actually visit
Vedlora tracks AI crawler visits to your pages and what ChatGPT, Perplexity and Gemini say about your brand.