Sitemap
Explore every public page and jump to the information you need.
Main pages
Account
Crawler tracking pages
Track this crawler on your sites
AI Bots
- AI Chatbot
- AI2Bot
- Ai2Bot-Dolma
- anthropic-ai
- Big Sur AI
- BigSurAI
- ChatGPT Operator
- ChatGPT-User
- ChatGPT-User/2.0
- Claude-Code
- Claude-SearchBot
- Claude-User
- Claude-Web
- ClaudeBot
- cohere-ai
- Cohere-Training-Data-Crawler
- DeepSeekBot
- DigitalOceanGenAI-Crawler
- DuckAssistBot
- GPTBot
- Grok
- GrokBot
- img2dataset
- Liner-Bot
- MistralAI-User
- MistralAI-User/1.0
- MyCentralAIScraperBot
- OAI-SearchBot
- PanguBot
- Perplexity-User
- Perplexity-User/1.0
- PerplexityBot
- QuillBot
- SBIntuitionsBot
- YouBot
Cloud Services
Google Bots
- Gemini-Deep-Research
- Google-CloudVertexBot
- Google-Extended
- Google-InspectionTool
- Googlebot
- Googlebot-Discovery
- Googlebot-Image
- Googlebot-News
- Googlebot-Video
- GoogleOther
- GoogleOther-Image
- GoogleOther-Video
- Storebot-Google
Other Agents
SEO Tools
- AhrefsBot
- AudigentAdBot
- AwarioRssBot
- AwarioSmartBot
- Brandwatch
- Chrome-Lighthouse
- DataForSeoBot
- DotBot
- Google-PageSpeed-Insights
- Jetslide
- MegaIndex
- Meltwater
- MJ12bot
- NeticlaBot
- NeticleBot
- Netvibes
- PageSpeed
- Peer39 Crawler
- Peer39 Crawler/1.0
- Screaming Frog
- Screaming Frog SEO Spider
- SearchmetricsBot
- SemrushBot
- SemrushBot-OCOB
- SemrushBot-SWA
- Sidetrade Indexer Bot
- SiteAuditBot
- TaranGroup Intelligent Bot
Scrapers
- Amazon Kendra
- Arquivo-web-crawler
- Barkrowler
- BLEXBot
- Bravest
- Bytespider
- CCBot
- Cotoyogi
- Crawl4AI
- crawler4j
- Crawlspace
- DiffBot
- EchoBot-Bot
- EchoBoxBot
- Factset-SpyderBot
- Firecrawl
- FriendlyCrawler
- Hada News
- iAskSpider
- iAskSpider/2.0
- ICC-Crawler
- ImagesiftBot
- iMediaEthics.org
- imgproxy
- ISSCyberRiskCrawler
- JenkersBot
- Kangaroo Bot
- LivelapBot
- magpie-crawler
- MauiBot
- MoodleBot
- NetEstate Imprint Crawler
- news-please
- NewsNow
- NovaAct
- omgili
- omgilibot
- Poseidon Research Crawler
- QualifiedBot
- Scrapy
- SeekportBot
- Seekr
- SeekrBot
- Timpibot
- Turnitin
- TurnitinBot
- VelenPublicWebCrawler
- Webzio-Extended
- Yacy
- yacybot
Search Engines
- adidxbot
- Amazon-SearchBot
- Applebot
- Applebot-Extended
- AspiegelBot
- Baiduspider
- Bingbot
- BingPreview
- BingVideoPreview
- coccocbot-web
- Daumoa
- DuckDuckBot
- MicrosoftPreview
- Mojeek
- MojeekBot
- MSNBot
- NaverBot
- PetalBot
- SeznamBot
- SeznamHomepageCrawler
- Slurp
- Sogou
- Teoma
- Yahoo
- Yahoo! Slurp
- Yahoo-Blogs
- Yahoo-FeedSeeker
- Yahoo-MMCrawler
- YahooSeeker
- Yandex
- YandexAdditional
- YandexAdditionalBot
- YandexBot
- Yeti
Social Bots
Coverage
196 pages covering crawler tracking, setup, and account tools.