GPTBOT

GPTBot is OpenAI's official web crawler, first publicly documented in August 2023. It systematically crawls publicly accessible web pages to collect training data for OpenAI's large language models, including GPT-4, GPT-4o, and future model generations. GPTBot identifies itself clearly in the User-Agent header and operates from a set of published IP ranges, making it straightforward to identify and control.

Unlike OpenAI's ChatGPT-User bot (which fetches pages in real-time during conversations), GPTBot performs batch crawling operations for training data collection. It respects robots.txt directives, and OpenAI provides clear documentation on how to opt out of crawling. The bot does not execute JavaScript, does not render pages, and focuses on extracting text content from HTML pages. It follows links discovered in sitemaps and page content.

NORAD.io tracks GPTBot activity across its global sensor network, providing real-time visibility into crawl frequency, geographic distribution, and behavioral patterns. Many website operators use NORAD to monitor how aggressively GPTBot crawls their content and to enforce access policies through the Agent Passport Standard.