TWITTERBOT
LOW RISK🔍 SEARCH & AI CRAWLERTwitter/X's crawler for generating link preview cards in tweets
📡 TWITTERBOT USER-AGENT STRING
Twitterbot/1.0
This is the User-Agent header sent by Twitterbot in HTTP requests. Use this to identify Twitterbot in your server access logs.
📋 ABOUT TWITTERBOT
Twitterbot is the web crawler operated by X (formerly Twitter) to generate link preview cards when URLs are shared in tweets. When a user posts or views a tweet containing a URL, Twitterbot fetches the page to extract metadata for generating a rich preview card displaying the page title, description, and image.
Twitterbot reads Twitter Card meta tags (twitter:card, twitter:title, twitter:description, twitter:image) and falls back to Open Graph tags when Twitter-specific tags aren't available. The bot's traffic is user-driven — each request corresponds to a link being shared or viewed on the platform. Twitterbot does not execute JavaScript and focuses purely on HTML meta tag extraction.
NORAD.io monitors Twitterbot activity to help site operators track social sharing patterns and ensure their Twitter Card implementations are working correctly. While Twitterbot is low-risk, monitoring its activity provides valuable insights into content virality and social media referral patterns.
🎯 HOW TO DETECT TWITTERBOT
- ▸User-Agent is simply 'Twitterbot/1.0'
- ▸Crawls from Twitter's IP ranges (199.16.156.0/22, 199.59.148.0/22)
- ▸Looks for Twitter Card meta tags (twitter:card, twitter:title, etc.)
- ▸Traffic patterns correlate with link sharing activity on Twitter/X
- ▸Also fetches og: meta tags as fallback when Twitter Card tags are missing
🌐 TWITTERBOT KNOWN IP RANGES
199.16.156.0/22199.59.148.0/22Use these CIDR ranges to verify Twitterbot identity at the network level. Always combine with User-Agent verification for accurate detection.
🔄 CRAWL BEHAVIOR
On-demand crawling triggered by link sharing on Twitter/X. Fetches page HTML and meta tags (Twitter Cards) to generate link previews. Re-fetches periodically to update cached previews. Does not execute JavaScript.
Generates rich link preview cards (Twitter Cards) when URLs are shared in tweets. Reads twitter:card, twitter:title, twitter:description, and twitter:image meta tags.
🤖 ROBOTS.TXT CONFIGURATION
User-agent: Twitterbot Allow: / # Blocking will prevent Twitter Card previews for your links
Twitterbot respects robots.txt directives. Add this to your robots.txt file at the root of your domain.
🗺️ WHERE IS TWITTERBOT ACTIVE?
⚠️ RELATED THREATS
Attempts to override bot instructions via malicious content embedded in web pages
Data ExfiltrationBots attempting to extract sensitive data from websites including PII and credentials
Credential StuffingAutomated login attempts using leaked credentials from data breaches
Aggressive Content ScrapingBots aggressively scraping content beyond robots.txt limits and ToS
📂 MORE 🔍 SEARCH & AI CRAWLERS
📚 RELATED GUIDES
PROTECT YOUR WEBSITE
Deploy SiteTrust to monitor and control AI bot access to your site with the Agent Passport Standard.
INSTALL SITETRUST →