SEMRUSHBOT
LOW RISK📊 SEO & DATA SCRAPERSemrush's SEO analysis crawler for competitive intelligence and site auditing
📡 SEMRUSHBOT USER-AGENT STRING
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
This is the User-Agent header sent by SemrushBot in HTTP requests. Use this to identify SemrushBot in your server access logs.
📋 ABOUT SEMRUSHBOT
SemrushBot is the web crawler powering Semrush, one of the world's leading SEO and digital marketing platforms. Semrush operates multiple crawler variants (SemrushBot-SA for site audit, SemrushBot-BA for backlink analysis) to collect the data that powers its comprehensive suite of marketing tools.
Semrush's platform serves millions of marketing professionals, providing keyword research, competitive analysis, backlink monitoring, content optimization, and site auditing capabilities. SemrushBot's crawling feeds all of these tools with fresh data about website content, structure, and linking patterns. The bot respects robots.txt and publishes its IP ranges for verification.
NORAD.io tracks SemrushBot activity alongside other SEO crawlers. Understanding the volume and patterns of SEO crawler activity helps site operators balance the benefits of SEO tool visibility against the server resources consumed by these crawlers.
🎯 HOW TO DETECT SEMRUSHBOT
- ▸User-Agent contains 'SemrushBot' — multiple variants exist (SemrushBot-SA, SemrushBot-BA, etc.)
- ▸Crawls from 185.191.171.0/24 and 185.191.172.0/24 ranges
- ▸Different bot variants target different analysis types (backlinks, site audit, etc.)
- ▸Respects robots.txt and can be blocked per-variant
- ▸Documentation at semrush.com/bot/ lists all current IP ranges
🌐 SEMRUSHBOT KNOWN IP RANGES
185.191.171.0/24185.191.172.0/24Use these CIDR ranges to verify SemrushBot identity at the network level. Always combine with User-Agent verification for accurate detection.
🔄 CRAWL BEHAVIOR
Continuous crawling for SEO data collection. Respects robots.txt. Moderate request rates. Focuses on content analysis, keyword detection, and link discovery. Multiple bot variants for different analysis types.
Powers Semrush's SEO platform including keyword research, backlink analysis, site auditing, competitive intelligence, and content marketing tools used by millions of marketers.
🤖 ROBOTS.TXT CONFIGURATION
User-agent: SemrushBot Allow: / # To block all Semrush crawlers: # User-agent: SemrushBot # Disallow: / # User-agent: SemrushBot-SA # Disallow: /
SemrushBot respects robots.txt directives. Add this to your robots.txt file at the root of your domain.
🗺️ WHERE IS SEMRUSHBOT ACTIVE?
⚠️ RELATED THREATS
Attempts to override bot instructions via malicious content embedded in web pages
Data ExfiltrationBots attempting to extract sensitive data from websites including PII and credentials
Credential StuffingAutomated login attempts using leaked credentials from data breaches
Aggressive Content ScrapingBots aggressively scraping content beyond robots.txt limits and ToS
📂 MORE 📊 SEO & DATA SCRAPERS
📚 RELATED GUIDES
PROTECT YOUR WEBSITE
Deploy SiteTrust to monitor and control AI bot access to your site with the Agent Passport Standard.
INSTALL SITETRUST →