What is the DataForSeoBot User-Agent string?

The DataForSeoBot User-Agent string is: Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

How do I block DataForSeoBot in robots.txt?

To control DataForSeoBot access, add the following to your robots.txt file: User-agent: DataForSeoBot Allow: / Crawl-delay: 5 # To block: # User-agent: DataForSeoBot # Disallow: /

Does DataForSeoBot respect robots.txt?

Yes, DataForSeoBot respects robots.txt directives.

DATAFORSEOBOT

LOW RISK📊 SEO & DATA SCRAPER

DataForSEO's crawler for SEO data API services and SERP analysis

ORGANIZATION

DataForSEO

FIRST SEEN

2019-01

RESPECTS ROBOTS.TXT

✓ YES

DOCUMENTATION

dataforseo.com

—

DAILY VISITS

—

COUNTRIES ACTIVE

TRACKING

STATUS

—

LAST SEEN

📡 DATAFORSEOBOT USER-AGENT STRING

Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

This is the User-Agent header sent by DataForSeoBot in HTTP requests. Use this to identify DataForSeoBot in your server access logs.

📋 ABOUT DATAFORSEOBOT

DataForSeoBot is the web crawler operated by DataForSEO, a company providing SEO data through API services. DataForSEO's APIs are used by hundreds of SEO tools and agencies to access search engine results data, backlink information, keyword rankings, and on-page SEO analysis. DataForSeoBot crawls websites to collect the underlying data that powers these API products.

DataForSeoBot is a well-behaved crawler that identifies itself clearly, respects robots.txt and Crawl-delay directives, and provides documentation about its behavior. It focuses on structural and content analysis rather than full content extraction, collecting the metadata and link information needed for SEO analysis.

NORAD.io tracks DataForSeoBot as part of the SEO crawler ecosystem. DataForSEO's API model means that DataForSeoBot's crawling ultimately powers many third-party SEO tools and platforms, making it a significant node in the web data collection landscape despite being less well-known than AhrefsBot or SemrushBot.

🎯 HOW TO DETECT DATAFORSEOBOT

▸User-Agent contains 'DataForSeoBot'
▸Moderate crawl volume
▸Respects Crawl-delay directives
▸Focuses on content and structural analysis
▸Documentation at dataforseo.com/dataforseo-bot

🔄 CRAWL BEHAVIOR

Moderate crawl rates. Respects robots.txt and Crawl-delay. Focuses on content analysis for SEO metrics. Does not execute JavaScript.

PURPOSE

Collects web data to power DataForSEO's API products, which provide SEO data to agencies, tools, and platforms. Services include SERP analysis, backlink data, keyword data, and on-page SEO metrics.

🤖 ROBOTS.TXT CONFIGURATION

User-agent: DataForSeoBot
Allow: /
Crawl-delay: 5

# To block:
# User-agent: DataForSeoBot
# Disallow: /

DataForSeoBot respects robots.txt directives. Add this to your robots.txt file at the root of your domain.

→ Complete Guide: robots.txt for AI Bots

🗺️ WHERE IS DATAFORSEOBOT ACTIVE?

⚠️ RELATED THREATS

Prompt Injection

Attempts to override bot instructions via malicious content embedded in web pages

Data Exfiltration

Bots attempting to extract sensitive data from websites including PII and credentials

Credential Stuffing

Automated login attempts using leaked credentials from data breaches

Aggressive Content Scraping

Bots aggressively scraping content beyond robots.txt limits and ToS

📂 MORE 📊 SEO & DATA SCRAPERS

AhrefsBotAhrefs SemrushBotSemrush ScrapyOpen Source Python RequestsUnknown cURLOpen Source MJ12botMajestic

📚 RELATED GUIDES

How to Detect AI Bots →robots.txt for AI Bots →NORAD API Docs →

PROTECT YOUR WEBSITE

Deploy SiteTrust to monitor and control AI bot access to your site with the Agent Passport Standard.

INSTALL SITETRUST →