Does Selenium respect robots.txt?

No, Selenium does not reliably respect robots.txt. Additional measures like IP blocking may be needed.

SELENIUM

Q: What is the Selenium User-Agent string?

The Selenium User-Agent string is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Q: How do I block Selenium in robots.txt?

To control Selenium access, add the following to your robots.txt file: # Selenium does not check robots.txt. # Uses standard browser User-Agent strings. # Detection requires WebDriver protocol detection or behavioral analysis.

HIGH RISK⚡ AUTOMATION AGENT

The original browser automation framework — still widely used for testing and scraping

ORGANIZATION

Open Source

FIRST SEEN

2004-01

RESPECTS ROBOTS.TXT

✗ NO

DOCUMENTATION

selenium.dev

—

DAILY VISITS

—

COUNTRIES ACTIVE

TRACKING

STATUS

—

LAST SEEN

📡 SELENIUM USER-AGENT STRING

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

This is the User-Agent header sent by Selenium in HTTP requests. Use this to identify Selenium in your server access logs.

📋 ABOUT SELENIUM

Selenium is the original browser automation framework, first created in 2004 by Jason Huggins at ThoughtWorks. It established the WebDriver protocol that became a W3C standard and remains the most widely-used browser automation tool globally. Selenium supports all major browsers through browser-specific driver implementations (ChromeDriver, GeckoDriver, etc.).

As the oldest and most established automation framework, Selenium has an extensive ecosystem of detection and evasion techniques. The WebDriver specification requires that automated browsers expose the navigator.webdriver property, making baseline detection straightforward. However, various patches and extensions can modify this behavior. ChromeDriver historically left detectable artifacts ($cdc_ variables) that bot detection services use.

NORAD.io tracks Selenium-based automation as high-risk traffic. Despite being primarily a testing tool, Selenium is widely used for commercial web scraping and automated data collection at scale. NORAD's multi-layered detection approach identifies Selenium sessions through WebDriver property detection, browser artifact analysis, and behavioral patterns that distinguish automated from human browsing.

🎯 HOW TO DETECT SELENIUM

▸navigator.webdriver property is true (W3C WebDriver spec requirement)
▸window.document.$cdc_ or $wdc_ properties present in older ChromeDriver versions
▸ChromeDriver adds 'cdc_' prefixed variables to the document object
▸Specific ChromeDriver port patterns (typically 9515)
▸Missing or inconsistent browser feature sets compared to real browsers
▸Selenium-managed browsers have detectable differences in WebGL, Canvas, and AudioContext fingerprints

🔄 CRAWL BEHAVIOR

Full browser automation with JavaScript rendering via WebDriver protocol. Supports Chrome, Firefox, Safari, Edge. No default bot identification. Behavior entirely script-determined. Can mimic human browsing patterns.

PURPOSE

Browser automation primarily for testing web applications. Also used extensively for web scraping, automated form submission, data collection, and automated workflows.

🤖 ROBOTS.TXT CONFIGURATION

# Selenium does not check robots.txt.
# Uses standard browser User-Agent strings.
# Detection requires WebDriver protocol detection or behavioral analysis.

⚠ Selenium may not fully respect robots.txt. Consider supplementing with IP-level blocking or bot detection middleware.

→ Complete Guide: robots.txt for AI Bots