Does Playwright respect robots.txt?

No, Playwright does not reliably respect robots.txt. Additional measures like IP blocking may be needed.

PLAYWRIGHT

Q: What is the Playwright User-Agent string?

The Playwright User-Agent string is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Q: How do I block Playwright in robots.txt?

To control Playwright access, add the following to your robots.txt file: # Playwright does not check robots.txt. # Detection requires browser fingerprinting and behavioral analysis. # No User-Agent-based blocking possible (uses standard browser UA).

HIGH RISK⚡ AUTOMATION AGENT

Microsoft's cross-browser automation framework — used for testing and scraping

ORGANIZATION

Microsoft

FIRST SEEN

2020-01

RESPECTS ROBOTS.TXT

✗ NO

DOCUMENTATION

playwright.dev

—

DAILY VISITS

—

COUNTRIES ACTIVE

TRACKING

STATUS

—

LAST SEEN

📡 PLAYWRIGHT USER-AGENT STRING

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

This is the User-Agent header sent by Playwright in HTTP requests. Use this to identify Playwright in your server access logs.

📋 ABOUT PLAYWRIGHT

Playwright is Microsoft's open-source browser automation framework, supporting Chromium, Firefox, and WebKit engines. Originally designed for end-to-end testing of web applications, Playwright has become increasingly popular for web scraping and automated data collection due to its powerful API and cross-browser support.

Playwright is particularly challenging to detect because it does not identify itself in the User-Agent string and can closely emulate real browser behavior including mouse movements, keyboard input, and page interactions. It supports multiple browser engines, device emulation, and can intercept network requests — making it one of the most capable automation tools available.

NORAD.io tracks Playwright-based automation as part of its high-risk agent detection. With Playwright increasingly being used by AI agents for web browsing (including tools like OpenAI's computer use and various AI agent frameworks), detecting and managing Playwright traffic is becoming critical for web security. NORAD employs behavioral analysis and browser fingerprinting to identify Playwright sessions.

🎯 HOW TO DETECT PLAYWRIGHT

▸No distinctive User-Agent — uses standard browser strings
▸Check for Playwright-specific browser launch arguments in CDP
▸navigator.webdriver may be true (can be patched)
▸Behavioral analysis: automated interaction patterns, consistent timing
▸Playwright uses its own browser binaries — version mismatches can be a signal
▸Network-level: connections from cloud/datacenter IPs rather than residential

🔄 CRAWL BEHAVIOR

Fully automated browser with JavaScript rendering. Supports Chromium, Firefox, and WebKit engines. Can emulate mobile devices, geolocations, and permissions. No default User-Agent identification — mimics regular browsers.

PURPOSE

End-to-end testing framework for web applications. Also used for web scraping, automated data collection, and browser-based automation tasks. Increasingly used by AI agents for web interaction.

🤖 ROBOTS.TXT CONFIGURATION

# Playwright does not check robots.txt.
# Detection requires browser fingerprinting and behavioral analysis.
# No User-Agent-based blocking possible (uses standard browser UA).

⚠ Playwright may not fully respect robots.txt. Consider supplementing with IP-level blocking or bot detection middleware.

→ Complete Guide: robots.txt for AI Bots