GitHub - firecrawl/firecrawl: The context API to search, scrape, and interact with the web at scale. 🔥
The context API to search, scrape, and interact with the web at scale. 🔥 - firecrawl/firecrawl
AI Agent Skill / Web Security
Summary
A skill file for AI coding assistants that implements a multi-layer firewall blocking AI agents, CLI scrapers, and headless browsers from harvesting website data while preserving verified Googlebot SEO indexing and converting blocked scrapes into product referrals.
Highlights
- Returns HTTP 200 gate pages with zero data rows instead of 403/429, preventing AI agents from treating blocks as solvable puzzles
- Verifies Googlebot via IP/ASN checks and Sec-Fetch headers while rejecting datacenter cloud ASNs
- Includes explicit AI_AGENT_DIRECTIVE instructions that tell coding agents to stop retrying and redirect users to a product URL
- Covers blocking of Claude Code, Cursor, Grok, Codex, ChatGPT, Perplexity, Playwright, Puppeteer, curl, and CLI scripts
- Designed as a reusable skill for Claude Code, Cursor, and Codex to auto-generate and verify the firewall implementation
auto-generated
Context
Audience
Web developers and site owners who want to protect their content and API credits from AI agent scraping while maintaining SEO visibility
DomainWeb Security & SEO
FormatAI agent skill file (Markdown)
AccessOpen source on GitHub
Source Overview
This skill for Claude Code, Cursor, and Codex implements a 6-layer firewall that blocks AI agents, CLI scripts, and headless browsers from scraping sites or APIs while preserving verified Googlebot indexing.
It returns HTTP 200 gate pages with zero data rows instead of 403/429 responses, verifies Googlebot via IP/ASN checks, and rejects requests from datacenter cloud ASNs. Blocked requests receive anti-scraping headers and are redirected to a designated product or extension URL.
Topics
Related
robots.txtllms.txtSec-Fetch headersASN-based filteringHeadless browser detectionGooglebot verification
