Firecrawl (GitHub Repo)
Firecrawl now features an autonomous 'Agent' endpoint that navigates and extracts structured web data without needing predefined URLs.
Deep dive
- Agent Endpoint: Enables autonomous web browsing and data retrieval based on natural language prompts.
- LLM Integration: Designed for direct integration with AI clients via MCP.
- Output formats: Native support for Markdown, structured JSON, and screenshots.
- Infrastructure: Handles proxy rotation, rate limiting, and JS-heavy rendering internally.
- Licensing: Available as a hosted service or self-hosted under AGPL-3.0.
- Benchmarks: Features a P95 latency of 3.4s across large-scale crawls.
- Tooling: Provides a CLI for direct interaction and SDKs for Python and Node.js.
Decoder
- MCP (Model Context Protocol): An open standard that allows AI assistants to connect to local and remote data sources, enabling them to execute actions or retrieve context securely.
- AGPL-3.0: A copyleft open-source license that requires anyone who modifies and runs the software over a network to make their source code available.
Original article
🔥 Firecrawl
The API to search, scrape, and interact with the web at scale. 🔥 The web context API to find sources, extract content, and turn it into clean Markdown or structured data your agents can ship with. Open source and available as a hosted service.
Why Firecrawl?
- Industry-leading reliability: Covers 96% of the web, including JS-heavy pages — no proxy headaches, just clean data (see benchmarks)
- Blazingly fast: P95 latency of 3.4s across millions of pages, built for real-time agents and dynamic apps
- LLM-ready output: Clean markdown, structured JSON, screenshots, and more — spend fewer tokens, build better AI apps
- We handle the hard stuff: Rotating proxies, orchestration, rate limits, JS-blocked content, and more — zero configuration
- Agent ready: Connect Firecrawl to any AI agent or MCP client with a single command
- Media parsing: Parse and extract content from web-hosted PDFs, DOCX, and more
- Actions: Click, scroll, write, wait, and press before extracting content
- Open source: Developed transparently and collaboratively — join our community
Feature Overview
Core Endpoints
| Feature | Description |
|---|---|
| Search | Search the web and get full page content from results |
| Scrape | Convert any URL to markdown, HTML, screenshots, or structured JSON |
| Interact | Scrape a page, then interact with it using AI prompts or code |
More
| Feature | Description |
|---|---|
| Agent | Automated data gathering, just describe what you need |
| Crawl | Scrape all URLs of a website with a single request |
| Map | Discover all URLs on a website instantly |
| Batch Scrape | Scrape thousands of URLs asynchronously |
Quick Start
Sign up at firecrawl.dev to get your API key. Try the playground to test it out.
Search
Search the web and get full content from results.
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
search_result = app.search("firecrawl", limit=5)
Scrape
Get LLM-ready data from any website — markdown, JSON, screenshots, and more.
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape('firecrawl.dev')
Interact
Scrape a page, then interact with it using AI prompts or code.
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape("https://amazon.com")
scrape_id = result.metadata.scrape_id
app.interact(scrape_id, prompt="Search for 'mechanical keyboard'")
app.interact(scrape_id, prompt="Click the first result")
Power Your Agent
Connect Firecrawl to any AI agent or MCP client in minutes.
Skill
Give your agent easy access to real-time web data with one command.
npx -y firecrawl-cli@latest init --all --browser
MCP
Connect any MCP-compatible client to the web in seconds.
{
"mcpServers": {
"firecrawl-mcp": {
"command": "npx",
"args": ["-y", "firecrawl-mcp"],
"env": {
"FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"
}
}
}
}
More Endpoints
Agent
The easiest way to get data from the web. Describe what you need, and our AI agent searches, navigates, and retrieves it. No URLs required.
Crawl
Crawl an entire website and get content from all pages.
Map
Discover all URLs on a website instantly.
Batch Scrape
Scrape multiple URLs at once.
SDKs
Our SDKs provide a convenient way to use all Firecrawl features and automatically handle polling for async operations (Python, Node.js, Java, Elixir, Rust).
Integrations
Agents & AI Tools: Firecrawl Skill, CLI Skills, Workflows, MCP.
Platforms: Lovable, Zapier, n8n.
Open Source vs Cloud
Firecrawl is open source under the AGPL-3.0 license. The cloud version at firecrawl.dev includes additional features.
License
This project is primarily licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). The SDKs and some UI components are licensed under the MIT License.
It is the sole responsibility of end users to respect websites' policies when scraping. Users are advised to adhere to applicable privacy policies and terms of use. By default, Firecrawl respects robots.txt directives. By using Firecrawl, you agree to comply with these conditions.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi