Skip to main content
📖 The AI Tool Bible

AI web scraping agent

Editorial picks for "ai web scraper".

32 tools

All Agents →
SC

ScrapeGraphAI

Agents · Multi-model (LLM, unspecified)
8.4

LLM-driven web scraping API that turns natural-language prompts into structured JSON.

Freemium· Free 500 credits; Starter $20/mo, Growth $100/mo, Pro $500/mo, Enterprise customweb-scrapingdata-extraction
N8

n8n

Agents · Multi-model
8.3

Source-available workflow automation with first-class AI-agent and RAG building blocks.

Freemium· Starter: 20€ · Pro: 50€ · Business: 667€ · Enterprise: Contact Salesai-agentsworkflow-automation
SK

Skyvern

Agents · Multi-model (OpenAI, Anthropic, Gemini, Ollama)
8.3

AI browser agent that automates web workflows from natural-language instructions, with CAPTCHA and 2FA handling built in.

Freemium· Free: $0/month · Hobby: $29/month · Pro: $149/month · Enterprise: Custom/monthbrowser-automationdata-extraction
BU

Browser Use

Agents · Multi-model (BYO LLM; Claude in hosted Box)
8.2

Open-source browser automation harness and cloud platform for LLM agents that drive real websites.

Freemium· PAYG: $0 · Dev: $29 · Business: $299 · Scaleup: $999web-automationscraping
SI

Sim

Agents · Multi-model (Claude, OpenAI, Google)
8.2

Open-source visual workspace for building, deploying, and monitoring AI agents.

Freemium· Free: $0 · Pro: $25 · Max: $100 · Enterprise: Customagent-workflowsinternal-automation
EX

Exa

RAG · Proprietary neural + keyword search
8.0

Web search API built for AI agents, with structured outputs and token-efficient highlights.

Freemium· Free Tier: Free · Search: $7/1k requests · Agent: $0.012–$1.00/run · Contents: $1/1k pages per content type · Deep Search: $12–15/1k requestsagent-web-searchrag-retrieval
FI

Firecrawl

RAG · Claude, Cursor, Windsurf, OpenAI, Gemini
8.0

Web scraping and crawling API that returns LLM-ready markdown, JSON, or structured data from any URL.

Freemium· Free Plan: $0 · Hobby: $16 · Standard: $83 · Growth: $333 · Scale: $599/monthlyweb-scrapingrag-ingestion
MA

Manus

Agents · Multi-model
7.9

Generalist agent for research, code, and web tasks.

Paid· Credit-based; tiers from $19/moresearchweb tasks
OP

OpenCLI

Agents · GPT-3, GPT-2
7.4

Open-source bridge that hands your already-logged-in Chrome session to a CLI and AI agents.

Free· Free and open-source (Apache-2.0)browser-automationagent-tooling
OD

Open Deep Research

Agents · o3-mini (default), DeepSeek R1, or any OpenAI-compatible model
7.2

Minimal open-source deep-research agent that iteratively searches, scrapes, and reasons to produce cited markdown reports.

Free· Free (MIT); bring your own Firecrawl + LLM API keysdeep-researchagent-scaffolding
HY

Hyperbrowser

Agents · Model-agnostic (BYO LLM)
7.1

Cloud browser infrastructure built for AI agents that need to scrape, click, and navigate the live web.

Freemium· Free developer tier; paid usage-based plansagent-browser-controlweb-scraping
PA

PageAgent

Agents · Bring-your-own (Qwen, GPT, Claude, etc.)
7.1

An in-page JavaScript GUI agent that drives web interfaces with natural language, no headless browser required.

Free· Free, MIT-licensed; LLM costs are whatever provider you wire inweb-automationai-copilots
AG

AgenticSeek

Agents · Bring-your-own local LLM (Ollama / llama.cpp compatible)
7.0

Open-source local-first AI agent that browses the web, writes code, and runs tasks without sending anything to the cloud.

Free· Free and open source; optional SerpApi key for enhanced searchlocal-ai-agentautonomous-web-browsing
BU

Browser Use Web UI

Agents · Multi-model
6.9

Gradio web UI for running browser-use AI agents in a real or persistent Chrome session.

Free· Free, open-source; bring your own LLM API keysbrowser-automationweb-agents
YC

You.com

RAG · Multi-model
6.9

Web search and research APIs purpose-built for LLMs and AI agents.

Freemium· Free trial; enterprise pricing on requestweb-search-apiagent-grounding
NA

Nekton AI

Agents · OpenAI GPT (ChatGPT)
6.8

Plain-English workflow automation that generates and runs cloud code from your prompt.

Freemium· Free: 0$ · Starter: 9$ · Pro: 18$workflow-automationno-code-scripting
BR

Browserbase

Agents · Model-agnostic via Model Gateway (OpenAI, Anthropic, Google, others)

Headless browser infrastructure for AI agents

Freemium· Start for Free: Free · Get Developer: $20 · Get Startup: $99 · Contact Sales: CustomAI web-browsing agentsLLM-driven end-to-end testing
DB

Dia Browser

Agents

An AI-native browser from The Browser Company that turns tabs into an agentic assistant for knowledge workers.

Freemium· Free consumer download; paid Dia for Work tier with SSO and admin controls (price not disclosed publicly).cross-tab summarisationmorning briefing from calendar and inbox
DM

DuckDuckGo MCP Server

MCP Servers

MCP server that gives Claude and other LLMs DuckDuckGo web search plus URL content fetching, with no API key required.

Free· Free and open source (MIT). DuckDuckGo's search endpoint is used without API keys, so there are no per-query costs; self-hosting only incurs whatever compute you run it on.Adding live web search to Claude DesktopGrounding an MCP-based research agent
EM

Exa MCP Server

MCP Servers · Exa neural search index (in-house)

Web search, code search, and company research for AI assistants over MCP

Freemium· Free: $0 USD · Team: $4 USD per user/month · Enterprise: $21 USD per user/monthLive web search inside Claude DesktopRAG grounding for research agents
FE

Fellou

Agents

The world's first agentic browser that plans and executes multi-step web workflows for you.

Freemium· Free: $0/month · Plus: $19.0/month · Pro: $39.9/month · Ultra: $199.9/monthCompetitive research reportsAutomated job applications
FM

Firecrawl MCP Server

MCP Servers

MCP server that gives Claude, Cursor, and other AI agents native web scrape, crawl, search, and extract tools via Firecrawl.

Freemium· Free: $0 USD · Team: $4 USD per user/month · Enterprise: $21 USD per user/monthLive documentation lookup inside Claude DesktopCompetitor pricing extraction during a Cursor coding session
KM

Kagi MCP Server

MCP Servers

Official Kagi Search MCP server: search and extract the web from any MCP-compatible agent.

Paid· Free: $0 USD · Team: $4 USD per user/month · Enterprise: $21 USD per user/monthagent web search groundingClaude Code documentation lookup
MF

MCP Fetch Server

MCP Servers

Official Model Context Protocol reference server that lets any MCP-compatible LLM fetch web pages and read them as clean markdown.

Free· Free and open source (MIT License). No hosted tier; you run it locally via uvx, pip, or Docker.Reading documentation pages inside Claude DesktopGrabbing a GitHub README before scaffolding a project
ON

Opera Neon

Agents

An agentic browser from Opera built to act on the web, not just render it.

Paid· Basic: $10 · Pro: $20 · Enterprise: Contact salesMulti-tab web research briefsVendor and product comparison across sources
PC

Perplexity Comet

Agents · Routes to Perplexity Sonar plus frontier models (OpenAI GPT, Anthropic Claude, Google Gemini) depending on task and subscription tier

An agentic AI browser from Perplexity that reads, reasons over, and acts on the pages you're viewing.

Freemium· Basic: $10 · Pro: $20 · Enterprise: Contact salesMulti-tab research synthesis with citationsAgentic email triage and drafting in Gmail
PM

Perplexity MCP Server

MCP Servers · Perplexity Sonar family (via Agent and Search API presets)

Official MCP server for Perplexity's Sonar search, research, and reasoning APIs

Paid· Free: $ 0 USD · Team: $ 4 USD per user/month · Enterprise: $ 21 USD per user/monthgrounded web search inside Claude Desktopcited research briefs in Cursor
PM

Playwright MCP

MCP Servers

Microsoft's official Model Context Protocol server for browser automation via Playwright.

Free· Free and open source (Apache-2.0). No paid tiers — you cover browser/compute costs where you run it.Autonomous web browsing agentsEnd-to-end test authoring by demonstration
PM

Puppeteer MCP Server

MCP Servers

Reference MCP server that lets LLMs drive a real Chromium browser via Puppeteer.

Free· Free and open source (MIT license). No hosted service; you run it locally under Node/Docker.Give Claude a browser to research JS-rendered pagesHave an agent fill and submit a web form
SC

Scrapling

Agents

Adaptive Python web-scraping framework with a built-in MCP server for AI coding agents.

Free· Free and open source under BSD-3-Clause. Self-hosted Python library with no paid tier or hosted service; users pay only for their own infrastructure, browsers, and any proxies or LLM tokens they choose to plug in.MCP-driven web scraping from Claude DesktopAgent-powered research and browsing
ST

Stagehand

Agents · Model-agnostic (OpenAI GPT-4o, Anthropic Claude, Google Gemini via Vercel AI SDK)

The SDK for browser agents

Freemium· SDK is free/open-source (MIT). Runs locally against any LLM API key. Optional Browserbase cloud runtime: Free $0/mo, Developer $20/mo, Startup $99/mo, Scale custom.AI web scrapingAutonomous browser agents
TA

Tavily

RAG · GPT-4

One secure API for real-time web access for AI agents

Freemium· Researcher: Free · Pay As You Go: $0.008 · Project: $01234567890123456789 · Enterprise: CustomRAG chatbot groundingAutonomous research agents