Skip to main content
📖 The AI Tool Bible

AI web scraping agent

Editorial picks for "ai web scraper".

32 tools

All Agents →
SC

ScrapeGraphAI

Agents · Multi-model (LLM, unspecified)
8.4

LLM-driven web scraping tools designed to extract structured data from any website and HTML content.

Freemium· Free Plan: $0/mo (one-time purchase); Starter Plan: $20/mo billed monthly; Growth Plan: $100/mo billed monthly; Pro Plan: $500/mo billed monthlyData analysisDataset creation
N8

n8n

Agents · Multi-model
8.3

Workflow automation platform combining AI capabilities with business process automation, offering code flexibility and no-code speed.

FreemiumBuilding AI agentsWorkflow automation
SK

Skyvern

Agents · Multi-model (OpenAI, Anthropic, Gemini, Ollama)
8.3

Skyvern automates browser-based workflows using LLMs and computer vision.

Freemium· Free plan; Hobby plan at $29/month; Pro plan at $149/monthAccounts payable invoice downloadingData extraction from websites
BU

Browser Use

Agents · Multi-model (BYO LLM; Claude in hosted Box)
8.2

Cloud browser infrastructure and hosted agents for AI automation, with stealth browsers available at $0.02 per hour.

Freemium· Free tier
SI

Sim

Agents · Multi-model (Claude, OpenAI, Google)
8.2

Open-source AI workspace for building, deploying, and managing AI agents.

Freemium· Free: $0; Pro: $25 per user/month; Max: $100 per user/monthStoring structured data for agentsManaging team uploads and agent outputs
EX

Exa

RAG · Proprietary neural + keyword search
8.0

Search API for AI agents providing real-time web data, deep research, and structured content.

Freemium· Free plan; Starter: Free, $10 credits every month plus a $10 onboarding bonusCoding agentsChatbots
FI

Firecrawl

RAG · Claude, Cursor, Windsurf, OpenAI, Gemini
8.0

Web data API to search, scrape, and interact with the web at scale.

Freemium· Free tier · from $19/moDeep research agentsSite crawling
MA

Manus

Agents · Multi-model
7.9

Generalist agent for research, code, and web tasks.

Paid· Credit-based; tiers from $19/moresearchweb tasks
OP

OpenCLI

Agents · GPT-3, GPT-2
7.4

OpenCLI is a browser automation tool for AI agents.

Free· Free and open-source (Apache-2.0)Browser automation for AI agentsCreating reusable commands for repeated workflows
OD

Open Deep Research

Agents · o3-mini (default), DeepSeek R1, or any OpenAI-compatible model
7.2

An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models.

Free· Free (MIT); bring your own Firecrawl + LLM API keysDeep research on any topicIterative research direction refinement
HY

Hyperbrowser

Agents · Model-agnostic (BYO LLM)
7.1

Cloud browsers, isolated sandboxes, and web data APIs for AI agents.

Freemium· Free plan: $0 to start, 5,000 credits included; Startup plan: $30 per month, 30,000 credits included; Scale plan: $100 per month, 100,000 credits includedWebsite automationIsolated code execution
PA

PageAgent

Agents · Bring-your-own (Qwen, GPT, Claude, etc.)
7.1

An in-page JavaScript GUI agent that drives web interfaces with natural language, no headless browser required.

Free· Free, MIT-licensed; LLM costs are whatever provider you wire inweb-automationai-copilots
AG

AgenticSeek

Agents · Bring-your-own local LLM (Ollama / llama.cpp compatible)
7.0

Open-source local-first AI agent that browses the web, writes code, and runs tasks without sending anything to the cloud.

Free· Free and open source; optional SerpApi key for enhanced searchlocal-ai-agentautonomous-web-browsing
BU

Browser Use Web UI

Agents · Multi-model
6.9

Gradio web UI for running browser-use AI agents in a real or persistent Chrome session.

Free· Free, open-source; bring your own LLM API keysbrowser-automationweb-agents
YC

You.com

RAG · Multi-model
6.9

Core AI infrastructure for maximizing accuracy while reducing latency and costs.

Freemium· Free plan; Web Search API Standard: $5 per 1k calls; Contents API: $1 per 1k pages; Answer API: $5 per 1k calls; Research Lite: $12 per 1k calls; Research Standard: $50 per 1k calls; Research Deep: $100 per 1k calls; Research Exhaustive: $450 per 1k calls; Research Frontier: $1,200 per 1k calls; Finance Research Deep: $110 per 1k callsReal-time web and news search for AI applicationsContent extraction from URLs
NA

Nekton AI

Agents · OpenAI GPT (ChatGPT)
6.8

Plain-English workflow automation that generates and runs cloud code from your prompt.

Freemium· Free: 0$ · Starter: 9$ · Pro: 18$workflow-automationno-code-scripting
BR

Browserbase

Agents · Model-agnostic via Model Gateway (OpenAI, Anthropic, Google, others)

Browserbase makes the web as reliable and programmable as APIs for AI agents.

Freemium· Free Plan: $0/mo; Developer Plan: $20/mo; Startup Plan: $99/moHealthcare insurance verification and claims processingFinancial services loan applications and compliance
DB

Dia Browser

Agents

An AI-native browser from The Browser Company that turns tabs into an agentic assistant for knowledge workers.

Freemium· Free tier · from $10/mocross-tab summarisationmorning briefing from calendar and inbox
DM

DuckDuckGo MCP Server

MCP Servers

MCP server providing DuckDuckGo web search and content fetching capabilities.

FreeFetching and parsing webpage contentProviding search results to large language models
EM

Exa MCP Server

MCP Servers · Exa neural search index (in-house)

Exa MCP for web search and web crawling

FreemiumWeb search for AI agentsContent fetching
FE

Fellou

Agents

Fellou is an AI browser that takes automated web action.

Freemium· Free: $0/month; Plus: $19.0/month; Pro: $39.9/month; Ultra: $199.9/monthGenerating research updates from notesData scraping and form filling
FM

Firecrawl MCP Server

MCP Servers

MCP server that brings Firecrawl's web search, scrape, and interaction capabilities to MCP-compatible clients, returning clean Markdown or structured data.

FreemiumWeb search with optional full page contentURL scraping to Markdown
KM

Kagi MCP Server

MCP Servers

Official Kagi Search MCP server: search and extract the web from any MCP-compatible agent.

Paid· Free: $0 USD · Team: $4 USD per user/month · Enterprise: $21 USD per user/monthagent web search groundingClaude Code documentation lookup
MF

MCP Fetch Server

MCP Servers

Official Model Context Protocol reference server that lets any MCP-compatible LLM fetch web pages and read them as clean markdown.

Free· Free and open source (MIT License). No hosted tier; you run it locally via uvx, pip, or Docker.Reading documentation pages inside Claude DesktopGrabbing a GitHub README before scaffolding a project
ON

Opera Neon

Agents

Opera Neon is a browser built to act.

Freemium· Free plan; Subscription plan at $19.90/monthShaping agendas and preparing deep researchEveryday AI conversations and creative tasks
PC

Perplexity Comet

Agents · Routes to Perplexity Sonar plus frontier models (OpenAI GPT, Anthropic Claude, Google Gemini) depending on task and subscription tier

An agentic AI browser from Perplexity that reads, reasons over, and acts on the pages you're viewing.

FreemiumMulti-tab research synthesis with citationsAgentic email triage and drafting in Gmail
PM

Perplexity MCP Server

MCP Servers · Perplexity Sonar family (via Agent and Search API presets)

Official MCP server for Perplexity's Sonar search, research, and reasoning APIs

Paid· Free: $ 0 USD · Team: $ 4 USD per user/month · Enterprise: $ 21 USD per user/monthgrounded web search inside Claude Desktopcited research briefs in Cursor
PM

Playwright MCP

MCP Servers

A Model Context Protocol server that provides browser automation capabilities using Playwright.

FreeExploratory automationSelf-healing tests
PM

Puppeteer MCP Server

MCP Servers

Reference MCP server that lets LLMs drive a real Chromium browser via Puppeteer.

Free· Free and open source (MIT license). No hosted service; you run it locally under Node/Docker.Give Claude a browser to research JS-rendered pagesHave an agent fill and submit a web form
SC

Scrapling

Agents

A Python web scraping framework combining fast HTTP requests, stealth browsers, parsing, and concurrent crawling.

FreeScraping static pages and rendering JavaScriptBuilding crawlers with proxy rotation and pause/resume
ST

Stagehand

Agents · Model-agnostic (OpenAI GPT-4o, Anthropic Claude, Google Gemini via Vercel AI SDK)

The SDK for browser agents

Freemium· SDK is free/open-source (MIT). Runs locally against any LLM API key. Optional Browserbase cloud runtime: Free $0/mo, Developer $20/mo, Startup $99/mo, Scale custom.AI web scrapingAutonomous browser agents
TA

Tavily

RAG · GPT-4

One secure API for real-time web access.

Freemium· Researcher plan is Free / month; Pay As You Go plan is $0.008 / credit; Project plan is $30 / monthWeb search, extraction, research, crawling, and mappingGrounding models with fresh web context