
VisualWebArena
Not yet source-checkedOpen benchmark for evaluating multimodal web agents on realistic visual browsing tasks.
Machine-drafted from the vendor's site · not yet source-checkedProfile updated Link checked How we rate
In short
VisualWebArena is a free, open-source benchmark for evaluating multimodal web agents on realistic visual browsing tasks across three self-hosted environments.
Pick VisualWebArena if you are benchmarking a multimodal web agent and need a respected, execution-graded score the research community already reads.
Skip it if you want a hosted eval dashboard or a generic LLM benchmark without browser environments.
VisualWebArena is an academic evaluation suite built by Carnegie Mellon researchers to measure how well multimodal agents can plan and act on real-looking websites where vision actually matters. It bundles 910 tasks spread across three self-hosted environments, Classifieds, Shopping, and Reddit, plus execution-based scoring and a Set-of-Marks rendering trick that overlays bounding boxes and IDs onto interactive elements so models can reference them by index.
The target user is anyone building or judging web-browsing agents: lab researchers, frontier-model evaluators, and agent-framework authors who need a harder, more visual successor to the original WebArena. Tasks deliberately require looking at images, layouts, and product photos rather than scraping DOM text, which exposes the gap between text-only LLM agents and true multimodal ones. Everything is free and open source under the web-arena-x GitHub org, with a public leaderboard tracking GPT-4V, Gemini, Claude, and open-weight runs.
It is not a hosted product or API, so you stand up the Dockerized environments yourself and wire in your agent and model keys. Expect a serious setup before you get a number out, but the payoff is a citation-grade score the field already recognizes from the ACL 2024 paper.
This is the serious benchmark to beat for vision-grounded web agents in 2026. It is not friendly, you will fight Docker and proxy configs, but the score actually means something. Treat it as infrastructure for your eval pipeline, not a product.
— The AI Tool Bible editorial team
Pros
- ✅ 910 realistic tasks across Classifieds, Shopping, and Reddit environments
- ✅ Execution-based scoring, not LLM-judged fuzzy matching
- ✅ Set-of-Marks rendering makes element grounding tractable for VLMs
- ✅ Public leaderboard and reproducible Docker environments
- ✅ Recognized benchmark from ACL 2024, widely cited
Cons
- ⚠️ Self-hosted Docker setup is non-trivial to spin up
- ⚠️ No managed UI, API, or one-click runner
- ⚠️ Tasks are static, agents can overfit the fixed set
Facts
- Platforms
- api
- Open source
- Yes (license: MIT)
- GitHub stars
- 489 (checked Nov 13, 2026)
- Last GitHub push
- Nov 9, 2024
- First commit
- Jan 2024
- Model used
- Model-agnostic (GPT-4V, Gemini, Claude, open VLMs)
- Earliest Wayback capture
- 2022-09-06 (earliest archived capture of the site, not a launch date)
Where this tool sits
Use cases
Frequently asked
- How much does VisualWebArena cost?
- VisualWebArena is free and open source under an MIT-style research release. It is not a hosted product or API, so you must stand up the Dockerized environments yourself and wire in your own agent and model keys.
- What is the learning curve for setup?
- Expect a serious setup before getting a score. It is not a hosted dashboard; you must self-host the Dockerized environments for Classifieds, Shopping, and Reddit, and configure your specific agent and model integrations manually.
- Which AI models are supported?
- The benchmark is model-agnostic. The public leaderboard tracks performance for GPT-4V, Gemini, Claude, and open-weight vision-language models. You can wire in your own model keys to test any compatible multimodal agent.
- How does VisualWebArena differ from generic LLM benchmarks?
- It focuses on realistic visual browsing where vision matters, requiring agents to look at images and layouts rather than just scraping DOM text. It uses execution-based scoring and a Set-of-Marks rendering trick to overlay bounding boxes on interactive elements.
- Who is this tool best for?
- It is best for lab researchers, frontier-model evaluators, and agent-framework authors who need a harder, more visual successor to the original WebArena. It provides a citation-grade score recognized by the research community from the ACL 2024 paper.