VisualWebArena

VisualWebArena is an academic evaluation suite built by Carnegie Mellon researchers to measure how well multimodal agents can plan and act on real-looking websites where vision actually matters. It bundles 910 tasks spread across three self-hosted environments, Classifieds, Shopping, and Reddit, plus execution-based scoring and a Set-of-Marks rendering trick that overlays bounding boxes and IDs onto interactive elements so models can reference them by index.

The target user is anyone building or judging web-browsing agents: lab researchers, frontier-model evaluators, and agent-framework authors who need a harder, more visual successor to the original WebArena. Tasks deliberately require looking at images, layouts, and product photos rather than scraping DOM text, which exposes the gap between text-only LLM agents and true multimodal ones. Everything is free and open source under the web-arena-x GitHub org, with a public leaderboard tracking GPT-4V, Gemini, Claude, and open-weight runs.

It is not a hosted product or API, so you stand up the Dockerized environments yourself and wire in your agent and model keys. Expect a serious setup before you get a number out, but the payoff is a citation-grade score the field already recognizes from the ACL 2024 paper.