Skip to main content
πŸ“– The AI Tool Bible
InfiBench preview image
InfiBench logo

InfiBench

βœ“ Text source-checked Nov 11, 2026

Source-checked against the vendor's site Β· rebuilt from the vendor's own pagesProfile updated Link checked How we rate

FreeΒ· Free and open source (CC BY-SA 4.0)(not source-checked)Evaluation6.9 / 10

In short

InfiBench is a free, open-source benchmark for evaluating code LLMs on 234 real-world Stack Overflow questions across 15 languages using four model-free metrics.

The vendor says that InfiBench is a comprehensive benchmark for code large language models that evaluates model ability on answering freeform real-world questions in the code domain. InfiBench comprises 234 carefully picked high-quality Stack Overflow questions covering 15 programming languages, largely following the natural question distribution of Stack Overflow.

The InfiBench framework integrates four types of model-free metrics for evaluating correctness: keywords matching, blank filling, unit testing, and dialogue similarity. The vendor says that five domain experts created the benchmark and annotated the correctness evaluation criteria.

The project conducts a systematic evaluation for over 100 latest code LLMs on InfiBench. InfiBench is fully open source and continuously expanding to foster more scientific and systematic practices for code LLM evaluation.

The vendor says that the benchmark currently supports only the Linux environment. The vendor says that users must convert or save model weights in Hugging Face Transformers format to run the benchmark.

From the vendor's pages

According to InfiBench: InfiBench comprises 234 carefully picked high-quality Stack Overflow questions covering 15 programming languages, Largely following the natural question distribution of Stack Overflow. Its pages list this strength: Comprises 234 high-quality Stack Overflow questions covering 15 programming languages. They also state this limitation: Supports only the Linux environment.

β€” The AI Tool Bible editorial team

Pros

  • βœ… Comprises 234 high-quality Stack Overflow questions covering 15 programming languages.
  • βœ… Integrates four types of model-free metrics: keywords matching, blank filling, unit testing, and dialogue similarity.
  • βœ… Provides a systematic evaluation for over 100 latest code LLMs.
  • βœ… Is fully open source and continuously expanding.

Cons

  • ⚠️ Supports only the Linux environment.

Facts

Platforms
linux (not source-checked)
Pricing
free - Free and open source (CC BY-SA 4.0) (not source-checked)
Open source
Yes

Where this tool sits

Use cases

Evaluating code LLMs on real-world questionsSystematic evaluation of code LLMsScientific and systematic practices for code LLM evaluation

Frequently asked

Is InfiBench free to use?
Yes, InfiBench is free and open source under the CC BY-SA 4.0 license. It is designed to foster scientific and systematic practices for code LLM evaluation without cost barriers.
What programming languages does InfiBench support?
The benchmark covers 15 programming languages. It comprises 234 high-quality Stack Overflow questions that largely follow the natural question distribution found on the platform.
How does InfiBench evaluate model correctness?
It integrates four model-free metrics: keywords matching, blank filling, unit testing, and dialogue similarity. Five domain experts created the benchmark and annotated the correctness evaluation criteria.
What are the system requirements for running InfiBench?
The benchmark currently supports only the Linux environment. Users must convert or save model weights in Hugging Face Transformers format to run the evaluation framework.
How many models has InfiBench evaluated?
The project has conducted a systematic evaluation for over 100 latest code LLMs. The benchmark is continuously expanding to support more scientific and systematic evaluation practices.

Compare InfiBench with similar tools

All alternatives β†’

Explore related

Reviews