
InfiBench
β Text source-checked Nov 11, 2026Source-checked against the vendor's site Β· rebuilt from the vendor's own pagesProfile updated Link checked How we rate
In short
InfiBench is a free, open-source benchmark for evaluating code LLMs on 234 real-world Stack Overflow questions across 15 languages using four model-free metrics.
The vendor says that InfiBench is a comprehensive benchmark for code large language models that evaluates model ability on answering freeform real-world questions in the code domain. InfiBench comprises 234 carefully picked high-quality Stack Overflow questions covering 15 programming languages, largely following the natural question distribution of Stack Overflow.
The InfiBench framework integrates four types of model-free metrics for evaluating correctness: keywords matching, blank filling, unit testing, and dialogue similarity. The vendor says that five domain experts created the benchmark and annotated the correctness evaluation criteria.
The project conducts a systematic evaluation for over 100 latest code LLMs on InfiBench. InfiBench is fully open source and continuously expanding to foster more scientific and systematic practices for code LLM evaluation.
The vendor says that the benchmark currently supports only the Linux environment. The vendor says that users must convert or save model weights in Hugging Face Transformers format to run the benchmark.
According to InfiBench: InfiBench comprises 234 carefully picked high-quality Stack Overflow questions covering 15 programming languages, Largely following the natural question distribution of Stack Overflow. Its pages list this strength: Comprises 234 high-quality Stack Overflow questions covering 15 programming languages. They also state this limitation: Supports only the Linux environment.
β The AI Tool Bible editorial team
Pros
- β Comprises 234 high-quality Stack Overflow questions covering 15 programming languages.
- β Integrates four types of model-free metrics: keywords matching, blank filling, unit testing, and dialogue similarity.
- β Provides a systematic evaluation for over 100 latest code LLMs.
- β Is fully open source and continuously expanding.
Cons
- β οΈ Supports only the Linux environment.
Facts
- Platforms
- linux (not source-checked)
- Pricing
- free - Free and open source (CC BY-SA 4.0) (not source-checked)
- Open source
- Yes
Where this tool sits
Use cases
Frequently asked
- Is InfiBench free to use?
- Yes, InfiBench is free and open source under the CC BY-SA 4.0 license. It is designed to foster scientific and systematic practices for code LLM evaluation without cost barriers.
- What programming languages does InfiBench support?
- The benchmark covers 15 programming languages. It comprises 234 high-quality Stack Overflow questions that largely follow the natural question distribution found on the platform.
- How does InfiBench evaluate model correctness?
- It integrates four model-free metrics: keywords matching, blank filling, unit testing, and dialogue similarity. Five domain experts created the benchmark and annotated the correctness evaluation criteria.
- What are the system requirements for running InfiBench?
- The benchmark currently supports only the Linux environment. Users must convert or save model weights in Hugging Face Transformers format to run the evaluation framework.
- How many models has InfiBench evaluated?
- The project has conducted a systematic evaluation for over 100 latest code LLMs. The benchmark is continuously expanding to support more scientific and systematic evaluation practices.