QuantProbe

Physics-based calculator that predicts LLM decode speed, memory fit, and quantization quality on any hardware.

Free· Free and open source; install with `pip install quantprobe` or use the hosted web calculator.Evaluation

Best for

ML engineers, homelab builders, and inference-infra teams sizing hardware or picking quantization levels for local LLM deployment.

Skip if

End users looking for a chatbot, hosted inference, or a fine-tuning platform — QuantProbe only predicts performance, it does not run models.

QuantProbe is an open-source predictive calculator that tells you, before you download a single weight, whether a given LLM will fit on your hardware and roughly how fast it will decode. It applies a fitted "tiered decode law" — a mathematical model derived from real-world benchmarks across 7B to 753B parameter models — to forecast tokens-per-second based on memory bandwidth, model architecture (dense vs. MoE), quantization scheme, and KV-cache depth. Alongside the speed model, a "depth-aware recipe" estimates the perplexity cost of different quantization choices so you can weigh throughput against quality without running the experiment.

The interactive web tool lets you plug in a target machine (VRAM, system RAM, measured or estimated bandwidth) and a candidate model, then produces a chart of predicted speed against validated measurements and a memory-fit verdict for various context lengths. A companion Python package (`quantprobe`) exposes calibration probes so you can measure your specific machine's actual bandwidth and feed it back into the model for tighter predictions. All laws, probes, calibration recipes, and raw benchmark logs are published in the GitHub repository, making the tool auditable and extensible.

Typical workflows include: sizing a homelab or workstation build before buying GPUs, deciding between Q4/Q5/Q8 quantization for a target latency budget, comparing dense vs. MoE checkpoints on the same hardware, and sanity-checking vendor claims about inference throughput. Reported accuracy is within roughly plus or minus 25% for off-VRAM (partially offloaded) scenarios, tighter for pure-VRAM runs. It is aimed at engineers and researchers who need to reason about inference economics without spending hours on trial-and-error deployments.

Editor's take

A refreshingly rigorous corner of the local-LLM ecosystem. QuantProbe treats inference sizing as a physics problem instead of vibes, and the fact that every coefficient and raw log is in the repo makes it trustworthy in a way most benchmark sites aren't. If you're spec'ing a GPU or arguing about Q4 vs. Q5, start here before you argue on Reddit.

— The AI Tool Bible editorial team

Pros

✅ Predicts decode speed and memory fit before you download or deploy a model, saving hours of trial-and-error
✅ Validated against real measurements across 7B to 753B parameter models, including MoE architectures
✅ Quantifies the perplexity cost of quantization, letting you trade speed against quality with numbers rather than intuition
✅ Fully open source — laws, probes, recipes, and raw logs live in the repo and can be audited or extended
✅ Ships both a browser calculator and a `pip install`-able Python package with hardware calibration probes
✅ Explicit density coefficients for GPU dense (~0.62) vs. MoE (~0.38) tiers make assumptions transparent

Cons

⚠️ Prediction envelope is roughly plus or minus 25% for off-VRAM scenarios, so it is a planning tool rather than a benchmark replacement
⚠️ Coverage is centered on llama.cpp-style decode; training, prefill, and batched serving throughput are out of scope
⚠️ Small, single-maintainer project — no SLA, support channel, or guarantee of long-term upkeep
⚠️ The web UI assumes familiarity with quantization schemes, KV cache, and bandwidth concepts, so newcomers face a learning curve
⚠️ Not itself an LLM — practitioners looking for a generative or agentic tool will find only a calculator here

Use cases

GPU and workstation sizing for local LLM inferenceQuantization tier selection (Q4/Q5/Q8) under a latency budgetPredicting tokens-per-second for a candidate model on target hardwareComparing dense vs. Mixture-of-Experts checkpoints on the same boxEstimating KV-cache memory at long context lengthsCalibrating a machine's real memory bandwidth for tighter forecastsSanity-checking vendor or community throughput claims