SGLang

SGLang is a production-grade serving framework for large language and multimodal models, built around aggressive throughput and latency optimizations like disaggregated prefill/decode, speculative decoding, a zero-overhead scheduler, and hand-tuned GPU kernels. You point it at a model (DeepSeek, Qwen, Llama, Mistral, GLM, GPT-OSS and friends) and get an OpenAI-compatible HTTP endpoint you can drop into existing clients, with scaling from a single GPU up to multi-node clusters.

It sits in the same competitive bracket as vLLM and TensorRT-LLM, and has become one of the go-to engines for teams serving large open-weight models in-house. NVIDIA, xAI, Oracle, LinkedIn, and Google Cloud have all shipped workloads on it, and it runs on NVIDIA, AMD, TPU, Ascend NPU, Intel XPU and even CPU backends. SGLang itself is free and Apache-licensed; the cost is the hardware you point it at and the ops effort to tune it.

This is infrastructure, not a hosted product. There is no SaaS dashboard, no managed inference SKU, no billing page. If you want a turnkey API you call from an app, you want Together, Fireworks, or Anyscale; if you want to own the serving stack and squeeze every token-per-second out of your own GPUs, SGLang is one of the strongest options available.