AutotuneLLM

AutotuneLLM (marketed simply as 'Autotune') is an open-source optimization layer that sits between your applications and a local Ollama runtime, automatically tightening memory and latency behaviour without requiring any code changes on the caller side. Rather than fine-tuning model weights, it 'tunes' the runtime environment around the model: dynamic KV cache sizing that right-sizes the cache to the actual token count instead of Ollama's default 4,096-token allocation, system-prompt caching that pins system prompts in KV cache so they are only evaluated once, adaptive KV precision that drops from F16 to Q8 under memory pressure, model keep-alive to avoid repeated cold starts, and a live RAM-pressure manager that shifts context window and precision across four tiers as the host machine's memory fills up. It exposes an OpenAI-compatible API server, so existing OpenAI SDK code can be pointed at localhost with no rewrites, and ships a built-in monitoring dashboard at localhost:8765/dashboard plus an 'autotune proof' benchmark harness for verifying gains on your specific hardware. There is a Docker bundle that packages Ollama and autotune together, an MLX backend for Apple Silicon, and named performance profiles (fast, balanced, quality) for common workloads. The published claims are around 300 MB of RAM freed per request on tested models, up to 53% faster first-word latency, and 67% reduction in KV cache memory with no impact on generation speed or output quality. The target user is a developer running local LLMs on consumer hardware (Apple Silicon with 8 GB+ RAM is the sweet spot) who is bumping into RAM ceilings, slow prompt evaluation, or context-window thrash when serving models like qwen3.5:9b, gpt-oss:20b, or gemma4:12b to a local application or agent loop.