Three engines for LLM inference memory. Compress a cache to 2–4 bit with no calibration pass, or convert one model's cache into another model's so the receiving model skips prefill entirely.
from aither_kvcache import TurboQuant tq = TurboQuant(head_dim=128, bits=4, device="cuda") packed, norms = tq.encode(kv_vectors) decoded = tq.decode(packed, norms) # [..., 128] fp16 -> [..., 64] uint8 + [...] f32
The KV cache is usually the reason you cannot raise the context or the batch size. Put your model in and see the number move — this is the real formula, not an illustration.
Defaults are a 70B-class model with grouped-query attention at 128K context. Bits below 4 are the aggressive end of the TurboQuant range; measure quality on your own workload before shipping one.
Pick the trade-off that fits: general-purpose vector quantization, spectral compression tuned to a model family, or moving a cache between models entirely.
2–4 bit quantization reaching 3.8–7.1× compression against FP16, provably near the information-theoretic bound.
Works on streaming tokens with no calibration dataset and no offline profiling step before you can use it.
Retains the top RoPE frequency pairs and scores through a trigonometric series without materializing full K/V — 14–26x at F=8–16. Which pairs carry the energy is a property of the model, so it is calibration-dependent: profiles ship for four families and anything else warns rather than falling back silently.
Fit per-(layer, head) linear maps that convert one model's KV cache into another's, so the receiving model can skip prefill. RoPE is stripped with the source schedule and re-applied with the target's; source layers are chosen per target layer by held-out R².
A converted cache does not fail loudly — an approximate one makes the model produce fluent, confidently wrong text. So a mapper pack will not load without held-out reconstruction scores AND a downstream measurement, over a large enough sample, against a control arm that catches a mapper ignoring its input.
Handles latent-attention models (DeepSeek V2/V3/V4), whose cache is a compressed latent plus a decoupled RoPE key rather than per-head K/V — captured at the compression projection, where it is position-free.
Install the vllm extra to slot into vLLM v0.15+ rather than rewriting your serving stack.
An optional Triton extra provides fused GPU kernels for the hot encode/decode path.
TurboQuant implements Zandieh et al., arXiv:2504.19874 — the method is documented and citable, not a black box.
Install the core library, or pull in the vLLM plugin and fused kernels as extras.
pip install aither-kvcache pip install "aither-kvcache[vllm]" pip install "aither-kvcache[triton]" pip install "aither-kvcache[all]"
These projects are organs of the same system — the same identity plane, the same agent runtime, the same operational discipline. Follow any of them.
The Aither World is an operating system for agents — a Linux you can hand to one, the runtimes it works in, and the tools it works with. aitherkvcache is one of its 64 bricks — each installs on its own, runs offline, and needs no account. All 64 →