Two compression engines for LLM inference memory. TurboQuant reaches 2–4 bit within 2.7× of the information-theoretic optimum, with no calibration data and no offline pass.
from aither_kvcache import TurboQuant tq = TurboQuant(head_dim=128, bits=4, device="cuda") packed, norms = tq.encode(kv_vectors) decoded = tq.decode(packed, norms) # [..., 128] fp16 -> [..., 64] uint8 + [...] f32
The KV cache is usually the reason you cannot raise the context or the batch size. Put your model in and see the number move — this is the real formula, not an illustration.
Defaults are a 70B-class model with grouped-query attention at 128K context. Bits below 4 are the aggressive end of the TurboQuant range; measure quality on your own workload before shipping one.
Pick the trade-off that fits: general-purpose vector quantization, or spectral compression tuned to a model family.
2–4 bit quantization reaching 3.8–7.1× compression against FP16, provably near the information-theoretic bound.
Works on streaming tokens with no calibration dataset and no offline profiling step before you can use it.
Retains the top RoPE frequency pairs and scores through a trigonometric series without materializing full K/V — around 10× with bounded error.
Install the vllm extra to slot into vLLM v0.15+ rather than rewriting your serving stack.
An optional Triton extra provides fused GPU kernels for the hot encode/decode path.
TurboQuant implements Zandieh et al., arXiv:2504.19874 — the method is documented and citable, not a black box.
Install the core library, or pull in the vLLM plugin and fused kernels as extras.
pip install aither-kvcache pip install "aither-kvcache[vllm]" pip install "aither-kvcache[triton]" pip install "aither-kvcache[all]"
These projects are organs of the same system — the same identity plane, the same agent runtime, the same operational discipline. Follow any of them.