aitherkvcache — the crucible

Python · arXiv:2504.19874

◉ You are in The Crucible — How small can it get?

KV cache, sub-byte — and portable

Three engines for LLM inference memory. Compress a cache to 2–4 bit with no calibration pass, or convert one model's cache into another model's so the receiving model skips prefill entirely.

Get started View source

python

from aither_kvcache import TurboQuant

tq = TurboQuant(head_dim=128, bits=4, device="cuda")

packed, norms = tq.encode(kv_vectors)
decoded = tq.decode(packed, norms)
#   [..., 128] fp16  ->  [..., 64] uint8 + [...] f32

2–4bits

3.8–7.1×compression

3engines

2.4version

What it costs you, in GB

The KV cache is usually the reason you cannot raise the context or the batch size. Put your model in and see the number move — this is the real formula, not an illustration.

kv cache calculator

layers

kv heads

head dim

context tokens

batch

quant bits

fp16 cache--

compressed--

freed--

ratio--

compressedfp16 baseline

Defaults are a 70B-class model with grouped-query attention at 128K context. Bits below 4 are the aggressive end of the TurboQuant range; measure quality on your own workload before shipping one.

Three engines

Pick the trade-off that fits: general-purpose vector quantization, spectral compression tuned to a model family, or moving a cache between models entirely.

turboquant

Vector quantization

2–4 bit quantization reaching 3.8–7.1× compression against FP16, provably near the information-theoretic bound.

streaming

No calibration pass

Works on streaming tokens with no calibration dataset and no offline profiling step before you can use it.

triattention

Spectral compression

Retains the top RoPE frequency pairs and scores through a trigonometric series without materializing full K/V — 14–26x at F=8–16. Which pairs carry the energy is a property of the model, so it is calibration-dependent: profiles ship for four families and anything else warns rather than falling back silently.

kvtransfer

Cross-model transfer

Fit per-(layer, head) linear maps that convert one model's KV cache into another's, so the receiving model can skip prefill. RoPE is stripped with the source schedule and re-applied with the target's; source layers are chosen per target layer by held-out R².

gate

Refuses bad mappers

A converted cache does not fail loudly — an approximate one makes the model produce fluent, confidently wrong text. So a mapper pack will not load without held-out reconstruction scores AND a downstream measurement, over a large enough sample, against a control arm that catches a mapper ignoring its input.

latent

MLA aware

Handles latent-attention models (DeepSeek V2/V3/V4), whose cache is a compressed latent plus a decoupled RoPE key rather than per-head K/V — captured at the compression projection, where it is position-free.

vllm

vLLM plugin

Install the vllm extra to slot into vLLM v0.15+ rather than rewriting your serving stack.

triton

Fused kernels

An optional Triton extra provides fused GPU kernels for the hot encode/decode path.

research

Published basis

TurboQuant implements Zandieh et al., arXiv:2504.19874 — the method is documented and citable, not a black box.

Quick start

Install the core library, or pull in the vLLM plugin and fused kernels as extras.

bash

pip install aither-kvcache
pip install "aither-kvcache[vllm]"
pip install "aither-kvcache[triton]"
pip install "aither-kvcache[all]"

Citation. TurboQuant implements Zandieh et al., arXiv:2504.19874. TriAttention is calibrated for the Qwen3.5 family; other RoPE bases need their own calibration. Cross-model transfer works within a model family — the two models must share a tokenizer, because the map converts position i to position i, and a mapper is only usable once it carries a downstream measurement that clears its own floor.

One estate, 8 doors

These projects are organs of the same system — the same identity plane, the same agent runtime, the same operational discipline. Follow any of them.

the osaitherium.comThe Living OS these four repos are cut from. Open, no login.pythonawdkBuild AI agent fleets — 3 lines, any backend, local or cloud.markdownawskillsPortable agent skills — self-contained procedures an agent loads on demand.markdownawknowledgeHow to run a coding agent so the result survives — the laws, with evidence.cudaaitherkvcacheNear-optimal KV cache quantization for LLM inference — sub-byte compression.powershellAitherZeroPowerShell 7+ automation framework — numbered, self-describing scripts.writingThe BlogHow all of it is actually built, measured and operated.pythonawrunA priority-aware queue and dispatcher for agentic runs and ad-hoc CI builds. It also judges whether the runner pool is big enough for the queue it is draining, and can ask a host to grow it -- reserving capacity is zero-sum, so a saturated pool needs more of it, not a different share of it.

The ecosystem this belongs to

Every Aitherium repository publishes an aither-manifest.json beside its page, and each surface reads all of them — so the network is browsable from any node in it. A repository that publishes nothing shows as unknown, never as zero.