KV cache analytics for production inference
Your cache hit rate isn't enough.
It is one ratio per instance, and four things are already gone by the time it reaches you. Which blocks missed. Whether a miss was a first compute or a block you had stored and then lost. Which rank it landed on. And what the other caches were holding at that moment. No per-instance metric carries any of them, and scraping it more often does not bring them back.
infertap subscribes to each vLLM engine's KV-cache event socket from outside the request path and measures those directly: recomputed prefill and duplicate KV footprint, charged to the rank that holds them rather than averaged over a node, each figure carrying the coverage it was observed at and stated as a lower bound. What the numbers mean for your fleet is yours to decide. We report what happened.
What you get
Three quantities, separated by cause.
Two of them are GPU time, one is GPU memory, and they are never added together.
They arrive as a dashboard, not a file: a fleet overview, each figure's trend over time, and a drill-down to the rank and workload behind it.
-
Evicted, then recomputed
Tokens that were stored, evicted, and then stored again on the same engine. Prefill the fleet paid for more than once, counted per rank rather than averaged over a node.
GPU time · per rank
-
Recomputed after cache clear
The same quantity, separated by cause: tokens recomputed after an operational flush rather than after eviction under pressure. Two different owners, kept apart because a single recompute figure cannot be told apart afterwards.
GPU time · per rank
-
Duplicate KV footprint
The same content resident on more than one rank at the same instant. It moves with your instance count and your rank shape. Prefix-aware routers already track where blocks live, because they use it to place requests, but none of them report it back to you as a quantity.
GPU memory · per rank
What becomes decidable
Each quantity is an input to a different decision.
Separated by cause, because the causes are not interchangeable and cannot be told apart once summed. What any of them warrants on your fleet depends on your traffic and your shape, and we would be guessing if we told you now. These are the decisions the quantities inform, not the decisions themselves.
- Evicted, then recomputed Capacity and policy Pressure and policy churn produce the same total and different distributions across ranks. This is that distribution, at the grain where the two stop looking alike.
- Recomputed after cache clear Restart behaviour Same GPU hours, different cause, usually a different owner: flushes follow restarts and config changes, not load. Once summed the two can never be separated again, so they never are.
- Duplicate KV footprint Prompt mass A 400-token system prompt across twenty caches is arithmetic you can do yourself. The prefix your stack assembles is not: tool schemas, few-shot blocks, retrieved context, moving with every framework upgrade and resident once per rank all the same. It is the one term you can cut once and have it count everywhere, instead of fighting the number of caches.
- The same, shared by part of your traffic Prefix-aware routing llm-d and AIBrix send matching prefixes to one instance, consolidating overlap between pods. Whether that is worth deploying depends on how much of yours is that kind.
- The same, inside one instance Rank shape Data parallelism replicates the whole cache per rank; tensor parallelism shares one and pays interconnect for it. Ten nodes at DP=8 hold their shared trunks eight times over, and no router reaches inside an instance today.
- A change you shipped Before and after Prompt rewrites, engine upgrades, capacity and routing changes. Every figure carries a daily trend, and a trend crossing a rules change shows you the seam instead of quietly summing numbers that were never comparable.
- Any of them Nothing Batch and offline fleets churn cache by design, so a high figure there is the shape of that workload rather than a problem with it. Whether it warrants action stays your call, and a flat baseline is what makes the next change visible.
Why scraping harder won't help
Four things are gone before a metric ever reaches you.
Your engine's cache metrics are aggregate, anonymous, per-instance and
instantaneous. Four properties are discarded inside the process before
anything reaches /metrics, and anything needing
one of them cannot be recovered by scraping more often.
Three more are facts about your deployment that no endpoint reports at all: how many instances sit behind a load balancer that does not route on prefix, whether any instance runs more than one data-parallel worker, and whether an agent framework assembles your prefix. What replaces them is in What it measures: the two numbers, what comes attached to each one, and the metrics you can scrape yourself.
- Identity
- which block: a histogram of lifetimes never becomes a token count
- Cause
- why: “evicted under pressure” and “flushed by a restart” land in the same bucket, with different owners and opposite fixes
- Elsewhere
- what the other caches held at that moment; every metric above is per-instance and cannot be joined afterwards
- History
- the events, not the summary
One identity, fleet-wide
Every engine seeds its block ids from a random root, so the same prompt looks like unrelated data in every cache: sibling workers on one node disagree, and a node does not even match itself after a restart. That is why duplication has never been countable. infertap derives its own identity from what a block holds, rooted at a fixed constant, so identical content is the same value across processes, restarts, nodes and engine versions. Your fleet becomes one thing you can count instead of a hundred that cannot be compared.
What you deploy
One tap per engine. One analyzer for the fleet.
Each vLLM instance gets its own subscriber, so a TP=4 node hosting two instances runs two taps. One socket per process is deliberate: a single process fanning in several sockets lets a busy one starve the rest. The analyzer is a single VM for the whole fleet, and to leave you stop the taps and delete a directory.
Hand this to your security review
- Subscribes to the vLLM KV-event sockets you list and opens nothing else. No inbound connections, and it never calls out to us.
- Never touches your inference API, holds no credential, and writes metadata-only records to one directory on local disk.
The full version, with each claim named against the test or the systemd directive that enforces it, is the trust page.
Transport is yours
Records land on local disk as Parquet, one file per sealed segment, so your existing tooling reads them without anything of ours in the path: rsync, an object store, a data lake that queries the files where they land, a mounted volume, or physical media if that is your posture. Nothing in that path is ours to trust, so an air-gapped fleet needs no exception made for it.
Get in touch
Interested?
Send your GPU count, your rank shape, and what your
/metrics endpoint shows. That is enough for us to tell
you what this would measure on your fleet, and to say so plainly if the answer
is not much.