The Life of a DeepSeek V4.1 Flash KV Cache in vLLM
Introduction
The goal of this blogpost is to walk through the whole lifetime of a deepseek-ai/DeepSeek-V4.1-Flash request as it relates to the KV cache. For the sake of blog length, we'll be assuming no KV offloading, but that will be the topic of future posts.
The main inner loop in vLLM looks deceptively simple and can stated succinctly as, "if there are requests, schedule and allocate space for those requests that get to run, run the forward pass, sample, update book-keeping state, return outputs."
We pick up our friendly (text-based) DeepSeek request in InprocClient::add_request. At this point, our request, lets call it Rocco, has already gone through several converation turns and has already been tokenized and validated and is now of type EngineCoreRequest. Rocco is also decidedly large. 50k tokens large, in fact. We'll assume that Rocco has no structured outputs that we need to deal with as well, else we'd need to compile Rocco's grammar (i.e. his structured output specification). At this point Rocco is to be added to the internal runtime's scheduler.
Async vs sync scheduling
Based on the compatibility of your vLLM config, the scheduler defaults to AsyncScheduler as opposed to the synchronous Scheduler. The core difference between the two is that async-scheduling allows the engine to schedule the next batch ahead of the previous batch's returned model outputs, which helps to avoid GPU bubbles. Importantly, what is being overlapped is the CPU-side book-keeping (things like KV cache management) and the actual device-side execution.
Thinking through the engine loop, we generally see something like:
schedule -> execute -> get output -> update state -> repeat
Say we start with some sequence Explain why cats > dogs. The answer, though obviously true, may necessitate many forward passes. In the synchronous loop, we would only schedule the next token (call it token B) to be executed after the first token (call it token A) has been sampled. But the act of reserving KV cache slots and scheduling token B does not depend on what the value of token A actually becomes. To schedule token B we just need to know how many tokens (both regular AR token and speculative decoding) get produced per step, as this comment makes clear.
Let’s walk through an example (with MTP) to make this more clear. Say forward-iteration N verifies five draft tokens, q1 through q5, and produces the next regular autoregressive token a, either as a correction token after a rejection or as a bonus token if every draft is accepted. The MTP drafter then generates the next five drafts, p1 through p5. Iteration N+1 will process six input positions, i.e. the still-unprocessed token a, followed by p1 through p5.
The scheduler can reserve KV capacity for these six positions (a, and p1 through p5) without knowing their token IDs, which remain on the worker, device-side. _prepare_input_ids then copies the previous iteration’s sampled token into the ordinary input position and the newly proposed drafts into the five speculative positions.
Now, when the CPU schedules iteration N+1, the scheduler does not yet know how many q draft tokens from iteration N were accepted, however its computed-token "frontier" must assume that all of them were accepted, which implies setting the new inputs (a, p1 through p5) after all five q positions when calculating how much KV capacity Rocco needs. The actual device-side worker will have the true acceptance result from iteration N, and so before executing the next batch it may need to take corrective actions, which determine the logical positions and KV slot mappings for a and p1 through p5. For instance, if all five q draft tokens were rejected, then those six new inputs would begin five positions earlier than the CPU scheduler predicted. That is to say, the scheduler may have reserved enough physical capacity for the optimistic case, but the worker writes the new state at positions corresponding to the actual accepted sequence.
In the case where the scheduler allocated a new block for iteration N+1, but the draft-token-correction moved the current iteration's starting position back so that the new block would not get used, that new block remains in Rocco's block table, not returned to the free pool and so is unavailable to other requests, ready for him to grow into without vLLM needing to allocate more KV blocks.
Rocco gets scheduled
If Rocco squints he can make out the name of line that he's quite far back in: "Waiting". Beside him are a couple other lines. To his right, looking at their watches, wait a forlorn bunch of requests in their own line named "Skipped Waiting". Some mention that they're waiting for their structured output grammar to finish compilation (RequestStatus.WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR), while others mention that they've been through this process before but now need to wait for their KV caches on this ride (RequestStatus.WAITING_FOR_REMOTE_KVS). On his left, with wind blown hair from just having taken a ride, another line whose sign reads "Running". He notices that in each line, all are ordered oldest to youngest -- he's been told the lines are first-come-first-serve.
The "Running" line is the first to move. A lot of bickering at the front of the line as each request must meet a lot of requirements.
For instance, as we are in the async-scheduling regime, one request cannot be scheduled to run (call it N+1) because its Nth iteration would cause the full output token sequence to be equal to the request's maximum length, even accounting for the speculative decode tokens all being rejected.
Another request, with multimodal features, is told that its image embeddings are still being transferred into the local encoder cache, and so it cannot run. The EncoderCacheManager, which manages the device-side worker's local encode-cache capacity, references, allocation, eviction, etc, and the ECConnector, which transfers encoder-cache entries outside the local cache, closely mirror the KV management we will speak more to below.
This is to say that being in the "Running" line does not guarantee that a request will actually run on every engine step. It merely means that the request has been admitted and still has live engine state. There are any number of reasons it might not get a turn -- its previous asynchronous step may already guarantee that it is finished, there may not be enough KV space, some higher-priority request may have pushed it aside, or its encoder state may not yet be ready. The full list of bickering can be found here.
Now the "Waiting" line begins to move. When Rocco finally reaches the front, the first question he is asked is how much of him has already been seen before -- i.e. whether or not he has a prefix-cache hit.
A digression on DeepSeek V4.1 Flash's attention
I find it useful to separate DeepSeek's attention along three axes. MLA latent vectors, rather than conventional per-head KV, is how KV gets represented. A local sliding window plus top-k positions from global history answers which past positions are visible to a given token. And some layers produce, and others reuse, global state in a few specific modes. vLLM's attention implementation can be found here and DeepSeek's config can be found here.
The first attention mechanism that DeepSeek uses is a local sliding-window MLA of head dimension 512, which they call DeepseekV4SWACache and defaults to a lookback of 128 tokens. The local cache stores one MLA latent per token per layer, where the cache specification is defined by the SlidingWindowMLASpec.
The second attention mechanism is a compressed MLA allowing some layer to access any other previous layer's MLA latent, which we'll refer to as a global cache. This comes in three flavors. A compress_ratio of 0 means that attention for a specific layer does not access this global cache and its attention will only use the local sliding window. A compress_ratio of 1 refers to layers that store state per single token and have global, top-k access to previous layers' state. A compress_ratio of 2 refers to layers that pool two consecutive token latents into one KV latent via a learn softmax gate -- ratio 1 has no gate and no pooling.
Now, it is the role of the DeepseekCompressor to project each token's hidden state into a candidate KV latent and a scalar gate score, and its cache is backed by the CompressorStateCache. The CompressorStateCache is necessary because, when pooling, you'll need to store some token's latent and gate score until the subsequent token arrives. The CompressorStateCache is backed by a state dimension of 2 * head_dimension == kv_state + score_state == 1024 (by default). Once the compressor emits a completed latent z_j, it feeds to related storage paths. One is the main global KV cache, where the latent goes through a few functions before being cached. The other is the DeepseekV4Indexer, which derives a small retrieval key from the latent and whose job it is to estimate global cache relevance cheaply. The head_dim of the DeepseekV4IndexerCache is of size 132 bytes (128 fp8 bytes and 1 scale byte) if not using 4-bit KV and 68 bytes (64 packed fp8 values plus 4 scales) if you do use FP4 KV.
Now we can speak to how DeepSeek uses Compressed Sparse Attention 2 (CSA2) and the three roles they define -- full, reindex, and reuse. The terminology makes sense if you just ask the two questions of "does this layer create new global main KV cache" and "does this layer calculate new top-k selection". In full mode, a layer produces a new compressed main cache and indexer K cache as well as calculates new top-k indices. Both reindex and reuse modes share cache by reusing the cache from the source layer at or shallower than itself. Reindex mode will reuse the current source's main compressed cache and source's indexer K cache, but will produce a new indexer query and recalculate top-k positions for its own layer (a bit more on that below). Reuse mode merely reuses the current source's main compressed cache and also reuses the latest top-k selection.
Let's now speak a bit to the indexer itself. Suppose some context has one million cached global states -- how might some state know which top-k (i.e. 512) previous states are relevant? The smaller index keys (128 bytes in size) are used by the new token's query heads as approximate relevance scores. For a given cached index key k_j, the learned relevance score is a function of the scores for k_j and each of the query heads. We then retrieve the main-cache states (512 bytes in size) of the top-k indices. Now, looking back to the previous paragraph, by reusing the indexer cache you avoid the costly top-k and sparse retrieval of your latents.
Now, reindexing over that entire one million cached state is expensive. It is the role of the heirarchical indexer to reduce this cost, and it looks very much like the top-k routing done in the DeepSeek-V3 MoE if you are familiar with that. Recall the scores mentioned above derived from the DeepseekCompressor. Blocks hold candidate_block_size (by default 8) position scores and the block-score itself is the max of these candidate_block_size positions. The top candidate_topk_blocks (by default 2048) are chosen (which implies 2048 * 8 == 16,384 positions). So the full CSA2 role provides a broad set of potential token positions, ideally having high recall so that the positions chosen are what the deeper layers find important. Deeper reindexing layers then select their own top-k (512) from this broader set of 16,384 positions.
How vLLM Caches DeepSeek Attention
Before we can find Rocco's prefix, we need to speak to how Rocco's prefixes get stored. Rocco's caches, as described above, are more complicated than merely KV tensors per token per layer as you would find in regular attention. In order to understand how vLLM handles Rocco's request, we need to look in get_kv_cache_groups. A group, which are the cache-owning modules in our model (i.e. the DeepseekV4IndexerCache, DeepseekV4SWACache, and CompressorStateCache), can share one logical block table because they use the same positions in a request's token history. For instance, given a physical block (say block ID 18), its group may be composed of the layers that globally produce KV (like layers 2 or 8), as well as layer 2's indexer cache. For us, each layer owns some sliding-window cache (so this might be a separate group), and only the full CSA2 layers (there are four of them) own both a global KV and index K (a separate group from the SWA cache).
Now the question becomes, how do we group our caches given that some have local spatiality (i.e. only the last 128 tokens or 2 tokens) and some have global spatiality. It is the role of _get_packed_kv_cache_groups to answer that. Given that a group requires caches to use the same positions, DeepSeek V4.1 will produce 3 groups (called buckets in _get_packed_kv_cache_groups). Bucket A might contain the full-context state for the four layer's main MLA caches and indexer K caches. Bucket B might contain the sliding window state where there is one SWA cache for every target/draft layer. And bucket C might contain the compressor state for compress-ratio-2 layers. Calculating the actual block-size then is a function of the sizes of each of the bucket sizes, where the widest group (in bytes) determines the number of bytes per block.
Conventionally, for a regular transformer with standard attention, there would exist one cache group where an allocator block represents the KV for N tokens across all layers, each layer contributing its own page of data, and all layers sharing the same block table entry. These per-layer pages don't necessarily need to be physically adjacent in every memory layout, merely that one block ID allocates and identifies them together. However, with hybrid attention, vLLM relaxes this constraint, where "across all layers" becomes "across all members of this group."
Rocco's cache hunting season
Before Rocco was placed into the waiting line, he was given, among other things, a long list of hashes. Each hash, he has been told, depends not only on the 32 token-sized chunks being hashed but also the parent hash, as shown here. As Rocco is 50k tokens long, he'll have 50,000 // 32 == 1562 hashes.
Importantly, the common resume alignment uses the least common multiple, whereas the hashing interval uses the greatest common divisor. This is to say that, intuitively, in order to resume from a given position then that position must be a point at which all caches can resume from. The hashing interval, using the greatest common divisor, ensures that every cache group’s block boundary lands on a hash boundary, so that, for instance, hashing every 32 tokens gives the SWA cache a hash for each block, while a global cache with 128 token-blocks uses every fourth hash.
Now, given our global cache’s blocks each cover 128 tokens, but we're hashing every 32 tokens, we use every fourth hash to identify those blocks. Because each hash incorporates its parent, that fourth hash already identifies the entire prefix through the end of the 128-token block. To find the actual physical block, we look up this prefix hash together with the cache group ID, which is performed by BlockPool.get_cached_block.
It is the role then of the KVCacheCoordinator to find the longest cache hit. In our case, the global main cache, owned by KV source layers, needs all tokens in chunks of size 128, whereas sliding-window only needs the last 128. So, in order to find a valid resume point, the KV manager supporting global cache will scan blocks left to right, stopping at the first missing block. The sliding window manager will then search right to left because it only needs the last 128 tokens cached.
If the global cache can take him through 32,768 tokens, but the sliding-window cache has no usable window there and can only take him through 24,576, then 24,576 is where Rocco must resume. The coordinator finds a boundary that every participating cache group can support, reducing the candidate and checking again as needed. It returns the blocks for each group alongside this common number of tokens that Rocco can skip computing. The global indexer cache follows the same prefix-availability rules as the global main cache, but using FullAttentionManager. The compressor’s temporary ring state opts out of prefix caching and does not participate in this search.
Rocco gets allocated
So, lets say we're able to find 32,768 reusable prefix tokens, and we want to further prefill 1,024 tokens on this step. It is the role of allocate_slots to ensure that Rocco can reference this cached prefix and has storage for the 1,024 tokens that he is about to compute. For a global cache group, we have a reusable prefix of 32,768 / 128 = 256 existing blocks and 1,024 / 128 = 8 new blocks of scheduled work. Each SWA group needs four cached 32-token blocks immediately before the resume point, followed by 32 new blocks for this step. Each compressor ring group needs only one block. The coordinator asks each group's manager how much capacity this requires and adds up their answers.
One subtlety here is that finding a cached prefix does not necessarily mean Rocco holds references to those blocks. A cached block with ref_cnt == 0 is still in the free queue, available to be used by any request. So, the capacity calculation must account for both the cached blocks Rocco will claim and the additional blocks he will need for new computation. Once there is enough room, allocate_new_computed_blocks attaches the reusable prefix blocks to Rocco’s block tables. Through BlockPool.touch, their reference counts increase and they are removed from the free queue, so some unrelated allocation can’t rugpull Rocco by repurposing the blocks he's using.
With the prefix protected, allocate_new_blocks gets storage for the next 1,024 tokens and the compressor rings. These blocks also come from the free queue, but Rocco will write new state into them, so BlockPool.get_new_blocks removes any old prefix-cache identities before assigning those blocks to him.
Rocco now has both sides of what he needs -- i.e. existing state to read and storage for new state to write. Eligible blocks are also registered under his prefix hashes for reuse, but the actual new KV will be written during the upcoming forward pass.
Rocco finally takes his turn
Now that Rocco has all the blocks that he needs, he can finally take his turn and send his next 1,024 tokens to the worker. Alongside these tokens, the worker receives Rocco's positions and block tables, which contain the information needed to actually find his KV.
It is useful here to distinguish between a block table and a slot mapping. The block table tells us which physical blocks back some part of Rocco's token history, whereas the slot mapping tells us the exact locations into which the state produced by the current forward pass should be written. This mapping will depend on the type of cache that we're writing to -- for instance, the local SWA cache stores one state per token, while a ratio-2 global cache stores one state per pair of tokens. It is the role of the sparse MLA metadata builder to account for this by constructing a compressed slot mapping when necessary.
Now, let's follow Rocco through one of the layers responsible for producing new global state. In _prepare_and_attn, the layer prepares Rocco's attention queries and writes the new per-token state into that layer's local SWA cache. Alongside this, the compressor takes the newly produced state and turns it into global MLA latents. For a ratio-2 source layer, each completed pair of tokens produces one latent, whereas a ratio-1 source produces one latent per token (and therefore does no pooling). This is all written into the new blocks that we just allocated for Rocco.
Once the compressor has produced a latent, it needs to add both the actual global state and the smaller indexer key used to find that state later. DeepseekCompressor.insert_cache transforms and writes the latent into the main global cache, while the indexer produces its smaller retrieval key and prepares the query used to score previous keys. Thinking to a library analogy, we are adding both a new book and its corresponding catalog entry. The indexer can then find the globally relevant positions, and sparse attention uses those indices to read the actual latents from the main cache. These globally selected states are attended to alongside the states in Rocco's local 128-token window.
Now, again speaking to CSA2, we do not necessarily produce a new global cache or perform a new global search at every layer. A reindex layer reuses the most recent source layer's global main and indexer caches, but produces a new query and calculates its own top-k selection. A reuse layer reuses both the source's global caches and the existing top-k selection. This is to say that sharing global state avoids storing another copy of the same global history, and sharing the top-k avoids performing another indexer search.
Rocco keeps going
Rocco has now moved his computed-token frontier from 32,768 to 33,792, and the scheduler will continue giving him prefill chunks until he makes it through the rest of his 50,000 token prompt. As he moves forward, each of his caches will behave a bit differently. The global main and indexer caches continue accumulating history, whereas old SWA blocks can be released by remove_skipped_blocks once they fall entirely behind the window, and the compressor ring is reused to carry unfinished pooling work across steps. This is to say that the global cache holds completed state that may be selected later, while the SWA cache only needs the relevant local history and the compressor ring only needs enough state to finish producing the next compressed latent.
Eventually Rocco finishes prefilling all 50,000 tokens and samples his first output token. This is where the async and MTP behavior discussed earlier comes back into play, where the scheduler can reserve space before it knows exactly how many speculative tokens will survive.
What's in Rocco's wake
At some point Rocco will reach some stopping condition and finish. At this point, KVCacheManager.free releases the references that Rocco holds to his blocks. BlockPool.free_blocks decreases its reference count, but a block reaching ref_cnt == 0 does not necessarily mean that its cached state immediately disappears.
As we spoke about above, if the block was registered under a prefix hash, its contents and hash identity can remain intact while it sits in the free queue. The block is in a state whereby nobody currently owns it, but a future matching request can still find and claim it. Of course, it is also available for some unrelated allocation to repurpose, at which point its old prefix identity is removed and its contents can be overwritten.
What Rocco leaves behind will depend on the type of cache. Eligible global main and indexer blocks can remain available across his history, as any future token may globally select any of those old positions. For SWA, we do not need to keep the entire history, but we do need to retain the local window immediately before any position from which we hope to resume. vLLM will therefore keep window-sized tails of SWA blocks at selected resume boundaries, which we can think of as a sort of checkpoint.
This behavior is controlled by prefix_cache_retention_interval. With the default value of 0, vLLM keeps an SWA checkpoint near Rocco’s original prompt boundary, but does not continue creating checkpoints as he decodes. A future conversation turn may therefore need to replay Rocco’s generated tokens to reconstruct the local state, even if their global state remains cached.
The compressor's ring state, meanwhile, is merely per-request scratch space and does not participate in prefix caching, and so there is no future request trying to recover Rocco's unfinished compressor state.
Conclusion
At this point, we have followed Rocco from the waiting line, through prefix-cache lookup and block allocation, into the forward pass, and finally back out into the free queue. However, everything we have spoken to has mostly been from the perspective of a single vLLM engine managing its own device memory. Once we begin serving across multiple workers, it becomes quite useful for the rest of the system to know what each worker has cached -- i.e. which prefixes were stored and where a request might find reusable state.
This is the role of kv_events.py. vLLM can publish events describing changes to its local KV cache, which downstream systems such as Mooncake or NVIDIA Dynamo can consume to maintain a broader view of where prefixes live. In a future blog post, we'll follow these events from the block pool to their consumers and speak to how Rocco's cached state becomes useful not only for avoiding computation on one worker, but also for deciding which worker should receive the next request.