Breaking the Memory Bottleneck Part 1: How CXL Unlocks the Memory Capacity KV Cache Needs
Deep dive into different approaches from Enfabrica, Penguin Solutions, Marvell, and Meta
For most of the AI buildout, the scarce resource was compute. In 2026, the binding constraint has shifted to memory, and the pressure point is KV Cache, the working memory of inference. Its footprint grows with every extra token of context and every additional user in the batch, and in the first half of 2026, surging KV Cache demand coupled with limited memory supply resulted in severe memory bottlenecks.
Chipmakers, CSPs, and infrastructure vendors are now converging on the problem from opposite ends. Part 1 of this series looks at capacity expansion: how CXL technology and KV Cache offloading are creating a new tier in the memory hierarchy, and which companies are positioned along the way. Part 2 looks at footprint compression, where attention mechanism improvements and KV Cache quantization shrink how much room the KV Cache needs in the first place.
Why KV Cache becomes the bottleneck
If you read our previous article on AI memory, you will recall that the KV Cache is a dedicated memory space used during inference to store the KV (Key-Value) vectors generated in the prefill phase. By reading from the KV Cache instead of recalculating those vectors at every decode step, AI models dramatically improve inference efficiency.
The catch is that the KV Cache is dynamically allocated. Its memory footprint scales with both the context window and the batch size:
Total Size of KV Cache (Bytes) = 2 × layers × KV heads × head dimension × sequence length × batch × precision (Bytes)
Consider Llama 3 70B loaded in FP16 without GQA (using standard MHA). The model weights occupy roughly 140GiB, so early on, most of the available memory capacity is used to store the weights. But as the context window scales (assuming batch size of 1), the KV Cache consumes 160GiB at 64K, already exceeding the memory capacity required for the model weights. At 128K, it swells to 320GiB, roughly 2.3 times the size of the model weights. KV Cache becomes the primary driver of memory exhaustion.
Behind the paywall, we dig into capacity expansion solutions designed to resolve the KV Cache bottleneck:
The mechanics behind CXL
Enfabrica’s EMFASYS
Penguin Solutions’ MemoryAI KV Cache Server
Marvell’s Structera S
Meta’s Vistara and MemServer
Related report: Overcoming the Memory Bottleneck: CXL Expansion and KV Cache Compression Innovations