We’ve shared a series of articles on Substack mapping the memory hierarchy as AI shifted from training to inference, and introduced the SSD POD as a new tier between local SSD and shared storage. This piece looks at what is going into that tier. TLC SSD plus HDD has been the default, but TrendForce has observed some operators adding QLC SSD to the mix. This article explains what is driving that change, and which operators have already begun purchasing QLC for their SSD PODs.
Where KV Cache Lands
AI inference consists mainly of two stages:
Prefill: The input prompt is processed in parallel in a single pass. The Key and Value vectors for each layer are computed and stored in the KV Cache, and the first output token is generated at the same time.
Decode: Key and Value vectors are read from the KV Cache to generate tokens one at a time, while newly computed Key and Value vectors are continuously written back into it, until the response is complete.
This prevents models from recalculating all previous keys and values every time a token is generated.
As the context windows of AI users and batch size (including the number of AI agents in play) grow, the memory capacity required by KV Cache expands by multiples. The industry has therefore been developing KV Cache offloading technology and SSD POD to break through this capacity bottleneck.
KV Cache offloading moves cache that is temporarily unused or repeatedly readable from HBM, which is costly and limited in capacity, down to a lower-cost tier with more room.
An SSD POD is an independent storage unit formed by multiple SSD racks, sitting between local SSD and shared storage, and it extends the capacity available for holding KV Cache.
More on SSD POD:
