Thanks for the question! 3D DRAM fills the gap between SRAM and HBM in the memory hierarchy. In theory, SRAM offers the highest bandwidth and lowest latency, but its low capacity density limits capacity to the MB range. In long-context, high-batch scenarios, model weights and KV cache can't stay resident in SRAM, which ends up adding communication latency.
3D DRAM, by comparison, is a middle-ground option: its bandwidth can approach SRAM's, while its capacity reaches the GB level. In these scenarios, it does have an advantage over SRAM. That said, the capacity a single chip can carry is still limited, so 3D DRAM still needs to be paired with other memory tiers such as HBM, CPU-attached DRAM, and SSDs.
Anyone has any views on whether on chip models are good or bad? Basically what happens if at a future point of time, there are significantly better models making it largely inefficient to use the programmed model on the chip? Do you just bin the chip at that point or what lol?
Does 3D-DRAM be a better option for inference?
Thanks for the question! 3D DRAM fills the gap between SRAM and HBM in the memory hierarchy. In theory, SRAM offers the highest bandwidth and lowest latency, but its low capacity density limits capacity to the MB range. In long-context, high-batch scenarios, model weights and KV cache can't stay resident in SRAM, which ends up adding communication latency.
3D DRAM, by comparison, is a middle-ground option: its bandwidth can approach SRAM's, while its capacity reaches the GB level. In these scenarios, it does have an advantage over SRAM. That said, the capacity a single chip can carry is still limited, so 3D DRAM still needs to be paired with other memory tiers such as HBM, CPU-attached DRAM, and SSDs.
Anyone has any views on whether on chip models are good or bad? Basically what happens if at a future point of time, there are significantly better models making it largely inefficient to use the programmed model on the chip? Do you just bin the chip at that point or what lol?
cheaper to make a new chip than it is to train a new model, so yes, i would say bin outdated ones. can't speak to the inefficiency problem lol