Samsung accelerates AI memory: from HBM4E to the leap into true 3D

The memory bottleneck in AI

Samsung has presented its strategy to overcome what the company calls the "digital memory wall" created by AI workloads. The roadmap includes advances in both high-bandwidth memory (HBM) and NAND flash, all organized around 3D architectures and tighter integration with accelerators.

The starting point is stark: today’s AI infrastructure is already strained to support roughly 100 tokens per second per user in 2026. Samsung projects that by 2030 that figure will reach 1,000 tokens per second per user. That is a tenfold increase in under five years. Memory systems will have to change in proportion.

HBM4E and the near-term horizon

HBM4E is the starting point of Samsung’s strategy. It is already in sampling with key ecosystem partners. It uses a 4-nanometer base die, increases the number of through-silicon vias (TSVs) by a factor of four, and employs advanced packaging with more than 300,000 microbumps and tighter spacing between connections to improve reliability and thermal management.

Samsung estimates that HBM will account for more than 50% of DRAM sales by 2030. The HBM development cycle has accelerated more than that of any other memory technology, according to Leno Park, vice president of flash solutions at Samsung Electronics.

The leap to HBM5 and true 3D architecture

The next step is HBM5, which will use a 2-nanometer base die built with gate-all-around (GAA) transistors and shorten interposer channel distances to improve I/O signaling between the base die and the accelerator.

But the most significant piece is what Samsung calls zHBM: an architecture that leaves behind today’s side-by-side 2.5D design to place the AI accelerator directly on top of the HBM stack, as a single true-3D domain structure. By reducing the distance data must travel between processor and memory, this architecture improves bandwidth and energy efficiency while easing thermal constraints.

Samsung claims that a next-generation interface incorporating zHBM could deliver roughly eight times the performance of HBM5, more than ten times the memory density, three times the energy efficiency, and less than half the thermal resistance. Those numbers, however, require close co-development with accelerator partners.

Beyond HBM: the NAND strategy

Samsung has also introduced zNAND-O, a next-generation NAND concept aimed at edge AI and other data-intensive environments. Built on Samsung’s V-NAND technology, the architecture seeks to combine space efficiency with improved I/O performance, energy efficiency, and reduced latency by shortening the distance data must travel between processor and memory.

Samsung’s tenth-generation TLC V-NAND will feature more than 400 layers and an 11% lateral reduction, while memory density has increased by roughly 58%. According to Park, AI models keep getting larger and conventional memory architecture will not be able to keep up. "After 2029, we need some revolutionary technology to overcome the digital memory wall," he said.

Where this converges with smart hardware

The problem Samsung describes is real at every scale. It is not only in data centers that memory becomes a bottleneck. When you design hardware that needs conversational intelligence or fast decision-making at the edge, the balance between what fits in memory and what takes time to arrive over a network remains fundamental.

The architectural optimizations Samsung describes—shortening distances, improving integration, rejecting 2.5D design for truly 3D architecture—are principles that apply at every level of hardware, from data centers to microcontrollers. The pressure AI creates is not only quantitative; it is structural.

At bitbitbla we work with that same challenge on small hardware: how to orchestrate agentic intelligence on devices with limited resources. If you have room for a microcontroller, you can have an intelligent voice. That is possible because efficient architectures matter at every scale. The decisions Samsung makes today about how to move data between memory and processor are the same kind of decisions that define the design of a product with conversational capabilities at the edge.

Source: EE Times.