Nvidia lists five ways to get more AI work from every byte of HBM

HBM, the fast memory stacked next to a GPU, stopped growing in Nvidia’s latest generation step: GB300 and Vera Rubin both carry 288 GB per GPU. Nvidia engineers Eduardo Alvarez, Jamie Li and Ronil Prasad argue in a technical blog post of 5 November that the useful question has changed. “The question is no longer how much memory a GPU has, but how much useful work each byte delivers,” they wrote.

Nvidia claims that Vera Rubin NVL72 delivers up to 30 times the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. It measured that on SemiAnalysis AgentX, a benchmark that replays recorded coding-agent sessions. HBM bandwidth doubled to 19.2 TB/s over the same step, and the company credits the rest of the gain to five changes that keep HBM memory busy with useful data.

NVFP4: four bits per value

NVFP4 stores numbers in 4 bits with two levels of scaling factors. For model weights, it cuts the footprint about 3.5 times against FP16 and 1.8 times against FP8, and Nvidia reports 4 times the throughput per megawatt of FP8 for DeepSeek R1 on GB300. Applied to the KV cache, the memory a model keeps of the conversation so far, NVFP4 cuts the size by up to 50% against FP8. It kept more than 99% of BF16 accuracy across 10 DeepSeek-V4-Pro benchmarks.

Experts spread across GPUs

Mixture-of-experts models such as DeepSeek-R1 split each layer into many smaller expert networks. Nvidia’s TensorRT-LLM spreads those experts across eight or more GPUs connected by NVLink, so each GPU holds fewer weights and keeps more HBM for the cache. Nvidia reports that GB300 NVL72 delivers 10 times the throughput per megawatt of H200 NVL8 at 100 tokens per second per user. On Vera Rubin, each GPU gets 3.0 TB/s of NVLink 6 bandwidth, and one rack holds about 20.7 TB of HBM4.

Dynamo moves the KV cache

Nvidia’s Dynamo serving software treats the KV cache as a resource for the whole fleet. It sends each request to the GPU that already holds a matching prefix, runs the prefill and decode stages on separate GPU pools, and moves cache blocks between GPU memory, CPU memory and storage. In partner tests, VAST Data reached 35 GB/s into a single H100, and WEKA reached up to 270 GB/s across eight H100s.

Vera CPU memory for idle agents

An AI agent pauses while a tool runs, and its cache sits idle in HBM. The Vera CPU has 88 cores and up to 1.5 TB of LPDDR5X memory, more than five times the HBM4 of one Rubin GPU, and it connects to the GPU at 1.8 TB/s over NVLink-C2C. Dynamo can park an idle session there during a tool call and bring it back when the agent resumes, so the GPU skips recomputing the context. “A slow CPU therefore costs HBM twice: first as idle residency, then as recomputation,” the authors wrote.

CMX: memory shared by a pod

CMX adds a tier of Ethernet-attached flash storage that a whole pod shares, built on BlueField-4 storage processors and Spectrum-X networking. A prefix that one node computes can then serve compatible model instances on other nodes. For long-context inference, Nvidia claims up to 5 times more tokens per second and 5 times better power efficiency than traditional storage.

What it means for you

If you build or rent AI infrastructure, the post shows how Nvidia plans to cut the cost of each token: fewer bits per value, fewer copies of weights and less recomputation. Most of these gains assume you buy the whole Nvidia stack, from the CPU to the storage processors. All the figures come from Nvidia and its partners, and the post publishes no independent measurements. Memory and storage makers posted some of the biggest losses in this week’s chip selloff, and Nvidia’s pitch is that its systems get more out of every gigabyte customers buy.

Based on Nvidia’s technical blog post of 5 November 2026 by Eduardo Alvarez, Jamie Li and Ronil Prasad. All performance figures are Nvidia’s and its partners’ own.