This is the first post in a "Foundations" series — the background I keep reaching for when writing about CUTLASS, CuTe layouts, or WGMMA. None of those tools make sense in isolation; they exist because of specific decisions NVIDIA made in silicon, generation over generation. Before getting into any of that, it's worth being precise about what actually changed at each step, instead of treating "the GPU" as one static thing.

Timeline of NVIDIA data-center GPU architectures from Volta (2017) to Blackwell (2024), showing the headline feature of each generation

Five generations, five different bottlenecks each was built to remove.

Volta (2017): the generation that added Tensor Cores

The V100 (GV100 die, 21.1 billion transistors, TSMC 12nm FFN) was the first NVIDIA GPU with Tensor Cores — a dedicated execution unit for a 4×4 FP16 multiply with FP32 accumulation, rather than routing matrix multiplies through the general-purpose CUDA cores.

That distinction matters more than it sounds like it should. Every dense layer in a neural network is fundamentally a matrix multiply, and before Volta, that multiply competed for the same ALUs as everything else on the chip. Tensor Cores gave matrix multiply-accumulate its own hardware path — the single change that made mixed-precision training fast enough to be the default rather than a niche optimization.

Turing (2018): inference precision goes mainstream

Turing is best known for RT cores (ray tracing) on the consumer side, but the data-center-relevant change was INT8 and INT4 Tensor Core support. Training generally wants FP16 or better; serving a already-trained model often doesn't need that much precision. Turing's T4 was built around that observation — low power draw, strong INT8 throughput — and became a common choice for inference serving specifically because it matched the precision the workload actually needed instead of over-provisioning for training-grade math.

Ampere (2020): sparsity, and the generalist AI GPU

The A100 (GA100) introduced third-generation Tensor Cores along with two changes that shaped how datacenters actually deploy GPUs:

  • Structured sparsity: hardware support for 2:4 sparsity (2 nonzero values per 4-element block), giving up to 2x throughput on weights pruned to that pattern — sparsity stopped being a purely software-side trick.
  • Multi-Instance GPU (MIG): a single A100 can be partitioned into up to 7 isolated GPU instances, each with its own memory and compute slice. This is a scheduling/utilization change as much as a performance one — it's why a single physical A100 could serve several independent, isolated workloads instead of one.

Ampere is also where TF32 appeared — a 19-bit format that behaves like a drop-in replacement for FP32 in training code while running at close to FP16 speed. A100 became the default cloud AI GPU for years largely because it was good at both training and inference, not specialized for one.

Hopper (2022): hardware co-designed with the transformer block

This is the generation where the architecture and the transformer architecture stopped evolving separately. The H100 (GH100) shipped with:

  • Transformer Engine: dynamically mixes FP8 and FP16 per layer, choosing precision to preserve accuracy while using cheaper math where it's safe to.
  • Fourth-generation NVLink: 900 GB/s bidirectional per GPU.
  • Second-generation MIG: up to 7 instances, now with confidential computing isolation per instance.
  • DPX instructions: purpose-built for dynamic programming algorithms (sequence alignment, route optimization), roughly 40x faster than a dual-socket CPU server on those specific workloads.
  • Thread block clusters and the Tensor Memory Accelerator (TMA) — lower-level changes to how thread blocks share data and move tensors through memory, which matter once you're writing kernels rather than calling into a library. (Future posts in this series get into this directly.)

The throughline: Hopper wasn't a faster version of Ampere so much as a GPU shaped around what a transformer's forward and backward pass actually do.

Blackwell (2024): the rack, not the chip, as the unit of scale

Blackwell-class GPUs (B100/B200, GB200) are physically two reticle-limited dies acting as one logical GPU, connected by a 10 TB/s chip-to-chip link — 208 billion transistors total, on a custom TSMC 4NP process. The headline changes:

  • Second-generation Transformer Engine: adds FP4/FP6 support on top of Hopper's FP8/FP16, with NVIDIA citing roughly 2x attention-layer throughput and 1.5x more AI compute FLOPS at the same power envelope.
  • Fifth-generation NVLink: scales to a 576-GPU domain, up from Hopper's much smaller NVLink islands.
  • TEE-I/O: the first GPU with confidential computing that extends across I/O, not just on-chip execution.

The design center of gravity has shifted: Blackwell doesn't really make sense evaluated as a single accelerator. It's built to be one unit inside something like an NVL72 rack, where the "GPU" a workload actually sees is dozens of dies acting as one NVLink domain.

The pattern across all five

Every generation's headline feature maps to a specific bottleneck that was actually limiting real workloads at the time: Volta targeted raw matrix-multiply throughput, Ampere targeted utilization on sparse/mixed workloads, Hopper targeted the mismatch between generic hardware and the transformer's specific compute pattern, and Blackwell targets the fact that a single die is no longer the right unit of scale for the largest models. None of it was arbitrary — each generation is a direct response to what the previous one made obvious as the next constraint.

The next posts in this series build on this directly, starting with why CUTLASS exists at all — the software layer built to actually reach the throughput this hardware makes possible.

References

Timeline diagram above is original artwork made for this post.