Blackwell: Nvidia unveils its next generation of AI accelerators

Contents

Nvidia's in-house exhibition GTC 2024 in San José, California, took place in person for the first time in years and was unsurprisingly dominated by artificial intelligence. Nvidia currently dominates the data center market with its accelerator chips and is generating one record financial result after another in the wake of the AI boom.

Nvidia boss Huang wants it to stay that way. The newly unveiled Blackwell accelerator architecture plays a central role in this in the form of various products, ranging from the B100 to the DGX GB200 SuperPOD, which are to be launched on the market over the course of the year. On paper, Blackwell GPUs achieve twice the throughput of AMD's brand new MI300 accelerators for some data formats.

With the new Blackwell architecture and further reduced data formats such as 4-bit floating point as well as new functions, the focus is primarily on energy efficiency and data exchange between the individual chips. This is why the NVLink switch and the network technology have also been upgraded.

Nvidia traditionally does not name prices and refers to its partners, but as long as the AI hype continues unabated, the chips are likely to sell almost regardless of pricing. Nvidia has already won Amazon Web Services, Google Cloud and Oracle Cloud as customers, but does not specify exactly when instances will be bookable. Meanwhile, the older H100 products continue to roll off the production line.

Blackwell double chip

Nvidia is breaking new ground with Blackwell, but is remaining true to itself in some ways. The Blackwell "GPU" consists of two individual chips. Although Nvidia did not want to answer our question as to whether the two are functionally identical, it did state that both are at the limit of the exposure possibilities. This means that they should each be around 800 mm² in size and therefore take up about as much space individually as their predecessors, the H100 (814 mm²) and A100 (826 mm²). The word chiplet was not used, as this is usually understood to mean the coupling of different chips.

Nvidia has the Blackwell chips manufactured at TSMC in a process called "4NP", which does not correspond to the usual nomenclature of the forge. Nvidia did not answer whether this is a derivative of N4P and what characteristics the process has - however, we assume that it is N4P, in which a few parameters have been adjusted according to customer requirements as usual.

Both chips, which Nvidia only refers to together as the Blackwell GPU, are connected to each other via a fast interface with 10 TByte per second (5 TByte/s per direction). According to Nvidia, this is sufficient for them to behave like a single GPU in terms of performance. For comparison: We measure around 5 TByte per second for the level 2 cache as the data-connecting instance in Nvidia's high-end GeForce RTX 4090 graphics cards, while AMD's MI300 accelerator chips are connected to each other at up to 1.5 TByte/s.

192 GByte HBM3e memory

When it comes to memory, Nvidia goes all out and starts with eight 24 GByte stacks of the fast HBM3e memory. This adds up to a total of 192 GByte and 8 TByte/s transfer rate. In terms of memory size, the company is on a par with AMD's MI300X, but the B100 has a transfer rate that is around 50 percent higher – the upgrade was also necessary, as Nvidia's H100 generation was lagging behind in terms of memory size. With eight stacks, it is also theoretically possible to switch to 36 GByte stacks at a later date and increase capacity to 288 GByte.

There were only a few details about the internal structure at the presentation. Nvidia only provided performance data for the Tensor cores, but remained silent on traditional shader arithmetic units and other units.

For the presentation in the table, we have used the throughput values with sparsity; with densely populated matrices, the teraflops figures are halved. Exception: FP64, here Nvidia does not specify throughput with sparsity.

Accelerators for data centres and AI
Name	GB200	AMD MI300X	H100	A100
Architecture	1x Grace + 2x Blackwell	CDNA3	Hopper	Ada
Transistors	2x 104 billion. / 4NP / ~800 mm²	Various chiplets with 153 billion / N5 + N6	80 billion / 4N / 814 mm²	54,2 billion / N7 / 826 mm²
Format	Add-on-Board (2 pro 1U-Rack) / SXM	SXM5	SXM5	SXM4
Year	2024	2023	2022	2020
TDP	1200 watts per GPU + 300 watts Grace CPU	750 watts	700 watts	400/500 watts
Memory	192 GByte HBM3e	192 GByte HBM3	80 GByte HBM3	80 GByte HBM2E
Transfer rate	8 TByte/s	5,3 TByte/s	3,35 TByte/s	2,04 TByte/s
GPU-GPU-connection	NVLink Gen 5, 1,8 TByte/s	896 GByte/s	NVLink Gen 4, 900 GByte/s	NVLink Gen 3, 600 GByte/s
Computing power tensor core per GPU (TFLOPS, with sparsity)
FP64 (Teraflops)	45	163	67	19,5
FP32 (Teraflops)	n.a.	163	n.a.	n.a.
TF32 (Teraflops)	2500	1307	989	312
BF16 (Teraflops)	5000	2615	1979	624
FP8 (Teraflops)	10000	5230	3958	n.a. (INT8: 1248 TOPS)
FP4 (Teraflops)	20000	n.a.	n.a.	n.a.
Computing power shader cores per GPU (Teraflops)
FP64 (Teraflops)	n.a.	81,7	33,5	9,7
FP32 (Teraflops)	n.a.	163,4	66,9	19,5
BF16 (Teraflops)	n.a.	n.a.	133,8	39
FP16 (Teraflops)	n.a.	n.a.	133,8	78

FP4 and FP6

One of the new features of Blackwell is support for a 4-bit floating point format (FP4) in the chip's transformer engine, for which there is a research paper in collaboration with an Nvidia employee. This is intended to store weights and activations in AI inferencing of Large Language Models (LLM) with only 4 bits. The authors claim "For the first time, our method can quantize both the weights and the activations in the LLaMA-13B model to only 4 bits and achieves an average score of 63.1 on the common-sense zero-shot reasoning tasks, which is only 5.8 points lower than the full-precision model and significantly outperforms the previous state of the art by 12.7 points." FP4 could therefore handle significantly less data with only a relatively small loss in the accuracy of the result, which in addition to the processing speed also doubles the possible model size.

During the keynote, Nvidia CEO Huang also revealed that FP6 is also an option. Although this format does not provide additional computing throughput compared to FP8, it saves memory, cache and register space and therefore also energy.

With a GPT model with 1.8 trillion parameters ("1.8T Params"), Nvidia aims to be 30 times faster than an H100-based system with a GB200 with a so-called Mixture of Experts, i.e. adapted computing and data accuracy, and to work 25 times more efficiently. Nvidia claims to have calculated the performance down to one GPU, but in reality compares a system with 8 HGX100 and 400G Infiniband interconnect with 18 GB200 superchips (NVL36).

But even with the established FP8 data format, Blackwell is on paper almost twice as fast as AMD's MI300X and more than 2.5 times as fast as its predecessor, the H100.

Blackwell products

Nvidia wants to offer Blackwell as a GB200 board on which two Blackwell dual chips are combined with a Grace processor. According to Nvidia, the Grace CPU has only been slightly adapted, but details are also missing here.

While Nvidia has accelerated the proprietary fast GPU connection NVLink from 900 to 1800 GByte/s in what is now the fifth generation, both Blackwells have to share the C2C link with 900 GByte/s to the Grace CPU. Each Blackwell double chip has a maximum combined upstream and downstream transfer rate of 450 GByte/s at its disposal. Nvidia achieves the transfer rate of 1.8 TByte by doubling the speed per lane pair from 25 to 50 GByte/s and direction, i.e. 100 GByte/s full-duplex multiplied by 18 lane pairs.

Nvidia combines two GB200s for a liquid-cooled 1U rack slot; each Blackwell dual chip can be configured with up to 1200 watts of power consumption, leaving 300 watts for the Grace CPU, which together amounts to 2.7 kW. However, the classic HGX rack bays with eight SXM cards will also continue to be available. For HGX B200, each double chip can draw up to 1000 watts, in the air-cooled B100 still 700 watts.

An intermediate stage comes with the GB200 NVL72, which Nvidia calls the "New Unit of Compute" - any similarity to Intel's recently abandoned "Next Unit of Compute" for mini PCs is certainly only coincidental. GB200 NVL is a pre-configured, liquid-cooled rack with 36 Grace CPUs and 72 Blackwell dual chips connected via NVSwitches. As this connection is made at full speed, without any further NVLink intermediate stage, Nvidia also refers to the NVL72 as a DGX system.

NVSwitch with NVLink 5 for DGX SuperPODs

Nvidia is introducing a new NVLink switch for faster data transfer. The chip now also connects racks with each other and can connect up to 576 Blackwell double chips (i.e. 288 GB200) – previously the maximum was 256 GPUs. The switch has 50 billion transistors and, like Blackwell, is manufactured in TSMC's 4NP process. It transfers the full 1.8 TByte/s of each connected Blackwell dual chip from each to each client and manages a total of 7.2 TByte/s. Nvidia liquid cools two of these switches in a 1U tray, which is also used in the NVL72.

Up to 128 of the 576 chips can be combined in a confidential computing domain, in which access to the confidential memory by external partitions is prevented.

Nvidia has also brought the PCI Express connections up to date with PCIe 6. Together, the lanes can transfer another 256 GBytes – again in addition to upstream and downstream.

The improved connectivity is required, for example, for the SuperPOD with eight DGX-GB200 systems, which is also available as a turnkey solution. A SuperPOD therefore contains 288 Grace CPUs, 576 Blackwell double chips and 240 TBytes of total memory. According to Nvidia, the system achieves 11.5 exaflops (11,500 PFLOPS or 11,500,000 TFLOPS).

Nvidia wants to connect "hundreds of thousands" of GPUs with NVLink5, switches and 800 Gbit/s fast network from Mellanox.

Note: Nvidia has covered the author's travel and hotel costs to GTC 2024.

(csp)