Everything You Need to Know About the NVIDIA GH200

NVIDIA’s GH200 Grace Hopper Superchip platform significantly advances accelerated computing and generative AI. This platform combines the world’s most powerful GPU with the most adaptable CPU.

The NVIDIA GH200’s scalable design can manage intricate generative AI tasks, including large language models (LLMs), recommender systems, vector databases, graph neural networks (GNNs), and more.

“Data centers need specialized accelerated computing platforms to meet the rising demand for generative AI,” stated NVIDIA founder and CEO Jensen Huang. The GH200 is designed to address this exact need.

Huang noted, “The GPU provides outstanding memory technology and bandwidth to enhance throughput, allows GPUs to connect and aggregate performance without compromise, and features a server design that can be easily deployed across data centers.”

Revolutionary Memory and Bandwidth

The GH200’s architecture sets a new high-performance computing (HPC) standard. It integrates the advanced Hopper GPU and the flexible Grace CPU into a single super chip, connected by the high-speed, memory-coherent NVIDIA NVLink Chip-2-Chip (C2C) interconnect.

The GH200 Grace Hopper Superchip’s core is the NVLink-C2C interconnect, which offers 900GB/s of bidirectional CPU-GPU bandwidth-seven times the performance of PCIe Gen5 connections in traditional, accelerated systems. Additionally, interconnect power consumption is reduced by over five times.

NVLink-C2C allows applications to oversubscribe the GPU’s memory by leveraging the Grace CPU’s high-bandwidth memory directly. The GH200 provides up to 480GB of LPDDR5X CPU memory, and when combined with 96GB HBM3 or 144GB HBM3e, the Hopper GPU can access up to 624GB of high-speed memory.

Key Features and Initial Benchmarks

Key attributes of the NVIDIA Grace CPU include:

• Double the performance per watt compared to standard x86–64 platforms
• 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory featuring error-correction code (ECC)
• Up to 53% more bandwidth at one-eighth the power per GB per second compared to an eight-channel DDR5 design

The H100 Tensor Core GPU, built on the new Hopper GPU architecture, boasts several innovative features:

• Remarkably fast matrix computations via the new fourth-generation Tensor Cores, supporting a wider array of AI and HPC tasks
• Up to 9 times faster AI training and up to 30 times faster AI inference with its new Transformer Engine compared to the previous-generation NVIDIA A100
• Enhanced quality of service for smaller workloads through secure Multi-Instance GPU (MIG) partitioning, allowing the GPU to be divided into isolated, appropriately sized instances

In summary, the GH200 offers immense power, though comprehensive benchmarking data is still limited due to its recent release. However, let’s review some initial results.

Initial Benchmarks: Comparing the GH200 to Its Competitors

Author Michael Larabel shared preliminary HPC benchmarks conducted on a GH200 workstation provided by GPTshop.ai on the Linux benchmark site Phoronix. The tests focused on the performance of the Grace CPU.

The tested GH200 system featured 72 cores, a Quanta S74G motherboard, 480GB of RAM, and 960GB + 1920GB SAMSUNG SSD drives. (Detailed specifications and environment descriptions are available on the Phoronix site linked above.) These preliminary benchmarks emphasized CPU performance without power consumption data, yet they revealed noteworthy outcomes.

The GH200 Grace CPU achieved a solid 41.7 GFLOPS in the standard HPCG memory bandwidth benchmark.

Results for NVIDIA GH200 Running HPCG Benchmark (Image Credit: Phoronix)

Another significant result came from the NWChem benchmark, where the GH200 achieved second place at 1403.5 seconds.

Results for NVIDIA GH200 Running NWChem Benchmark (Image Credit: Phoronix)

The GH200 Grace CPU delivered a commendable performance overall, achieving a respectable geometric mean across all benchmarks.

(Image Credit: Phoronix)

Researcher Simon Butcher conducted a series of GPU benchmarks comparing the performance of the PyTorch ResNet50 training recipes published by NVIDIA. Using the 150GB ImageNet 2012 dataset, training ran for 90 epochs, approximately one hour. The GH200 demonstrated excellent performance in these tests.

(Image Credit: Simon Butcher, Queen Mary University of London)

NVIDIA has also released some performance comparisons that might be of interest.

Conclusion

The NVIDIA GH200 Grace Hopper Superchip provides the performance necessary for large-scale AI and HPC applications handling terabytes of data. Whether you’re a scientist, engineer, or managing a large data center, this superchip meets the demand.

Looking ahead, NVIDIA has unveiled the GH200’s successor: the Grace Blackwell B200, the next-generation data center and AI GPU.

As the demand for GPU resources continues to surge, especially for AI and machine learning applications, ensuring the security and ease of access to these resources has become paramount.

Spheron’s decentralized architecture aims to democratize access to the world’s untapped GPU resources and strongly emphasizes security and user convenience. Let’s unpack how Spheron protects your GPU resources and data and ensures that the future of decentralized compute is both efficient and secure.

Interested in learning more about Spheron’s network capabilities and user benefits?Review the whitepaper in full.

Originally published at https://blog.spheron.network on June 25, 2024.