New High Performance Supercomputer starts its operation at DKRZ

Levante the new, fourth high-performance computing system for Earth system research (HLRE-4) will start its operation on March 3, 2022, at the German Climate Computing Center (DKRZ) in the first expansion stage. The supercomputer, which like its predecessor “Mistral” is provided by the company Atos, will quadruple the computing power at DKRZ with 14 PetaFLOPS.

This enables researchers to perform, for example, more or longer simulations with particularly high-resolution global climate and Earth system models on the DKRZ system. Such models allow for the first time a purely physically based representation of important climate processes, whereas in the much coarser models used so far, these small-scale processes had to be parameterized. Furthermore, small-scale interactions between atmosphere, ocean and the other parts of the Earth system can be taken into account.

An important application of climate modeling is to project possible climate changes over the course of this century and to study them for different scenarios. So far, however, simulations with such high-resolution climate models could only be computed for very short periods of a few months. Longer simulation periods require significantly more computing time and storage volume which the DKRZ can now provide with Levante. BullSequana by Atos is an efficient solution for calculating and storing simulation results.

“The new system is the basis of our services. The powerful new systems are a better tool for research, and thus also for the risk assessment of climate change for society and ecology. The major part of the German climate simulations that contribute to the IPCC reports are calculated at DKRZ,” comments Prof. Thomas Ludwig, director of DKRZ, and adds: “As with the predecessor model, we attach great importance to high energy efficiency. The system features high-temperature liquid cooling, and part of the waste heat is fed into the heating system and re-used in the university building next door.”

“The systems at DKRZ are the most important infrastructure for our research at the Max Planck Institute for Meteorology (MPI-M),” says Prof. Jochem Marotzke, director of the department “Ocean in the Earth System” at MPI-M. “Initial tests have already shown that Levante gets us to the solution much faster than its predecessor system.”

“Levante represents a step change in computing, by enabling more physical models based on grids as fine as 2.5 km and for periods of decades, it will allow us to open new frontiers in climate science.” comments Prof. Bjorn Stevens, director of the department “Atmosphere in the Earth System” at MPI-M. “We look very much forward to working with the team at DKRZ, and within the broader German climate research community, to explore these frontiers.”

TECHNICALSPECIFICATIONS OF LEVANTE

Levante’s CPU partition includes 2,832 computing nodes with two processors each, which deliver a total peak performance of 14 PetaFLOPS. That is 14 quadrillion mathematical operations per second. The system is equipped with third-generation AMD EPYC processors, each with 64 processor cores. The total main memory of the system is more than 800 Terabytes; this is equivalent to the main memory of about 100,000 laptops. To cover different classes of requirements, the individual systems that make up the supercomputer have main memory sizes of between 256 and 1,024 Gigabytes.

In addition to the classic CPU processors, Levante will receive a partition with 60 GPU nodes in summer, which will deliver a peak performance of 2.8 PetaFLOPS. Each GPU node is equipped with two AMD EPYC processors and four NVIDIA A100 graphics processing units (GPUs); 56 of the GPUs with 80 Gigabytes of GPU memory and four nodes with 40 Gigabytes of GPU memory.

This increasingly heterogeneous hardware architecture poses major challenges for scientific software development. The DKRZ will support its users in adapting their workflows (e.g. the porting of program codes or the use of artificial intelligence) to make efficient use of such developments in high-performance computing.

For data transfer between the computer nodes and the storage components, Levante uses NVIDIA Mellanox InfiniBand HDR 200G technology, which can achieve a data transfer rate of up to 200 GBit/s.

To store and analyze the calculated simulation results, Levante is equipped with a storage system of about 130 Petabyte from the company DDN. This means that more than twice the previous storage space is now available. Compared to a conventional laptop with one Terabyte of disk space, the supercomputer achieves around 130,000 times the storage capacity.

Udo Littke, CEO of Atos in Germany, says: “We are very pleased that the new supercomputer is now available to the scientific users of DKRZ. With the help of state-of-the-art components in our BullSequana XH2000 system, the DKRZ now has significantly more power and memory for demanding climate simulations. At the same time, we were able to greatly reduce the costs per calculation transaction. This is crucial, because in the course of a sustainable digital transformation of the economy and science, it is important to constantly reduce energy consumption in computing centers.”

FINANCING

On the basis of the financing agreement from November 2017, the project HLRE-4 is funded with a total of 45 million Euros by the Helmholtz Association, the Max Planck Society and the Free and Hanseatic City of Hamburg.

Wissenschaftliche Ansprechpartner:

Michael Böttinger | DKRZ | boettinger@dkrz.de | +49 40 460094 344

Weitere Informationen:

https://www.dkrz.de/en/projects-and-partners/projects/focus/levante-spotlight?se…