Architectures of HPC clusters for massively parallel batch processing

By Smolkin Mikhail (mrsmolkin@edu.hse.ru)

High-Performance Computing (HPC) clusters underpin critical scientific and industrial tasks that require massive parallelism. These systems integrate thousands of processing cores, specialized accelerators, high-speed interconnects, and parallel storage solutions to tackle large-scale jobs, such as simulations, data analytics, and batch processing tasks. HPC clusters have consistently pushed the boundaries of performance and scalability, enabling breakthroughs in areas like climate modeling, astrophysics, bioinformatics, and artificial intelligence [1].

The roots of HPC trace back to the 1960s with machines like the CDC 6600, often regarded as the first commercial supercomputer. Costs and technological advances subsequently fueled the transition to cluster-based systems using commodity off-the-shelf hardware [2]. The popular Beowulf architecture demonstrated how open-source software combined with lower-cost hardware could yield powerful parallel platforms.

As processors became more powerful and multi-core, clusters grew in node counts, culminating in massively parallel processing (MPP) architectures. Many of these leveraged specialized interconnects with low latency and high bandwidth [3]. By the 2010s, GPUs and other accelerators entered HPC ecosystems, boosting performance-per-watt and enabling large-scale machine learning [4].

Today, heterogeneous HPC clusters dominate the world’s fastest systems [5]. They typically blend CPUs, GPUs, and sometimes specialized ASICs or FPGAs. Interconnects like InfiniBand deliver microseconds of latency, while parallel file systems manage petabytes of data. Advanced schedulers orchestrate batch workloads, ensuring efficient resource allocation across thousands of concurrent jobs [6].

Modern compute nodes rely on high-core-count CPUs (e.g., AMD EPYC, Intel Xeon) or alternative architectures (ARM, Power). Accelerators, chiefly GPUs (NVIDIA, AMD), provide substantial speedups for numeric and AI workloads. Memory technologies like DDR4/DDR5 or high-bandwidth memory (HBM) and Non-Uniform Memory Access (NUMA) configurations also affect performance [7].

Efficient communication is pivotal. InfiniBand stands out with up to 400 Gbps and RDMA, yielding both high bandwidth and low latency [8]. Ethernet, while more common in data centers, can serve cost-sensitive HPC deployments. Other fabrics, such as Intel Omni-Path or proprietary solutions, compete in specialized environments.

HPC clusters often employ parallel file systems (Lustre, GPFS) to stripe data across multiple storage servers, supporting simultaneous reads and writes [9]. Object-based approaches like Ceph can handle unstructured data at scale [10]. Many HPC centers also use SSD-based burst buffers to mitigate I/O bottlenecks for short-term, high-intensity writes.

A typical HPC software environment includes:

1. Operating System: Typically Linux.
2. Libraries: MPI (Message Passing Interface), OpenMP for shared memory, math libraries for distributed computing.
3. Schedulers: SLURM, PBS, or similar batch systems that orchestrate job execution.

Effective management and monitoring are crucial for maintaining HPC cluster performance and reliability. Tools such as Ganglia, Nagios, and Prometheus provide real-time insights into system metrics like CPU/GPU utilization, memory usage, network traffic, and I/O performance. These systems enable administrators to detect and address performance bottlenecks, hardware failures, and security issues promptly. Additionally, centralized logging and alerting mechanisms facilitate proactive maintenance and ensure that the HPC environment remains robust and efficient for all users.

Popular topologies for HPC interconnects include fat-tree [11] for hierarchical scalability; torus networks (e.g., IBM Blue Gene series) for low-latency wraparound connections; and dragonfly, which reduces global link usage by grouping nodes [12].

High bandwidth is essential for data-intensive tasks like domain decomposition, while low latency is critical for synchronization-heavy algorithms. Minimizing communication overhead often dictates HPC performance scaling across thousands of processes [13].

As HPC systems continue to evolve, emerging network technologies are being integrated to further enhance performance and scalability. Optical interconnects, for instance, offer higher bandwidth and lower power consumption compared to traditional copper-based solutions, making them ideal for next-generation HPC clusters. Additionally, software-defined networking (SDN) is gaining traction, allowing for more flexible and programmable network configurations that can adapt to the dynamic needs of batch processing workloads. These advancements enable more efficient data movement, reduce congestion, and support the increasing demands of exascale computing environments.

Fig. 1. HPC Network Topologies. From hpcWire.

HPC workloads commonly rely on batch job submission. Tools like SLURM and PBS let users specify resource requirements, which are then queued for execution [6]. Policies such as backfilling improve utilization by placing smaller jobs in available slots [14]. Systems may prioritize jobs based on size, user group, or urgency. Some HPC sites allow reservations for time-sensitive tasks. In multi-tenant environments, fair-share policies aim to balance resource usage across teams or departments.

Fig. 2. Batch Job Submission. From TACC.

1. Distributed-Memory via MPI
2. Shared-Memory and Hybrid Approaches
3. GPU/Accelerator Programming

MPI is the standard for message-passing across nodes in an HPC cluster [3]. MPI’s point-to-point and collective communication operations allow thousands of processes to coordinate effectively. Within a node, OpenMP provides thread-based parallelism, often combined with MPI in a hybrid model. This exploits distributed memory across nodes and shared memory within each node. CUDA and OpenACC facilitate offloading computations to accelerators. GPU-based HPC applications have shown orders-of-magnitude performance gains for suitable workloads (dense linear algebra, deep learning, etc.).

• Oak Ridge National Laboratory: Frontier
• Argonne National Laboratory: Polaris
• Europe’s LUMI Supercomputer
• TSUBAME at Tokyo Institute of Technology

One of the first exascale machines, Frontier integrates AMD CPUs, GPUs, and the Slingshot interconnect to surpass 1 exaflop of peak performance [15]. It is tailored for massive simulation and AI workloads. A pre-exascale system at Argonne featuring AMD and NVIDIA hardware with HDR InfiniBand for HPC and AI research [16]. It uses container-based workflows to simplify software deployment. Hosted by CSC, LUMI is an AMD CPU-GPU Cray EX system designed for EuroHPC projects. It employs advanced liquid cooling and a unified software stack for extreme-scale simulations [17]. TSUBAME has evolved through multiple generations, pioneering GPU acceleration and high-efficiency cooling [18]. Current iterations emphasize HPC-AI synergy.

• As clusters move to exascale (10^18 operations per second), even modest inefficiencies—like software overhead, OS jitter—become critical. Robust scalability, fault tolerance, and power efficiency are top concerns [4].
• Mega-scale HPC systems consume megawatts of power and generate tremendous heat. Innovations in liquid cooling, direct chip cooling, and energy-aware scheduling aim to curb operational costs.
• At extreme scales, mean time between failures drops. Techniques like checkpoint/restart help mitigate node failures, but overheads remain significant. Enhanced resilience strategies focus on adaptive runtimes and lightweight fault recovery.
• Experimental HPC systems now investigate quantum accelerators for specialized applications [19]. Meanwhile, the convergence of HPC and AI fosters HPC infrastructure that supports deep learning frameworks alongside traditional simulation workloads.
• Public clouds increasingly offer HPC-targeted instances with RDMA and GPU access. Hybrid cloud HPC solutions let organizations burst workloads offsite. Additionally, software-defined networking and composable infrastructures promise on-demand reconfiguration of cluster resources.

HPC clusters are the engines of large-scale simulation, data analytics, and AI-driven research. Architected with a synergy of commodity and specialized hardware (CPUs, GPUs, high-speed fabrics), robust parallel file systems, and sophisticated schedulers, these clusters unleash massive parallelism. Current architectures already support exascale computations, but ongoing R&D tackles looming challenges: energy efficiency, resilience at scale, and software complexity. Looking ahead, HPC systems will increasingly integrate heterogeneous accelerators, advanced interconnect topologies, and perhaps even quantum co-processors. Through continuous innovation, HPC clusters will remain the linchpins of scientific discovery and high-impact data processing.

1. Hennessy, J.L., & Patterson, D.A. (2011). Computer Architecture: A Quantitative Approach.
2. Beowulf cluster.
3. Gropp, W., & Lusk, E. (2002). The Sourcebook of Parallel Computing.
4. Filippo Spiga (2011). The rise of HPC accelerators.
5. TOP500. (2024). TOP500 List of Supercomputers.
6. Weinberg, J. (2003). Job Scheduling on Parallel Systems.
7. Hager, G., & Wellein, G. (2010). Introduction to High Performance Computing.
8. NVIDIA. (2024). InfiniBand Solutions.
9. Braam, P.J. (1999). File Systems for Clusters from a Protocol Perspective.
10. Weil, S.A., et al. (2006). OSDI.
11. Dally, W., & Towles, B. (2004). Principles and Practices of Interconnection Network.
12. Kim, J., et al. (2008). Technology-Driven, Highly-Scalable Dragonfly Topology.
13. Ishikawa, S., et al. (2024). Minimizing Communication Overhead in Distributed Lion.
14. Baraglia, R., es al. (2008). Backfilling Strategies for Scheduling Streams of Jobs On Computational Farms.
15. ORNL. (2024). Frontier Supercomputer.
16. Argonne. (2024). Polaris.
17. CSC. (2024). LUMI Supercomputer.
18. Matsuoka, S., et al. HPCA.
19. Preskill, J. (2018). Quantum.

ChatGPT from OpenAI, o1-mini (content 128k window, release from 12.12.2024) used for initial research. Promt: Provide a list of 20 sources (a.g. books or articles from a scientific journal) which could help in research in “Architectures of HPC clusters for massively parallel batch processing”