The mpi partition

The mpi partition is a dedicated pool of CPU nodes for jobs with tightly coupled parallel needs, such as MPI-driven simulations where many processes exchange frequent, small messages and communication latency matters. The partition is configured for exclusive access, so a job that requests a node automatically gets allocated the entire server.

Specifications

SpecificationValue
Number of nodes25
Cores per node64
CPUs2x Intel(R) Xeon(R) Platinum 8352Y (64 core)
Memory per node120 GB to 620 GB (varies by node)
SIMD instructionsAVX-512
NetworkingInfiniBand, 10/25 Gbps Ethernet
stylus_note
Consistent hardware
All nodes in the mpi partition share the same CPU model, which keeps performance predictable when your job spans multiple nodes. You can also request this consistency explicitly from any partition with --constraint=mpi.
stylus_note
Limits
The mpi partition is limited to 4 nodes per user. Cores used in the mpi partition do not count against the 512-core per-PI-group limit of the cpu partition.

How to use the mpi partition

Access to the mpi partition is granted to PI groups by request. Submit the MPI request form to request access, then add #SBATCH --partition=mpi (or -p mpi) to your job script:

#!/bin/bash
#SBATCH --job-name=my-mpi-job
#SBATCH --partition=mpi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --mem-per-cpu=1g
#SBATCH --time=2:00:00

module load openmpi/5.0.3
export OMP_NUM_THREADS=1
mpirun ./my_program
  • --ntasks-per-node=64 starts one MPI task on each of the node’s 64 cores. Exclusive access is implicit on this partition.
  • --mem-per-cpu requests memory per core; multiply by the number of cores to check it fits on the node (see CPU Summary for per-node memory).
  • --nodes should be sized to your job; with 64 tasks per node, two nodes run 128 MPI tasks.

See Use MPI for multi-node job execution for a guide to shaping and launching MPI jobs, and the Slurm cheat sheet for a quick reference of job submission options.

Which partition should I use?

If your job is tightly coupled and benefits from whole nodes, the mpi partition is a good fit. If your job is loosely coupled (for example, embarrassingly parallel jobs that rarely communicate between nodes), the general cpu partition can be a better use of the cluster. For the full list of partitions and their limits, see the Partition List, and the Node List for the specific nodes that belong to each partition.