Using Open MPI

Open MPI is freely available implementation of the Message Passing Interface. This version of Open MPI on Brazos was built with the 64-bit GCC compiler suite.

Environment

To use Open MPI you will need to load the appropriate environment variables. This can be done with the Modules package.

To use Open MPI compiled with GCC:
module load gcc
module load openmpi

To use Open MPI compiled with Intel:
module load intel
module load openmpi

In your SLURM batch scripts be sure to load the same modules:

#!/bin/bash
#SLURM -J myjob
#SLURM -p mpi-core8
#SLURM -t 01:00:00
#SLURM -o myjob-%J.out
#SLURM -N 4
#SLURM --ntasks-per-node=8
module load gcc
module load openmpi
cd subdir
mpirun -np 32 ./myapp
exit 0

Compiling

As with other MPI implementations, to compile an Open MPI application, use the mpicc, mpiCC, mpif77, or mpif90 commands. These will call the GNU C, C++, FORTRAN 77 and 90 compilers, respectively. Be sure to pass any optimization flags (e.g., -O) to these commands just as you would for the Intel compilers.

Examples:
mpicc -O myapp.c -o myapp -s -lm
mpiCC -O myapp.cc -o myapp -s -lm
mpif77 -O myapp.f -o myapp -s -lm
mpif90 -O myapp.f90 -o myapp -s -lm
(all text for each case above should be on a single line).

FORTRAN 90 note: The mpi.mod file is available under Open MPI. Insert use mpi near the top of your code to use it.

Running

All parallel applications must be launched from a SLURM batch script. We have modified the mpirun so that you simply need to use mpirun to launch your application without having to specify a host list. Our modifications to mpirun will determine that it is being run from SLURM and will obtain the list of nodes on which to launch your application. An example which will run on 32 nodes using all 8 cores on each node is shown below.


Example script (download):

#!/bin/bash
#SLURM -J myjob
#SLURM -p mpi-core8
#SLURM -t 12:00:00
#SLURM -o myjob-%J.out
#SLURM -N 4
#SLURM --ntasks-per-node=8

# The options above specify to use the mpi-core8 partition, requests 4
# nodes with eight processes per node, limits the job to 12 hours of
# walltime, and defines the standard output name as myjob-JOBID.out

# Load the Open MPI GCC environment
module load gcc
module load openmpi

# Print out the starting time and host.
echo "$SLURM_JOBID started on `hostname` at `date`"
echo "-=-"

# Change to a subdirectory
cd $HOME/mysubdir
 
# Run the application 'myapp' on 32 cores.  The '-f myfile.in' argument
# is passed to myapp.    The arguments to the left of myapp are passed
# to mpirun. All data between the 'EOT' markers is input to myapp
# that would normally be read from stdin on a terminal.
mpirun -np 32 myapp -f myfile.in << EOT
3
10 56.0
A B
9
EOT
echo "-=-"
 
# Print the end time and host.
echo "$SLURM_JOBID ended on `hostname` at `date`"
 
exit 0

InfiniBand

The mpi-core8 and mpi-core32 SLURM partitions will have access to 128 and 26 nodes respectively with DDR InfiniBand interfaces.

Open MPI has native support for InfiniBand. To use it, use mpirun -mca btl openib,self -np 64 myapp to have mpirun use OpenIB's libverbs InfiniBand interface. See below for an example.

The nodes with InfiniBand are available through SLURM by using the mpi-core8 (8 cores / node) or mpi-core32 (32 cores / mode) partitions. This can be specified in your SLURM script or on the command line:
sbatch -p mpi-core8 -N 8 -n 32 myibapp.slrm
In addition, the SLURM script needs to be modified to facilitate using Open MPI across the InfiniBand fabric.

Example script (download):

#SLURM -J myjob
#SLURM -p mpi-core8
#SLURM -t 12:00:00
#SLURM -o myjob-%J.out
#SLURM -N 8
#SLURM --ntasks-per-node=8

# The options above specify to use the iamcs queue, requests 8 InfiniBand
# nodes in the mpi-core8 partition with 8 processes per node, limits
# the job to 12 hours of wall time, and writes stdout to myjob-JOBID.out

# Load the Open MPI GCC environment
module load gcc
module load openmpi

# Print out the starting time and host.
echo "$SLURM_JOBID started on `hostname` at `date`"
echo "-=-"

# Change to a subdirectory
cd $HOME/mysubdir
 
# Run the application 'myapp' on 64 cores.  The '-f myfile.in' argument
# is passed to myapp.    The arguments to the left of myapp are passed
# to mpirun. All data between the 'EOT' markers is input to myapp
# that would normally be read from stdin on a terminal.
# The arguments "-mca btl openib" will instruct mpirun to use OpenIB's
# libverbs for InfiniBand.
mpirun -mca btl openib,self -np 64 myapp -f myfile.in << EOT
3
10 56.0
A B
9
EOT
echo "-=-"

# Print the end time and host.
echo "$SLURM_JOBID ended on `hostname` at `date`"
 
exit 0