Using MVAPICH2

MVAPICH2 is freely available implementation of the Message Passing Interface designed for high speed interconnect fabrics. This version of MVAPICH2 on Brazos was built with the 64-bit GCC and Intel compiler suites.

Environment

To use MVAPICH2 you will need to load the appropriate environment variables. This can be done with the Modules package.

To use MVAPICH2 compiled with GCC:
module load gcc
module load mvapich2

To use MVAPICH2 compiled with Intel:
module load intel
module load mvapich2

In your SLURM batch scripts be sure to load the same modules:

#!/bin/bash
#SLURM -J myjob
#SLURM -p mpi-core8
#SLURM -t 01:00:00
#SLURM -o myjob-%J.out
#SLURM -N 4
#SLURM --ntasks-per-node=8
module load gcc
module load mvapich2
cd subdir
srun --mpi=none ./myapp
exit 0

Compiling

As with other MPI implementations, to compile a MVAPICH2 application, use the mpicc, mpicxx, mpif77, or mpif90 commands. These will call the GNU C, C++, FORTRAN 77 and 90 compilers, respectively. Be sure to pass any optimization flags (e.g., -O) to these commands just as you would for the Intel compilers.

Examples:
mpicc -O myapp.c -o myapp -s -lm
mpicxx -O myapp.cc -o myapp -s -lm
mpif77 -O myapp.f -o myapp -s -lm
mpif90 -O myapp.f90 -o myapp -s -lm
(all text for each case above should be on a single line).

Running

All parallel applications must be launched from a SLURM batch script. MVAPICH2 applications are launched using SLURM's srun command.

MVAPICH2 should not be used on the nodes connected with Gigabit Ethernet. It should only be used on nodes connected via InfiniBand or on a single node. The mpi-core8 and mpi-core32 SLURM partitions will have access to 128 and 26 nodes respectively with DDR InfiniBand interfaces.

The nodes with InfiniBand are available through SLURM by using the mpi-core8 (8 cores / node) or mpi-core32 (32 cores / mode) partitions. This can be specified in your SLURM script or on the command line:
sbatch -p mpi-core8 -N 8 -n 32 myibapp.slrm
In addition, the SLURM script needs to be modified to facilitate using MVAPICH2 across the InfiniBand fabric.

Here's an example which will run on 4 nodes using all 8 cores on each node.

Example script (download):


#!/bin/bash
#SLURM -J myjob
#SLURM -p mpi-core8
#SLURM -t 12:00:00
#SLURM -o myjob-%J.out
#SLURM -N 4
#SLURM --ntasks-per-node=8

# The options above specify to use the mpi-core8 partition, requests 4
# nodes with eight processes per node, limits the job to 12 hours of
# walltime, and defines the standard output name as myjob-JOBID.out

# Load the MVAPICH2 GCC environment
module load gcc
module load mvapich2

# Print out the starting time and host.
echo "$SLURM_JOBID started on `hostname` at `date`"
echo "-=-"

# Change to a subdirectory
cd $HOME/mysubdir
 
# Run the application 'myapp' on 32 cores.  The '-f myfile.in' argument
# is passed to myapp. All data between the 'EOT' markers is input to myapp
# that would normally be read from stdin on a terminal.
srun --mpi=none myapp -f myfile.in << EOT
3
10 56.0
A B
9
EOT
echo "-=-"
 
# Print the end time and host.
echo "$SLURM_JOBID ended on `hostname` at `date`"
 
exit 0

Here's another example using the mpi-core32 partition and using 32 processes on 4 nodes (128 total processes).

Example script (download):


#!/bin/bash
#SLURM -J myjob
#SLURM -p mpi-core32
#SLURM -t 12:00:00
#SLURM -o myjob-%J.out
#SLURM -N 4
#SLURM --ntasks-per-node=32

# The options above specify to use the mpi-core32 partition,
# requesting 4 nodes with 32 processes per node, limits
# the job to 12 hours of wall time, and writes stdout to myjob-JOBID.out

# Load the MVAPICH2 GCC environment
module load gcc
module load mvapich2

# Print out the starting time and host.
echo "$SLURM_JOBID started on `hostname` at `date`"
echo "-=-"

# Change to a subdirectory
cd $HOME/mysubdir
 
# Run the application 'myapp' on all 128 cores.  The '-f myfile.in' argument
# is passed to myapp. All data between the 'EOT' markers is input to myapp
# that would normally be read from stdin on a terminal.
srun --mpi=none myapp -f myfile.in << EOT
3
10 56.0
A B
9
EOT
echo "-=-"

# Print the end time and host.
echo "$SLURM_JOBID ended on `hostname` at `date`"
 
exit 0