Using ScaLAPACK

ScaLAPACK 2.0.2 has been installed on Brazos. It has been compiled against ACML, which include the BLAS and LAPACK routines. It has also been compiled against two MPI implementations, MVAPICH2 and Open MPI.

module load gcc acml mvapich2 scalapack
-or-
module load gcc acml openmpi scalapack

Example: Compile your code and link with the the scalapack and acml (single-threaded) or acml_mp (multi-threaded) libraries:

mpif90 myapp.f90 -o myapp -lscalapack -lacml
-or-
mpif90 myapp.f90 -o myapp_mp -lscalapack -lacml_mp

If you application will use 32 processes on 4 nodes with 300MB memory per cpu for 1 hour in the mpi-core8 partition using the single-threaded ACML library, your SLURM script would look like this:

#!/bin/batch
#SLURM -N 4
#SLURM -n 32
#SLURM -t 01:00:00
#SLURM -p mpi-core8
#SLURM --mem-per-cpu=300m

echo "ScaLAPACK single-threaded job $SLURM_JOBID started at `time`"

module load gcc acml mvapich2 scalapack

srun --mpi=none ./myapp
echo "ScaLAPACK single-threaded job $SLURM_JOBID ended at `time`"

exit 0

For the multi-threaded case, you would still request 32 processes. The only change is the executable that you run.

#!/bin/batch
#SLURM -N 4
#SLURM -n 32
#SLURM -t 01:00:00
#SLURM -p mpi-core8
#SLURM --mem-per-cpu=300m

echo "ScaLAPACK single-threaded job $SLURM_JOBID started at `time`"

module load gcc acml mvapich2 scalapack

srun --mpi=none ./myapp_mp
echo "ScaLAPACK single-threaded job $SLURM_JOBID ended at `time`"

exit 0

Documentation for ScaLAPACK is available from Netlib's ScaLAPACK site.