Skip to main content

openmp

About

OpenMP is a shared-memory/multi-threading programming model for writing parallel processing codes. It is based on inserting hints into the source code, which the compiler uses in conjunction with the normal language features to enable multi-threading.

Version and Availability

This is not really a stand-alone piece of software. It is built into the various compilers, include those from Intel, PGI, and GCC. It requires the use of the appropriate compiler switch to activate at compile time. Please refer to the documentation for your preferred compiler for details.

Usage

OpenMP represents a programming methodology that is supported by the compilers found on the local clusters. It is a complex topic that can't be covered here. Please visit the Moodle-based tutorials, HPC113 - Introduction to Shared Memory Programming with OpenMP and HPC114 - Introduction to Hybrid Programming, for details on how to program with OpenMP.

Interactive Jobs

This sequence of commands illustrates how an OpenMP job can be run on an interactive compute node. The node is requested via QSub, and 4 processors are assumed to be available. (This number depends on the machine being used.). OMP_NUM_THREADS is used to specify how many threads to use when executing.

▶ Open Example?
$ qsub -I -l nodes=1:ppn=4 -l walltime=00:10:00
$ export OMP_NUM_THREADS=4
$ ./program.ex

Batch Jobs

Running as a PBS job in batch mode requires a script such as the following. Be sure to adjust the PBS options to match the requirements on the machine being used.

▶ Open Example?
#!/bin/bash
#
# No shell commands until PBS is set up.
#
# Use the default job queue (workq):
#PBS -q checkpt 
#
# Specify your project allocation code
#PBS -A ALLOCATION_CODE
#
# Set number of nodes and number of processors on each 
# node to be used. Use ppn=8 for QB, and ppn=4 for all other x86s.
#PBS -l nodes=1:ppn=4 
#
# Set amount of time job is allowed to run in hh:mm:ss
#PBS -l walltime=00:15:00 
#
# Send stdout messages to a named file:
#PBS -o OUT_NAME 
#
# Merge stderr messages with stdout.
#PBS -j oe 
#
# Give job a name for easier tracking.
#PBS -N JOB_NAME
#
# Shell commands may begin here.

cd /path/to/your/executable
export OMP_NUM_THREADS=4
./program.ex

Hybrid Jobs

Since OpenMP threads a limited to a single node, the limit is the number of processing cores on the node. However, it is possible to combine OpenMP with MPI and distribute work across more nodes. OpenMP is used for processing the work on a node, and MPI is used to exchange information between nodes. This can get complicated, and execution requires a little more finesse. The important thing to keep in mind is that only one MPI task is run per node, so some steps are necessary to make sure only one copy of the program is started on each node. The follow PBS script is an example of how to accomplish this.

▶ Open Example?
#!/bin/bash
#
# No shell commands until PBS is set up.
#
# Use the default job queue:
#PBS -q workq
#
# Specify the appropriate project allocation code
#PBS -A ALLOCATION_CODE
#
# Set number of nodes and number of processors on each
# node to be used: ppn=8 for QB, and ppn=4 for all other x86s.
#PBS -l nodes=4:ppn=4 
#
# Set time job is allowed to run in hh:mm:ss
#PBS -l walltime=00:15:00 
#
# Send stdout messages to a named file:
#PBS -o OUT_NAME 
#
# Merge stderr messages with stdout.
#PBS -j oe 
#
# Give job a name for easier tracking:
#PBS -N JOBNAME
#
# Shell commands many begin here.

export WORK_DIR=/work/uname/path
cd $WORK_DIR
cat $PBS_NODEFILE | uniq > ./mpd_nodefile_$USER

# get number of MPI processes 
export NPROCS=`wc -l mpd_nodefile_$USER |gawk '//{print $1}'`
# setting number of OpenMP threads (8 for QB, 4 otherwise)
export OMP_NUM_THREADS=4
ulimit -s hard

# launch your hybrid applications 
mpirun_rsh -np $NPROCS -hostfile mpd_nodefile_$USER \
       OMP_NUM_THREADS=$OMP_NUM_THREADS ./program.ex

Resources

  • OpenMP home page with links to the OpenMP specification and related information.

Last modified: January 22 2013 13:19:55.