This article describes how to use the cluster-openmp extension of Intel's icc C/C++ compiler. Since this is an experimental setup, there is no support yet for the Fortran compiler.

Open Multi-Processing (openmp) programs on the HPC cluster can only utilize the resources of a single compute node, which limits it to only 4 threads. Cluster-openmp is an extension to this paradigm and it lets openmp programs run on several nodes.

Setup

In this article I assume that your program is openmp-enabled; i.e. it is made parallel using the appropriate openmp pragma's. In most cases, you don't have to change your openmp code, or only add the following code and use MALLOC instead of malloc:

#ifdef _CLUSTER_OPENMP
#define MALLOC kmp_sharable_malloc
#else
#define MALLOC malloc
#endif

An extensive guide to cluster-openmp can be found on the Intel website.

All files used in this document can be find in the /panfs/storage.local/system/tutorial/example3 directory on the cluster.

Compile your program

To compile your openmp program for cluster-open, you have to use the -cluster-openmp compiler flag, instead of the -openmp flag. For example,

$ source /usr/local/profile.d/iccvars.csh
$ icc -cluster-openmp -o jacobi jacobi.c
jacobi.c(72): (col. 4) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(72): (col. 4) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(198): (col. 1) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(205): (col. 1) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(104): (col. 4) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(110): (col. 4) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(131): (col. 1) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.
jacobi.c(244): (col. 1) remark: OpenMP DEFINED LOOP WAS PARALLELIZED.

Submitting your job

The cluster-openmp implementation expects certain runtime settings in a file called kmp_cluster.ini. This includes the nodes the program is supposed to run on. Since this information is only available during runtime, we let the submission script create this file.

An example submission script can look like,

#!/bin/bash

#MOAB -l nodes=2:ppn=4
#MOAB -j oe

#MOAB -N CLUSTER-OPENMP

source /usr/local/profile.d/iccvars.sh

hosts=`cat $PBS_NODEFILE | uniq | xargs | sed -e 's/ /,/g'`

cd $PBS_O_WORKDIR

# process_threads: number of threads on each host
echo "--process_threads=4 --launch=ssh --hostlist=${hosts}" > kmp_cluster.ini

./jacobi --sharable-heap=300M << EOF
10
10
1
0.01
0.0001
10000
EOF

rm -f kmp_cluster.ini

Some remarks about this script:

• The test program, jacobi requires some settings from the user. These settings are hard-coded in the script and you probably don't need this.
• Cluster-openmp programs can be very memory intensive and you might need to change the heap size for your program. You can do this by using the --sharable-heap switch.
• We want to use as much processors per node as possible (4 for the current HPC cluster). However, this can result in our job waiting in the queue a little longer. The job has to wait until a set of complete nodes become available instead of a number of processors.

License

We are using a evaluation license, which will expire on Feb 10, 2008. This licence is for the C/C++ compiler. Please email hpc-support if you are interested in using cluster-openmp for Fortran.