In an effort to better manage the software provided on the HPC a new directory structure has been implemented for installed software. The result is influenced by the Linux Standards Base, the Filesystem Hierarchy Standard, and the Linux Assigned Names and Numbers Authority.

Software available for Red Hat Enterprise Linux (RHEL) 6 that is provided by Red Hat or other vendors and is easily integrated into our HPC system are in the usual expected paths (/bin, /usr/bin, etc). Software packaged specifically for the HPC has been grouped first by compiler (GNU, Intel, and PGI) and then by MPI implementation (OpenMPI, MVAPICH, MVAPICH2). This gives 12 total configurations. Software can be installed either locally in /opt/hpc, or on Panasas distributed storage in /panfs/storage.local/opt/hpc.

To make managing software environments easier the HPC will be providing Environment Modules to aid users in loading and unloading paths. By default, all GNU paths are loaded and all HPC software compiled using GNU compilers are available. Users may then load an Environment Module to access GNU-OpenMPI compiled software, Intel compiled software, PGI-MVAPICH2 compiled software, or any other compiler/MPI combination.

HPC users are encouraged to use Environment Modules instead of manually manipulating PATH, LD_LIBRARY_PATH, and other environment variables.

Our legacy profile.d scripts will be changed so that they use the module system. In almost all cases users loading paths with profile.d scripts should not be affected by this change.

After the September upgrade, use the module command to view and activate available configurations. You will see output such as:

[user@submit ~]$ module whatis
gnu-mvapich          : Set paths for GNU MVAPICH compiler & programs.
gnu-mvapich2         : Set paths for GNU MVAPICH2 compiler & programs.
gnu-openmpi          : Set paths for GNU OpenMPI compiler & programs.
intel                : Set paths for Intel compiler & programs.
intel-mvapich        : Set paths for Intel compiled MVAPICH & programs.
intel-mvapich2       : Set paths for Intel compiled MVAPICH2 & programs.
intel-openmpi        : Set paths for Intel compiled OpenMPI & programs.
pgi                  : Set paths for PGI compiler & programs.
pgi-mvapich          : Set paths for PGI compiled MVAPICH & programs.
pgi-mvapich2         : Set paths for PGI compiled MVAPICH2 & programs.
pgi-openmpi          : Set paths for PGI compiled OpenMPI & programs.


[user@submit ~]$ module load intel-openmpi
[user@submit ~]$ module list
Currently Loaded Modulefiles:
  1) intel           2) intel-openmpi

[user@submit ~]$ module unload intel-openmpi
[user@submit ~]$ module list
Currently Loaded Modulefiles:
 1) intel

[user@submit ~]$ module purge
[user@submit ~]$ module list
No Modulefiles Currently Loaded.

The complete list of paths managed by the Environment Modules, as described, is listed below. Each prefix listed contains bin, lib, share, etc directories.

Default Software Prefixes:
* / - system core software
* /usr - userspace core software, GNU development tools, system utilities
* /usr/local - HPC installed utilities, links to matlab, blender, etc
* /opt/hpc/gnu - HPC installed GNU software
* /panfs/storage.local/opt/hpc/gnu - HPC installed GNU software

Non-default paths loaded by Environment Modules:
* /opt/hpc/gnu/mvapich
* /opt/hpc/gnu/mvapich2
* /opt/hpc/gnu/openmpi
* /panfs/storage.local/opt/hpc/gnu/mvapich
* /panfs/storage.local/opt/hpc/gnu/mvapich2
* /panfs/storage.local/opt/hpc/gnu/openmpi

* /opt/hpc/intel
* /opt/hpc/intel/mvapich
* /opt/hpc/intel/mvapich2
* /opt/hpc/intel/openmpi
* /panfs/storage.local/opt/hpc/intel/mvapich
* /panfs/storage.local/opt/hpc/intel/mvapich2
* /panfs/storage.local/opt/hpc/intel/openmpi

* /opt/hpc/pgi
* /opt/hpc/pgi/mvapich
* /opt/hpc/pgi/mvapich2
* /opt/hpc/pgi/openmpi
* /panfs/storage.local/opt/hpc/pgi/mvapich
* /panfs/storage.local/opt/hpc/pgi/mvapich2
* /panfs/storage.local/opt/hpc/pgi/openmpi