How can I control the job node distribution topology?

The job node distribution topology is configured principally by the scheduler.  There are two options, to match the exact processors per node or the exact number of nodes requested.   Most users experience better performance with the exact node option and that is what has been implemented.   Users may control the distribution of nodes on a job-by-job basis.   If a user needs to request a specific number of processors per node (ppn), then the command line should be modified (or the job script) to include the following flags:

-W x=nmatchpolicy:exactproc

For example to request 8 node with 4 cores per node (32 cores in total), one can use

$ msub -l nodes=8:ppn=4  -W x=nmatchpolicy:exactproc example.sh 

 Equivalently, you can add to your job script:

#MOAB -W x=nmatchpolicy:exactproc

The reference for these command is at clusterresources.com.