Job Types Demystified (sort of)

There are three distinct job types that you must choose when you submit a job to the cluster.

1. (s)imple
The simple case is straightforward, you submit any number of distinct separate 1 cpu jobs.
2. (p)ool parallel
The first parallel job type uses the Matlabpool universe, which gives your code access to distributed parfor functionality.
3. (m)pi parallel
The second parallel job type uses the Matlab parallel universe, which gives your code access to parallel distribution using labindex/labnum functionality.

The two parallel job types and their functionality cannot be intermixed. If your code uses parfor loops and labindexing and you submit using the pool parallel job type, the code will never exit the function and will only time out when the scheduler kills it.

Matlabpool Parallel versus MPI Parallel

The Matlabpool can be viewed as a higher level parallel environment - you do not control the message passing, Matlab will do it automatically with their parallel functions. Only one copy of your function is run on a "client" session that takes care of everything for you in the background.

The mpi parallel job type is a lower level parallel environment - you control the message passing. Each lab of your workers receives a copy of your function and you determine how your code is parallelized based on the number of workers and labindexes.

There is one set of functions that can be mixed between the two parallel paradigms, distributed functions. But from the tests I have run, the results are vague and misleading how to implement the distributed functions correctly between the matlabpool and the mpi parallel environment. Both will distribute arrays but in the matlabpool environment, you have no access to the localparts of the arrays and many times, simple tests fail to return any results. In the mpi parallel environment, you have access to each local array part based on the labnum of the worker, but if you attempt to gather the array, each worker gets a full copy of the array. Best advice is to test your function in each environment to see how the code will behave.