[NOTE: The 3Leaf Systems SMP systems are currently offline until further notice.]

The FSU HPC facility has installed a dynamic computing framework from 3Leaf Systems (now defunct) that allows an application to seamlessly access CPUS and memory from 12 separate x86_64 AMD based computers. In addition to this dynamic system, there are two Dell 4-socket computers which have more memory and cores than the regular Dell compute nodes.

The new queue, smp_q, will accept jobs that will run on one of these 2 distinct resources. The 3leaf Systems currently only supports submission of applications that are compiled using threads (i.e., compiled with openMP directives, not MPI). The Dell computers will accept both MPI and openMP compiled jobs requesting 24 cores or less and will match jobs greedily (e.g., two jobs requesting 16 cores will run simultaneously on their own computer, even though 8 cores will be left idle on Dell Node 2).

To access only one or the other of the two distinct hardware resources, you can request the feature of "dell" or "3leaf". If not, your job will be placed on any of the available resources. One of the advantages of the new queue is that only a single job will run on a resource (to have access to the full memory in the system). Besides requesting the smp_q, the only difference between a regular submission and a submission to this new queue is that the requested processors must be in the form:

#MOAB -l nodes=1:ppn=X 

With the "X" value of the processors per node attribute (ppn) equal to one of the following core values:

Or if you only want to run on one or the other distinct hardware resources:

#MOAB -l nodes=1:ppn=X -l feature=dell

#MOAB -l nodes=1:ppn=X -l feature=3leaf

Dell Node access (accepts both MPI and openMP jobs)

X = 1 - 24 cores

(e.g., nodes=1:ppn=12, nodes=1:ppn=13, nodes=1:ppn=14, etc.)

Hardware Configuration

Node 1

• • 16 cores (4 AMD Quad-Core 8382, 2.6 GHz)

• • 64 GB memory

Node 2

• • 24 cores (4 AMD Six-Core 8431, 2.4 GHz)

• • 64 GB memory

3leaf Node access (accepts only openMP jobs)

The default setup is 3 nodes of X = 44 cores each (if you would like access from 55 - 132 cores in increments of 11, please email This e-mail address is being protected from spambots. You need JavaScript enabled to view it and we can set up that configuration).

Hardware Configuration

Node Size

• • 44 cores (4 Nodes of AMD Six-Core 8435, 2.6 GHz)

• • 184 GB memory

Queue policy

The queue has a maximum wallclock of 48 hours (policy change as of 6/17/2011) until the scheduler will correctly preempt jobs that run longer.

MPI

Since the MPI versions installed on the HPC are tightly integrated with our InfiniBand infrastructure and there is no Infiniband available on the 3leaf systems, we have temporary disabled MPI on the 3leaf systems. For the moment, you can only run single task programs with large memory requirements or SMP multi-threaded (e.g. OpenMP) programs. We are working on a solution for this. You can still run smaller MPI programs on the 4-socket Dell computers (24 cores or less).

Example Submission Script

#!/bin/sh

#MOAB -l nodes=1:ppn=44
#MOAB -N smp_test
#MOAB -joe
#MOAB -q smp_q

cd $PBS_O_WORKDIR
./a.out

Beta disclaimer

The 3leaf system is cutting edge technology. That means that most of the time your job will run fine, however, sometimes the individual nodes are not initialized in a "free"/"occupied" state after the end of a job or before a new job. In this case, no new jobs can get scheduled on that node. If you run showq -w class=smp_q and the run queue is empty for at least 30 minutes, but there are jobs in the queue, then this might be the case. It can take around 20 to 30 minutes for a virtual machine to be setup, so be patient if your job does not immediately starts. If no jobs start to run after 30 minutes, please contact This e-mail address is being protected from spambots. You need JavaScript enabled to view it , with all the relevant information like the output of checkjob -v your-job-id.