|Symmetric Multiprocessing Queue (smp_q)|
[NOTE: The 3Leaf Systems SMP systems are currently offline until further notice.]
The FSU HPC facility has installed a dynamic computing framework from 3Leaf Systems (now defunct) that allows an application to seamlessly access CPUS and memory from 12 separate x86_64 AMD based computers. In addition to this dynamic system, there are two Dell 4-socket computers which have more memory and cores than the regular Dell compute nodes.
The new queue, smp_q, will accept jobs that will run on one of these 2 distinct resources. The 3leaf Systems currently only supports submission of applications that are compiled using threads (i.e., compiled with openMP directives, not MPI). The Dell computers will accept both MPI and openMP compiled jobs requesting 24 cores or less and will match jobs greedily (e.g., two jobs requesting 16 cores will run simultaneously on their own computer, even though 8 cores will be left idle on Dell Node 2).
To access only one or the other of the two distinct hardware resources, you can request the feature of "dell" or "3leaf". If not, your job will be placed on any of the available resources. One of the advantages of the new queue is that only a single job will run on a resource (to have access to the full memory in the system). Besides requesting the smp_q, the only difference between a regular submission and a submission to this new queue is that the requested processors must be in the form:
#MOAB -l nodes=1:ppn=X
With the "X" value of the processors per node attribute (ppn) equal to one of the following core values:
Or if you only want to run on one or the other distinct hardware resources:
#MOAB -l nodes=1:ppn=X -l feature=dell
Dell Node access (accepts both MPI and openMP jobs)
3leaf Node access (accepts only openMP jobs)
The queue has a maximum wallclock of 48 hours (policy change as of 6/17/2011) until the scheduler will correctly preempt jobs that run longer.
Since the MPI versions installed on the HPC are tightly integrated with our InfiniBand infrastructure and there is no Infiniband available on the 3leaf systems, we have temporary disabled MPI on the 3leaf systems. For the moment, you can only run single task programs with large memory requirements or SMP multi-threaded (e.g. OpenMP) programs. We are working on a solution for this. You can still run smaller MPI programs on the 4-socket Dell computers (24 cores or less).
Example Submission Script
#!/bin/sh #MOAB -l nodes=1:ppn=44 #MOAB -N smp_test #MOAB -joe #MOAB -q smp_q cd $PBS_O_WORKDIR ./a.out
High Performance Computing @ FSU
Computational resources are a critical part of scientific research and engineering programs and play an increasingly important role in preparing students for careers in both commercial and academics fields. Recent advances in data acquisition, algorithm development, and computer hardware have made High Performance Computing (HPC) fundamentally necessary to remain competitive. FSU has long recognized the importance of local HPC resources for research and education. In 2002 FSU gained the distinct honor of having the largest university-owned computer facility worldwide and 34th largest among all facilities. Since its inception this facility has enabled research programs spanning such disciplines as biology, engineering, physics, meteorology, mathematics, oceanography, and more (see our research page for details).
The FSU HPC mission is to:
- Support multidisciplinary research
- Provide a general access computing platform
- Encourage cost sharing by departments with dedicated computing needs
- Provide a broad base of support and training opportunities