The cluster is running SLURM's default scheduler which is very (maybe too) simple.

• first-in first-out queue.
• Simple round-robin node selection.

The scheduler does not check which nodes are busy and try to avoid them.

This has an advantage in that it tends to leave some nodes empty for people who want a whole node.

You can use the -w option to select a specific node. Actually it asks for “at least” the nodes in the node list you specify. So a command like:

sbatch -n 20 -w node2

Would get you some cores on node2 and some on another node (since there is only 16 cores on node2). If there were no cores free on node2 the job would be queued until some became available.

You can use the -x option to avoid specified nodes.

You can use -c 16 to request all cores on a (standard) node.

You can use the –exclusive option to as for exclusive access to all the nodes your job is allocated. This is especially useful if you have a program which attempts to use all the cores it finds.