Job Dependencies

Using sbatch or srun you can make the start of your job dependent on the successful completion (or failure) of a prior job. This could be useful if you have some data preparation to do, followed by your analysis.

This code uses srun -d afterok:NNNNN to wait for some jobs submitted using sbatch to complete.

Warning: This only seems to work if you run the control script directly on the head node! (i.e. if you use sbatch or srun to execute the script the code does not actually wait for the dependencies to complete). Not sure why.

#!/bin/bash

dependencies="afterok"
while read pheno; do
if [ -n "${pheno}" ]; then
  job = `( sbatch -o $pheno.%j.out prepare_pheno $p $pheno | cut -f 4 -d ' ' )`
  dependencies+=:$job
fi
done < phenotypes.txt

# Wait for these jobs to complete (trick)
srun -d $dependencies echo "All phenotype preparation done"

# Run for each chromosome
dependencies="afterok"
for chr in {1..22}; do
  job = `(sbatch -o chr${chr}.%j.out run_model.sh $p $chr | cut -f 1 -d ' ')`
  dependencies+=:$job
done

# Wait for these jobs to complete (trick)
srun -d $dependencies echo "All analyses done"

# Do more stuff...