User Tools
submitting_many_jobs
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
submitting_many_jobs [2020/08/14 10:47] root |
submitting_many_jobs [2021/02/20 09:51] (current) root |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Submitting Many Jobs ====== | ====== Submitting Many Jobs ====== | ||
| + | |||
| + | You may need to run many (possibly similar) tasks on the cluster. You can do this by submitting each job separately to the cluster, techniques for doing that are discussed below. You could also do it by submitting a small number of jobs (possibly even just one) and have those jobs execute the many tasks you need to run. You should try not to submit thousands of separate jobs into the queue. | ||
| To submit many jobs to the cluster you can: | To submit many jobs to the cluster you can: | ||
| Line 37: | Line 39: | ||
| < | < | ||
| - | sbatch -n 2 run_multi_job_o | + | sbatch -n 2 run_multi_job_%t |
| </ | </ | ||
| Line 48: | Line 50: | ||
| Some other options for limiting the number of cores your jobs use are described below. | Some other options for limiting the number of cores your jobs use are described below. | ||
| + | |||
| + | ==== Singleton Jobs ==== | ||
| + | |||
| + | You can use the **--dependency** option of sbatch to make slurm run just one of your jobs at a time. Suppose you submit 10 jobs with the same job name, using the **--dependency=singleton** option will make slurm run these jobs one at a time. | ||
| + | |||
| + | < | ||
| + | for i in $(seq 1 10); do | ||
| + | sbatch --job-name oneatatime --dependency=singleton my_script.bash file_${i}.fasta | ||
| + | done | ||
| + | </ | ||
| ==== Sending a Job to a Specific Node ==== | ==== Sending a Job to a Specific Node ==== | ||
| Line 163: | Line 175: | ||
| So the " | So the " | ||
| - | This technique also works " | + | This technique also works " |
| < | < | ||
submitting_many_jobs.1597416456.txt.gz ยท Last modified: 2020/08/14 10:47 by root
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: Public Domain
