This is an old revision of the document!
Form the head node you can use the sstat command to check on memory use (and other details) of your jobs.
sstat --format=JobID,MaxRSS 1234.batch
The “.batch” part of the command above is literal (i.e. it is not a placeholder for some other piece of information, actually type “.batch”). This selects the memory usage of the submitted slurm batch job. If your job uses job steps you can replace the word “batch” with the job-step number, or use the “–allsteps” option to get details for all job steps.
Without the format option:
sstat 1234.batch
will give you a lot of details about your job including memory and CPU usage.
Suppose you have a program to run and you expect it to use a lot of memory, but are unsure of exactly how much.
Files: memmon, memlog, leakmem
srun memmon
leakmem allocates 1MB of RAM every second (for up to 10000 seconds).
We use the SLURM_JOB_ID environment variable to get a unique name for the log file.
/proc/PID/smaps provides a detailed view of the memory being used by a specific process. You can parse it (or use some previously written Perl code, Linux::Smaps.pl).
Files: memmon2, memlog2, leakmem.