Use nvidia-smi to monitor GPU use.

srun -p gpu nvidia-smi -q
srun -p gpu nvidia-smi -q -d UTILIZATION
srun -p gpu nvidia-smi -q -d MEMORY
srun -p gpu nvidia-smi -q -d PIDS

Using this program through srun depends on at least one core being free. You could also start a “watchdog” thread that runs it occasionally as part of your script.