Use nvidia-smi to monitor GPU use.
srun -p gpu nvidia-smi -q srun -p gpu nvidia-smi -q -d UTILIZATION srun -p gpu nvidia-smi -q -d MEMORY srun -p gpu nvidia-smi -q -d PIDS
Using this program through srun depends on at least one core being free. You could also start a “watchdog” thread that runs it occasionally as part of your script.