Cluster job peak memory usage: profiling the peak memory usage of a job run on the cluster

If it is useful to check the peak memory usage of a job that has run on the cluster (e.g. to determine an appropriate memory requirement for your qsub command for future cluster jobs), the following code snippet regularly checks the peak usage up until that moment of a specific process run as part of the cluster job:

runYourNiceProcess & #Important that this is run in the background

peakvmem = "0 kB"

cat /proc/${!}/status | awk '/VmPeak/{print $2" "$3}' &> tempVmPeak.txt
while [ -z "$(kill -0 $!)" ]; do
  peakvmem="$(cat tempVmPeak.txt)"
  cat /proc/${!}/status | awk '/VmPeak/{print $2" "$3}' &> tempVmPeak.txt
  sleep 10
done

This works by accessing /proc/PID/status to find information about your process (which has a process id of PID). ${!} returns the PID of the last process put in the background, and /proc/PID/status is the direct output of various statuses of this process, including peak memory usage. kill -0 $! won't actually do anything, but will return an empty string if the process exists, and a "No such process" error if the process does not exist.

The peakvmem variable is set from tempVmPeak.txt to avoid the (hopefully unlikely) scenario that the process ends between the while condition check and /proc/PID/status being read, as the /proc/PID folder will disappear once the process has ended. As such, you would just get a "No such file or directory" error if you tried to access /proc/PID once the process has finished.

Callum Kilby's Tutorials and Useful Code

Useful Code

krenewRun: running local jobs without your kerberos/AFS tokens either expiring or being lost

An issue when running jobs locally, particularly in a screen, is that kerberos and AFS tokens can either expire, or in the case of working on lxplus, will be cleared when you log off. To get around this, I use a script called krenew. This script generically allows you to start a daemon that copies your current tokens, and periodically refreshes them as desired.

I have made a wrapper for krenew, named krenewRun, to run any command line code passed in, with the tokens copied by krenew being refreshed until the process started by the command line code has finished running. This wrapper also performs a check for whether AKLOG is defined. aklog is the function that deals with AFS tokens, and AKLOG is the path to this function. krenew needs this function when handling your AFS tokens.

To make use of krenewRun, copy the below code into your ~/.bash_profile or ~/.bashrc. Usage is simply krenewRun "COMMAND", where COMMAND is whatever text you would have entered into your command line, with " marks escaped as \"

krenewRun( ) {
    if [ "$1" == "-h" ]
    then echo 'Usage: kenewRun "COMMAND"'
    else
        if [ -z $AKLOG ]
        then echo 'Please define AKLOG variable to be path to aklog (e.g. /usr/bin/aklog)"
        else
            krenew -t -- sh -c "$1"
        fi
    fi
}       

-- CallumKilby - 12 Apr 2016