Use top in batch mode
Assume a system is going down, and there's no explanation as to why, but perhaps oom_killer is being invoked. You might use top in batch mode to catch whatever process is causing a memory leak.
Whenever you run top, you can press Shift+F so that you can select the field by which to sort. We're looking for a memory leak so we would select 'N'. Note: 'N' is not a typo here:) Press enter to see each process, in order of the percentage of memory consumed. (You could also type lowercase 'f' to choose which fields are displayed.)
We'll want top, in batch mode, to open with this view the next time, and to do this, press Shift+W. This will save a file '.toprc' in the home directory. Now when top opens up it will have exactly the fields we wanted to be monitored, and the exact order. Back to batch mode!
top -b -n 1
Simple so far, b for batch mode; n for the number of iterations. I like to use just one, throw this into a script, and let cron handle the iterations. Like this:
date +'%Y%m%d %H:%M' >> /root/monitor.log
free -g >> /root/monitor.log
top -b -n 1 | head -n 12 | tail -n 5 >> /root/monitor.log
echo “-----” >> /root/monitor.log
Then you can specify the script to run every five minutes -
*/5 * * * * /root/bin/monitor.sh
In short, this will place a time stamp, followed by memory used in Gigabytes, followed by 5 of the processes using the most memory in /root/monitor.log.