Recently, the server has been crashed frequently for programs costing most of the memories. To avoid this, two suggestions are prosed.

• One is more flexible by changing the maximum memory usage for one user locally or globally. Here I suggest users set this variable locally in case they want to run programs need lots of memories by adding ulimit -v 30000000 to .bash_profiles (30,000,000 means maximum 30G is allowed). If one want to run programs cost larger than 30G RAM, just change this value and re-login.

• Another is running the following program serverWatchman to kill programs cost more than 90% memories.

#!/bin/bash

#84G is the maximally allowed total memory usage.
maxmem=84

while true
do
	mem=`free -g | grep -- '^-' | awk '{print $3}'`
	echo ${mem}
	echo ${maxmem}
	if [ ${mem} -gt ${maxmem} ]; then
		info=`ps aux | sort -k4nr | head -n 1`
		id=`echo ${info} | awk '{print $2}'`
		#echo $id
		echo ${info} | wall
		kill -9 $id
		# Be merciful, grieve the death of the process
		sleep 60s
		#exit 1
		#maxmem=10000
	fi
	sleep 5s
done