The following script will give you the total of memory usage of your desired processes and gives you message when it reaches more than targeted value.

Basically, this script just gets the sum of values in RSS column from ps aux command.

core:scripts# cat check-process-memory.sh 
#!/bin/bash

#process
PROCESS=$1

#exit script if $PROCESS is not mentioned
if [[ -z $PROCESS ]]
then
  echo "FATAL: You forgot to mention something!"
  exit 99
fi

#memory alert threshold
mem_alert=$2

#get memory usage of $PROCESS
mem=$(ps aux | grep $PROCESS | awk '{total+=$6} END{print total}')

if [ $mem -gt 1024 ] && [ $mem -lt 1024000 ];
 then
        #MB
        mem=$(expr ${mem} / 1024)
        mem="${mem}MB"
elif [ $mem -gt 1024000 ];
 then
        #GB
        mem=$(expr ${mem} / 1024 / 1024)
        mem="${mem}GB"
else
        mem="${mem}KB"
fi


case $mem_alert in
"") echo "NOTICE: $PROCESS is using ${mem}" ;;
*)
if [[ $mem -gt $mem_alert ]]
  then
    # display the alert
      echo "ALERT: $PROCESS is using ${mem}!"
fi
;;
esac

And the output will be:

core:scripts# ./check-process-memory.sh apache 300000
ALERT: apache is using 500808KB!

On the following, I have created a script which will ask you to kill "sleep" processes if they utilize more memory than mem_alert:

#!/usr/bin/ksh

#memory alert threshold 
mem_alert=209

#get sleep memory  usage
mem=$(ps aux | egrep -v "grep|awk" | awk ' /sleep/ {total+=$6} END{print total}')

#exit script sleep process is not running 
if [[ -z $mem ]]
then
  exit 0
fi

if [[ $mem -gt $mem_alert ]]
then
  echo "ALERT: sleep memory usage is ${mem}KB!"
  pidslist=$(ps aux | grep -v grep | grep sleep | awk '{print $2}' | xargs)
  echo "The following pids must be killed: ${pidslist}"
  print "Would you like to attempt to kill the processes? [Yes|No]"
  read answer
  case $answer in
        Yes|yes|Y|y)
        for pid in $pidslist; do
                printf "pid $pid will be killed...\r"
                kill -9 $pid 2>/dev/null 1>/dev/null && echo "The following pid has been killed: $pid"
        done
        printf "\nNow please run it again to double check!\n"
        ;;
        No|no|N|n)
        echo "Operation stopped at your request, nothing has been killed."
        ;;
  esac
fi