Click to enlarge

Changing the 'niceness' of only one type of daemon While recently playing with our cluster during a Terasort benchmark I realised just how dumb it was to leave everything running with the default level of niceness (0).  SSH sessions were timing out, reporting was going haywire, al-sorts of fun. I know the Cloudera CDH3 distribution allows you to set a global HADOOP_NICENESS level, but I was hesitant about dropping the priority of essential tasks like the namenode/datanode/jobtracker.  However I wouldn't mind if a tasktracker looses priority to a Ganglia ping - I want my pretty charts! After poking around a bit I realised that the "/etc/hadoop/conf/hadoop-env.sh" script is included at a perfect spot within the start-up scripts for all daemons.  This allows you to detect what is starting and mess with its parameters.  Tada, the following addition to your own hadoop-env.sh will result in the tasktrackers being prioritised slightly less than everything else.

# This hack drops the priority of mapred tasks
if [ "$command" = "tasktracker" ]; then
export HADOOP_NICENESS=5
fi