Hadoop Hack: TaskTracker priority
Changing the 'niceness' of only one type of daemon While recently playing with our cluster during a Terasort benchmark I realised just how dumb it was to leave everything running with the default level of niceness (0). SSH sessions were timing out, reporting was going haywire, al-sorts of fun. I know the Cloudera CDH3 distribution allows you to set a global HADOOP_NICENESS level, but I was hesitant about dropping the priority of essential tasks like the namenode/datanode/jobtracker. However I wouldn't mind if a tasktracker looses priority to a Ganglia ping - I want my pretty charts! After poking around a bit I realised that the "/etc/hadoop/conf/hadoop-env.sh" script is included at a perfect spot within the start-up scripts for all daemons. This allows you to detect what is starting and mess with its parameters. Tada, the following addition to your own hadoop-env.sh will result in the tasktrackers being prioritised slightly less than everything else.
# This hack drops the priority of mapred tasks
if [ "$command" = "tasktracker" ]; then
export HADOOP_NICENESS=5
fi
The Cutler.sg Newsletter
Weekly notes on AI, engineering leadership, and building in Singapore. No fluff.
Everyone Can Have a Personal Health Adviser Now
Personalized medicine used to mean being rich enough to afford a doctor who knew your name. Last week I built a version of it on my laptop, for free, from a file I'd been ignoring for seven years — and the real unlock is that I can re-run it forever.
Protect the Juniors: Cognitive Debt and the Stack Overflow Collapse
AI is making junior output look senior-level while preventing junior skill from forming — and the Stack Overflow collapse just removed the ambient learning layer that used to catch the deficit. Three interventions that work.
AI as the Great Equaliser: Neurodiversity, Disclosure, and the Tools That Change Everything
For neurodivergent professionals, AI isn't just a productivity tool — it's the first accommodation you can access privately, without disclosure, without stigma, and without asking anyone's permission.