Over at GridEngine.info they found a link on DanT’s Sun blog that has a sweet tutorial on setting up Hadoop using SGE’s parallel environments with loose integration.
Here we are relying on master node to start othe daemons ( [rs]sh the machine and start daemons) and distribute jobs , and we donot have control on the TaskTracker threads. This way of setting a pe in Grid Engine is called loose-integration
With some more effort one could also achieve a tighter integration wherein the task of starting daemons and tasks on other slaves could be done by SGE. But this would require further understanding of Hadoop internals.
Pretty dope.
Joe does your brain hurt cause after reading that my does
\o/
And I thought I was a GEEK
Maybe a little pain but it’s worth it.
It’s really sweet stuff.