Over at GridEngine.info they found a link on DanT’s Sun blog that has a sweet tutorial on setting up Hadoop using SGE’s parallel environments with loose integration.
Here we are relying on master node to start othe daemons ( [rs]sh the machine and start daemons) and distribute jobs , and we donot have control on the TaskTracker threads. This way of setting a pe in Grid Engine is called loose-integration
With some more effort one could also achieve a tighter integration wherein the task of starting daemons and tasks on other slaves could be done by SGE. But this would require further understanding of Hadoop internals.
Pretty dope.
4 responses so far ↓
1 Uncle Ed // May 24, 2008 at 11:16 pm
Joe does your brain hurt cause after reading that my does
\o/
2 Uncle Ed // May 24, 2008 at 11:18 pm
And I thought I was a GEEK
3 joe // May 25, 2008 at 6:25 pm
Maybe a little pain but it’s worth it.
It’s really sweet stuff.
4 Hadoop / SGE Grid Engine Convergence | Allen Day's Weblog // Aug 8, 2008 at 5:24 pm
[...] I’m an old hand with SGE and a more user of Hadoop / Pig. Good to see that there is interest in making these technologies interoperate. [...]
Leave a Comment