As a computing specialist that works in the Condor, Hadoop, and SGE communities, I wanted to post a follow up to Stephen Wilson’s Jan. 14 blog post on SGE, as it is factually incorrect.
At Cycle Computing, I started working with Condor users five years ago, as well as using Hadoop and SGE with clients in the past two years, in life sciences, insurance, finance, energy and chip design. Condor is very flexible and powerful, but all schedulers have use cases they’re great at.
We’re fans of computation management in general, but we were alarmed by Wilson’s post about Grid Engine being the “World’s First Cloud-Aware Distributed Resource Manager” because it supports machines running in EC2 through a VPN and can schedule Hadoop clusters. Simply put, it isn’t first.
Perhaps Wilson was misinformed about SGE being first with these features and will correct or update his post upon review of the following information.
The Condor Scheduler has had Hadoop Cluster scheduling since 2006, originally by Yahoo! using Condor in its Hadoop on Demand project. Condor has had Amazon EC2 scheduling since 2008. In 2007, CycleCloud offered Condor clusters as a service into the Amazon Cloud. Condor can even be used to do advanced, cost-based scheduling in Amazon EC2, as we discussed here. These dates are well published.
Condor is a freely available resource manager from the University of Wisconsin-Madison, with production users over the past 20 years including for many Fortune 500s and large companies like JP Morgan, Yahoo!, Fair Isaac, Altera and Hartford Insurance, who have also talked at Condor conferences.
For reference, the timeline looks like this:
Yahoo!, the major contributor
behind Hadoop, used Condor to
schedule Hadoop clusters on generic hardware; see Sameer Paranjpye’s
presentation on Hadoop on Demand using Condor
· 2007: CycleCloud Release 1 implements Condor Clusters in the Amazon Cloud
· 2008: Condor adds the ability to schedule machines in Amazon EC2
· 2009: Condor adds support for jobs not coded to use Hadoop to still use data directly out of HDFS using hdfs:// URLs
· 2009: Cycle works with Fortune 100 companies to deploy Hadoop clusters on a Condor pool of resources
2009: Sun supports some cloud functionality in the commercial-only
binary release; non-paying customers should build the source themselves
Clearly, Condor was years ahead of SGE in having these features, and has been doing it in production environments for far longer.
That’s not to say that SGE, including these features, isn’t cool. In fact, it is! It just isn’t the first to do so. The Condor Team deserves attribution (and a correction by Wilson) for enabling cloud scheduling using Amazon EC2, and Hadoop on Demand at Yahoo! itself, long before SGE.