Compute Cycles

Community Feedback: CondorWeek 2010 T-Shirts and Mugs

Hello All,
It's that time of year again: picking Condor t-shirt phrases for CondorWeek!

As we've done for the past few years for CondorWeek, we'd like to get suggestions for new phrases from the Condor Community. Below are some of the suggestions from this year.  If you have a new idea, please comment below this blog post (you don't have to use your real name, just give us your e-mail so we can let you know we're using it!).

If we print yours, we'll give you a special surprise at CondorWeek or send it to you if you won't be in Madison this year.

We're also interested in whether you'd prefer a mug to a t-shirt!

Suggestions so far for 2010:
1. Condor: There's a knob for that...

2. Condor: "ZKM" WTF?

3. Green Computing for Grizzlies:
HIBERNATE = (isWinter) && (ExcessFat > 200)

4. ~/2010_Condor_Odyssey $ condor_rm -all
Todd, I'm afraid that's something I cannot allow to happen...

5. JPMC uses Condor, BearStearns didn't. hmmmm...

6. PREEMPT = Spouse =?= "ELIN NORDEGREN"

7. My Condor pool's so big, my pool has a pool (hdfs), and even my pool's pool is bigger than your pool...

8. 2010 A Condor Odyssey: It's not HAL-9000, it's DiaGrid.

Oldies but goodies:
9. I am the condor_master.

10. Condor: an Evil Plot to control the world's computers. Enjoy!

EVERY VOTE COUNTS! If you like one of these over the others, please comment on our blog with your preferences.

ALSO, IF YOU'D RATHER GET A MUG than a T-SHIRT, let us know that as well.

Please comment! We look forward to seeing what everyone comes up with :)

Considersations for Financial HPC Applications in the Cloud

Julio Gomez had a great post in WallStreet & Technology about considerations for Building a Cloud Strategy that was created by one of the Innovation Councils including the following:

1. Prepare to educate vendors
2. Liability and indemnification are a major disconnect
3. Nail down your authentication and federated i dentity capability
4. Identify lowest risk, lowest value areas for initial forays
5. Get your physical infrastructure organized

This is a great piece, and the points mentioned by the council are spot on. It is important to consider security and authentication management up front, as well as applications that have low risk or downside from a business perspective. As companies start deciding the types of workloads that make sense to move into the cloud, there are a few steps that can make tremendous sense in this environment.

Based on my experience in this area with high performance computing (HPC), there are a few areas to consider to ensure that applications are suitable for the cloud. Most of them revolve around data and security. When looking at hedging, pricing, trading and other quantitative applications, I would recommend the standard steps for technology adoption:

• Assessing entry points/applications
• Launching a proof-of-concept (POC) with an initial application
• Rolling that application into production
• Planning wider adoption based upon lessons learned from the POC

In assessing potential HPC applications for cloud, I would recommend reviewing criteria including the authentication, risk factors, and regulatory requirements from the council meeting, plus the following:

• Latency requirements - External environments don't lend themselves to low latency at this point in time relative to internal InfiniBand environments
• Data volume and source - If the data for this application is generated externally, then running the calculations might be easier externally as opposed to applications where the data is internal and would have to be migrated
• Security level - HPC applications that don't contain individual customer data are likely easier to migrate from a regulatory perspective
• Burst usage - The cloud can generally best help applications that require lots of compute for limited periods of time as opposed to fixed numbers of internal servers
• Internal costs - In addition to organizing internal infrastructure, it is important to quantify the costs of internal infrastructure including capital equipment, power, real estate, staff, and utilization over time so you can compare with external offerings
• Data to compute ratios - HPC applications that have medium to high computation relative to lower data are well suited for cloud infrastructure
• Internal model software / open source / 3rd party licensing - HPC applications that utilize internally developed software or open source are easier to migrate and simpler to run at large, elastic scales, without having to worry about how many cores you have licensed from a 3rd party. Even using schedulers like Condor/Hadoop can significantly impact costs over Platform Computing and Tibco/Data Synapse
  ... and others ...

Once HPC applications have been assessed using the criteria above, there are generally a few that commonly become suitable for the cloud. These include back-testing of trading models, as the historical market data is generally externally produced and the model tweaking is compute intensive, while not involving individually identifiable customer data. Additionally, these trading models generally are internal software that runs on off the shelf Linux OS's, like Ubuntu/CentOS/OpenSuse, use open source schedulers like Condor/SGE/Hadoop, and therefore don't have licensing entanglements to 3rd parties. As corporations assess applications and plan a POC, besides talking to peers in other industries that have adopted cloud computing in greater numbers, like life sciences, talking with experts in the industry that have tremendous depth in these areas can save tremendous time and avoid potential pitfalls immediately.
 
The Innovation Council definitely outlined the major policy issues, and 2010 will be an exciting year to see how these will be addressed by the market.

Follow up on Life Science Leader

So a short while ago, I wrote an article on HPC in the Cloud for Life Science Leader, a widely-read monthly publication for life science executives.

We had some nice responses, but today google notified me that David Dooling posted a response to my article that has a few inaccurate, IMHO, points, so while my response is awaiting moderation on his page, I thought I'd post it here (sorry for the typo on my first comment, David):

David,
Good to meet you. I've read your blog before, and take issue with your arguments regarding the cloud. I have more posts on Cloud at my blog http://blog.cyclecomputing.com which, as well as the post below. This is an interesting area, and I'd love to correspond with you more at the e-mail at the bottom:

> No wonder he makes cloud computing sound so attractive. No mention of the
> IT expertise needed to get up and running on the cloud. No mention of the
> software engineering needed to ensure your programs run efficiently on
> the cloud.
You are implying that to get running in the cloud, an end user must worry about the "IT expertise" and "software engineering" needed to get applications up and running. I believe this is a straw-man, an incorrect assertion to begin with.

One of the major benefits of virtualized infrastructure and service oriented architectures is that they are repeatable and decouple the knowledge of building the service from the users consuming it. This means that one person, who creates the virtual machine images or the server code running the service, does need the expertise to get an application running properly in the cloud. But after that engineering is done once, a whole community of end-users of that service can benefit without knowledge of the specifics of getting the application to scale.

For example, does everyone that uses GMail/Yahoo/Hotmail know every line of software code to make it run? Do they know every operational aspect of how to make mail scale to tens of thousands of processors across many data centers?

Definitely not, and the point is they don't have to. The same is true for high performance and high throughput computing. To give examples of free services that don't require end user software engineering or IT expertise to do bioinformatics/proteomics/etc.:

- The NIH Website for BLAST has, for years, been running BLAST as a service so that researchers can use GUIs to run queries on parallel back-end infrastructure (see http://www.ncbi.nlm.nih.gov/genome/seq/BlastGen/BlastGen.cgi?taxid=9606) This requires no complicated knowledge or software engineering for scientists to run BLAST as a Service.

- Tools like ViPDAC have 2-minute tutorial videos to run proteomics on Amazon Web Service.

Lastly, because I recognize these benefits, and think they are tremendously valuable as a way to enable organizations to focus on what they are good at, and outsource others like IT and Software engineering, I translate my passion for this area into a business that I've built over the past 5 years. But that doesn't make the point any less valid.


> It may not be apparent from his article, but a program that
> runs well on one or ten computers does not necessarily run well on
> hundreds of computers. In fact, he implies the exact opposite.

>    For compute clusters as a service, the math is different: Having 40
>    processors work for 100 hours costs the same as having 1,000
>    processors run for 4 hours.
>
> It may cost the same under that scenario, but not everything scales
> linearly. In fact, most things don’t and that less-than-linear scaling
> actually ends up making it cost more to get a shorter turnaround.

This is also a straw-man, and a deceptive one because it does contain a kernel of truth. It is true that all applications don't perform perfectly linearly at infinite scales. So for applications like Genomic Sequencing  (CrossBow), or MPI apps like computational fluid dynamics, where there are various serial pieces to the computation including overhead if nothing else, you don't get 1000x the performance for 1000x the processors.  But there are many other applications where it is possible to achieve near-linear scaling, what Condor's Miron Livny calls "pleasantly parallel" problems: monte carlo molecular dynamics, BLAST searching with thousands of queries, risk analysis, proteomics runs with different analysis settings, etc.  

In fact, regardless of whether the job is linearly scalable, most companies and research institutions don't have 1 cluster to 1 user scenarios. There are multiple users with multiple jobs each. What if you have 10 crossbow users with 10 runs to do on various genomes? Then you can get 100x performance on the *workflow as a whole*.

And this is the problem most life science companies/research organizations face when they have multiple users. If 10 people are all submitting 10 jobs to an internal cluster, generally speaking, work can get done far faster with more resources, because the separate jobs from separate users can run along side each other. Hence, if you're in the common case of having many users on your cluster/grid, then on the cloud the math is different, odds are 25x the processors will run near 25x faster.

This is an interesting area, and I'd love to correspond with you more off-line, my e-mail is js at Cycle Computing. 

Poster & Lightning Talk: BLAST Workflow Performance on EC2 @ Rocky Mountain Bioinformatics

We've done quite a bit of application engineering for running various life science workflows in Amazon EC2, and we're (finally) getting some of our analysis of running BLAST on Condor in EC2 up on the web, so I wanted to share. 

Back in 2008, we did a benchmark of the performance of running BLAST workflows running on Condor on EC2, and gave a Poster & Lightning Talk at Rocky Mountain Bioinformatics .

Results: We were able to get 1.9825x performance for every 2x the cores on a Condor cluster in EC2. One of the more interesting visualizations that one of our talented guys, Ian Alderman, did, was on the reason that jobs run efficiently when done in high throughput environments.

Basically, the chart below shows the run times of the first(orange), second (yellow), third (green), etc. tasks running on various processors in the cluster (the separate rows in the chart), with time running left to right.

As you can see, high throughput computing takes advantage of the fact that different tasks/jobs, in this case from the same user, take different amounts of time. As a result, over time processors balance out their usage to finish comparatively close together in run-time:

We will post more details about how our BLAST pipeline works in an upcoming post, but you can find the poster with more detail here:

Life Science Leader: Cloudy Future for HPC in Life Sciences?

Early this week, Life Science Leader - a widely-read monthly publication for life science executives - posted an article I wrote about "Is the Future Of High-Performance Computing For Life Sciences Cloudy?" It covers a number of different topics including work we've done with Schrödinger on Amazon EC2, and the benefits of using Clusters As a Service on the Cloud. We think that cloud-based HPC/HTC are going to change the way life science researchers get their work done, especially with applications like Schrödinger's Glide.

Update: Since this came out it has also been covered by Matthew Dublin at Genome Web. Matthew has some other great articles, including an overview of acceleration techniques in bioinformatics and an article on Nature Biotech ponders the cloud.

Re: Condor Cloud-Aware Capabilities

We had a bunch of interest in the last post we made with a Correction to Stephen Wilson's blog from Sun indicating the SGE 6.2u5 release, in December 2009, was the first cloud-aware scheduler in enabling scheduling to Amazon EC2 and scheduling of Hadoop jobs.

Miha Ahronovitz posted some questions that I wanted to address specifically. First, Miha, thanks for your response about Condor's capabilities.  By the questions asked and other feedback we've gotten from the first post, it seems like folks may be interested in finding out more about Condor and what it can do, so I highly recommend that you attend CondorWeek that the Condor Team hosts every year or feel free to contact Cycle. CondorWeek will give you a broad spectrum of all the cool areas people use Condor.

As I stated in the last blog, as full disclosure, we started using Condor 6 years ago, and Hadoop/SGE/PBS within the last 2 years. It may sound like this is a Condor-only vantage point, but we are a unique company in that we support these other schedulers, and truly use the proper tool for the client in various scenarios. Given Miha's questions, please allow me to provide answers about Condor's capabilities, and ask about an SGE feature I saw:

"Can you use Condor to transform a private cloud from in a cost center to a profit center? On-demand means billing, billing means making money."

Yes. Condor was first able to manage internal VM-based infrastructure in the 6.9.4 release*, available on Sept. 5, 2007. Usage Tracking for billing/chargeback has been done by the Condor Accountant for 10-20 years to track usage and provide information for the fair-share engine (as Condor was an early advocate of fair-sharing for high throughput computing).  The Accountant, a part of Condor's job match-making daemon, tabulates Accumulated Usage in compute hours per User/AccountingGroup.

Additionally, Cycle's CycleServer management tool for Condor provides detailed analysis of usage for chargeback, and since 2005 has been used by many Fortune 100's, SMBs, government labs, and universities. Condor's Quill* is a system for storing all the actions of a grid in an RDBMS for tracking/billing/chargeback. It was first released in Sept 2005 in Condor 6.7.11, and is the Condor equivalent of SGE's ARCo (started in Dec 2006).

As you can see, Condor has been doing this a while...

"Can Condor manage a private cloud, based on customer's premises, to access Amazon only during peak demand?"
Yes. As mentioned earlier, Condor can perform scheduling of VMs across a pool of machines. It can also move jobs into the Amazon Cloud based upon business logic using the condor_job_router*, which was first released on July 15, 2008, about 1.5 years ago. Additionally, since 2007, we've used simple Condor scripts to look at the job queue, add nodes to a Condor pool from Amazon EC2 when jobs demand more compute, and separate Condor "Hawkeye"* scripts to terminate EC2 nodes when no further work is available to run on them.

"Can you provide billing content so the local organization makes money, not only Amazon.com?"
Yes, as you can see in the first response regarding internal chargeback.

"Can Condor recommend turning on and off power and create pools of on-demand hosts, saving power?"
Condor calls this Green Computing*, which Condor started providing features for in November 2008. Condor shuts down the nodes according to used-defined, fully-configurable policy expressions and re-powers them as needed. Other schedulers have innovated in this area as well, including PBS, SGE, and the like.

With regard to SGE, obviously a lot of cool work is being done, it just wasn't first with the two features described by Sun/Stephen Wilson's blog. I saw mention of an interesting feature in 6.2u5, for "topology-aware" scheduling, and as this is a pretty broad area, I'd love to hear more about how this feature works?

*Important note: All of these features and the ability to use Condor's Hawkeye framework for custom scripting, are available in the free, pre-built binaries for Condor from the University of Wisconsin-Madison. Condor Users don't have to build Condor from source because they want access to specific features without paying; Full-Featured Condor is freely available.

Correction to Stephen Wilson’s Blog: “The World’s First Cloud-Aware Distributed Resource Manager”

As a computing specialist that works in the Condor, Hadoop, and SGE communities, I wanted to post a follow up to Stephen Wilson’s Jan. 14 blog post on SGE, as it is factually incorrect. 

At Cycle Computing, I started working with Condor users five years ago, as well as using Hadoop and SGE with clients in the past two years, in life sciences, insurance, finance, energy and chip design. Condor is very flexible and powerful, but all schedulers have use cases they’re great at. 

 

We’re fans of computation management in general, but we were alarmed by Wilson’s post about Grid Engine being the “World’s First Cloud-Aware Distributed Resource Manager” because it supports machines running in EC2 through a VPN and can schedule Hadoop clusters.  Simply put, it isn’t first. 

 

Perhaps Wilson was misinformed about SGE being first with these features and will correct or update his post upon review of the following information.

 

The Condor Scheduler has had Hadoop Cluster scheduling since 2006, originally by Yahoo! using Condor in its Hadoop on Demand project.  Condor has had Amazon EC2 scheduling since 2008.  In 2007, CycleCloud offered Condor clusters as a service into the Amazon Cloud.  Condor can even be used to do advanced, cost-based scheduling in Amazon EC2, as we discussed here.  These dates are well published. 

 

Condor is a freely available resource manager from the University of Wisconsin-Madison, with production users over the past 20 years including for many Fortune 500s and large companies like JP Morgan, Yahoo!, Fair Isaac, Altera and Hartford Insurance, who have also talked at Condor conferences.

 

For reference, the timeline looks like this:

 

·      2006:  Yahoo!, the major contributor behind Hadoop, used Condor to
schedule Hadoop clusters on generic hardware; see Sameer Paranjpye’s
presentation on Hadoop on Demand using Condor

 

·      2007:  CycleCloud Release 1 implements Condor Clusters in the Amazon Cloud

 

·      2008:  Condor adds the ability to schedule machines in Amazon EC2

 

·      2009:  Condor adds support for jobs not coded to use Hadoop to still use data directly out of HDFS using hdfs:// URLs 

 

·      2009:  Cycle works with Fortune 100 companies to deploy Hadoop clusters on a Condor pool of resources

 

·      2009:  Sun supports some cloud functionality in the commercial-only
binary release; non-paying customers should build the source themselves

 

·      2010:  Wilson announces Grid Engine does Hadoop cluster scheduling and Amazon EC2 utilization as the “World’s First Cloud-Aware Distributed Resource Manager” 

 

Clearly, Condor was years ahead of SGE in having these features, and has been doing it in production environments for far longer. 

That’s not to say that SGE, including these features, isn’t cool.  In fact, it is!  It just isn’t the first to do so. The Condor Team deserves attribution (and a correction by Wilson)  for enabling cloud scheduling using Amazon EC2, and Hadoop on Demand at Yahoo! itself, long before SGE. 

Cycle on Condor, Hadoop and Clouds

In supporting large and small organizations using Condor, Hadoop, SGE/Torque, and the Cloud, Cycle encounters interesting tips, tricks and other notes from the field, that might be useful to all users of High Throughput and High Performance Computing on Desktops, Clusters, and Clouds. So we were thinking it would be great to share what we've discovered, talk about the cool work we're doing in HPC, and get comments from the community as a whole. So here we are. Watch for new posts, and give your comments when you have them...