Compute Cycles

Astounding Science: the CycleCloud BigScience Winners

 

“If we did all the things we are capable of, we would literally astound ourselves.”
  -- Thomas A. Edison

It is with this quote in mind that we get to do something wonderful. Today, we will honor the winner of the CycleCloud BigScience Challenge 2011.

 

What is the CycleCloud BigScience Challenge?

Last October, we defined "utility supercomputing" and challenged researchers to break out of a common habit: limiting the questions they asked to the size of their local compute environment.  Instead we asked them to propose big questions whose answers can move humanity forward. 

We offered the winner use of “utility supercomputing”, providing resources at a scale of the Top 500 supercomputing list, to run their BigScience for a few hours, then turn it off. With Cycle offering $10,000 in time, and Amazon adding another $2,500, together the Winner will have an equivalent of 10 hours on a 30,000-core CycleCloud cluster.

We announced the Finalists at Supercomputing 2011, and had a group of industry luminaries agree to be judges, including Kevin Davies of Bio IT World, Matt Wood of AWS, and Peter Shenkin of Schrodinger. Many thanks to all of you for your help.

 

The Finalists

And then we saw the Finalists' presentations. The experience was inspiring. The finalists all sought to tackle BigScience problems, which only utility supercomputing could help. And it was awesome!

So it is with great pleasure I would like to recognize the Finalists that presented:

• Alan Aspuru-Guzik & Johannes Hachmann, Harvard Clean Energy Project
• Jesus Izaguirre, University of Notre Dame
• Victor Ruotti, Morgridge Institute for Research
• Martin Steinegger, TU Munich ROSTLAB

Next up, we created an Honorable Mention for Alan Aspuru-Guzik & Johannes Hachmann of the Harvard Clean Energy Project, for great research into creating more efficient energy from organic photovoltaic cells. We will give Alan and Johannes the idle compute capacity on Cycle's R&D clusters.

 

And the CycleCloud BigScience Winner is...

Victor Ruotti, a computational biologist at the Morgridge Institute for Research. He and his team will use utility supercomputing to help create an index of RNA alignments that affect the differentiation of human stem cells into specific kinds of cells. The research itself is amazing, and the ability for it to be put into practice quickly in wet labs should have a big impact in the use of stem cells to treat disease.

In closing, we hold high hopes for all of this research, and are excited to report back to you on the results. And equally importantly, we hope that you start to think about how utility supercomputing, whether at 30 or 30,000 cores, that can help ignite your work, make your research more agile, develop products and treatments faster, and enable "all the things that we are capable of". As Edison wisely pointed out, it will literally be astounding.

Jason Stowe
CEO, Cycle Computing

CycleCloud BigScience Challenge Finalists = +5 for humanity

So we announced the finalists of the CycleCloud BigScience Challenge 2011 at Supercomputing 2011 in Seattle last night. Finalists were selected based on their proposal’s long-term benefit to humanity, originality, creativity and suitability to run on CycleCloud clusters.

We're excited to announce that in addition to the $10,000 Grand Prize, and $500 for Finalists, the awesome folks over at Amazon Web Services are adding their own $2500 for the Grand Prize, and $1000 per Finalist, bringing our totals to $12500 for the BigScience award and $1500 per Finalist. With $12500 we will be able to do about 10 hours on a 30000 core environment, or 30 hours on 10000 cores.

The finalists will be judged by me, and a panel of industry luminaries, including Kevin Davies, editor-in-chief, Bio-IT World, Matt Wood, technology evangelist for Amazon Web Services, and Peter S. Shenkin, vice president, Schrödinger. Thanks to Kevin, Matt, and Peter for helping judge the winner. Picking Finalists from all the entrants we got was hard enough. We're going to have our hands full picking a winner from all this great research!

So without further ado, below are the Finalists for the inaugural CycleCloud BigScience Challenge:

Alan Aspuru-Guzik, Harvard Clean Energy Project, professor in department of chemistry and chemical biology and Johannes Hachmann, postdoctoral fellow: Hachmann and Aspuru-Guzik wish to conduct computational screening and design of novel materials for the next generation of organic photovoltaics (OPVs). The goal is to facilitate creating the next generation of photovoltaic cells.

Jesus Izaguirre, University of Notre Dame, associate professor of computer science and engineering and concurrent associate professor of applied and computational mathematics and statistics: Izaguirre intends to explore mutations in proinsulin case misfolding and analyze the ability to stimulate the folding pathways of these mutations to provide mechanic insight into the events of onset of diabetes. He also plans to examine the dominant states in the folding pathways to enable structure-based drug design and the production of new therapies to combat this disease.

Soumya Ray, Harvard Medical School, assistant professor of neurology: Ray’s team has identified a mutation that represents the majority of Parkinson’s disease patients. They seek to utilize the additional computational power to explore the dynamics of the protein and how it interacts with inhibitors to understand how drugs interact with the mutation, benefitting a large number of researcher and other drug discovery programs around the world.

Victor Ruotti, Morgridge Institute for Research, computational biologist: Ruotti aspires to collect genetic information, specifically RNA alignments, from different types of cells to build an RNA-based indexing system for stem cells. Once these alignments are identified, analysis based on this knowledgebase will provide a better understanding of the overarching signaling mechanisms used by stems cells to support generation of personalized, cell-based therapies for a variety of diseases.

Martin Steinegger, TU Munich ROSTLAB, bioinformatics researcher: Steinegger’s team’s goal is to provide access to every possible mutation in the gene sequence that will ever be observed in humans. To achieve, they have started a new project called SNAP-Map, which strives to calculate every possible single-nucleotide polymorphism (SNP) in human proteins to make this technology and data available worldwide. With this data available, researchers will have the ability to access the effect of mutations in individuals and advance the efforts towards individual medicine based on understanding human diversity and variation.

We had many other wonderful submissions, including the runner-up from Purdue's Gerhard Klimeck, of the NanoHUB project. If you're interested in finding out more about the Challenge, and you're at Supercomputing, visit us at SC11 BOOTH 443.

Thanks to all the entrants, and we'll keep you posted on the winner!

Jason Stowe
CEO, Cycle Computing

CycleCloud BigScience Challenge Finalists Announced at SC11 Amazon Booth #6202 on Monday at 8PM

We got a great response to the CycleCloud BigScience Challenge 2011, and wanted to thank all of the entrants. The entries included calculations to fight Alzheimer's, fight Parkinson's, help understand stem cell differentiation, and improve photovoltaics to make greener energy. It's going to be hard to pick Finalists!

We are also excited that the gracious folks at Amazon Web Services are adding to the prize money that Cycle is providing. Amazon's contribution will help us get even more BigScience done in shorter periods of time using utility supercomputing.

We'll announce the Finalists, the final Judges, and more details about the additional prizes, at 8pm Monday, November 14th, at SC11 at the Amazon Booth #6202. Additional information will be available at Cycle Computing's Booth #443 throughout the show and on the BigScience Challenge Website.

Thanks again to the entrants. We're very excited about these scientists using utility supercomputing to perform research that will benefit humanity. We'll see you at SC11!

Mad Scientist could win CycleCloud BigScience Challenge…

Just kidding, he's just a potential finalist! ;)

As some of you may know, Cycle wants to help scientists answer big research questions that might help humanity by donating compute time using our utility supercomputing softare. But in the overwhelming response we've gotten to the CycleCloud BigScience Challenge we announced last week, we repeatedly get the question, "What kind of research benefits humanity?" And the answer isn't Dr. Evil researching "sharks with frickin' laser beams"!

Let's highlight a couple of the entries already received that might move us forward: There is the researcher doing quantum mechanics simulations for materials science to improve solar panel efficiency that might help "electrify 2.5 Billion people" with greener energy. Or the computational biologist that wants to use meta-genomics analysis to create a knowledgebase indexing system for stem cells and their derivatives, helping us "speed development of personalized cell-based therapies". Very exciting!

Maybe you analyze public government data to provide clarity. Or you research science that might help in the race to treat Alzheimer's, Cancer and Diabetes. Or you're simulating ways to more efficiently distribute food in places that need it. There's plenty of utility supercomputing applications ahead of us that could benefit humanity, and now's your chance to start.

Remember entries are due November 7th. So come join us. There's just four questions between you and the equivalent of 8 hours on a 30000 core cluster. So submit early & submit often, and let's change the speed that BigScience gets done!

Jason Stowe
CEO, Cycle Computing

CycleCloud BigScience Challenge 2011

I’m planning on offering the equivalent of 8 hours on a ~30,000-core cluster ($10,000 in free CycleCloud time) to help researchers answer questions that will help humanity. But before we get there, let’s talk about why:

Recently, Cycle got significant press coverage for using CycleCloud to create a ~30,000 core cluster on Amazon EC2 for a Top 5 Pharma to run research. It hit Ars Technica, then Slashdot, then Engadget (which had a great depiction of Nekomata, btw), then Wired's CloudLine ... unreal. (Update: Now Forbes and Wired too!)

In reading all the comments and questions, like "Would this run high-res Crysis?" or "How much capacity does Amazon have?", a thought went through my mind:

I became concerned. Concerned because these shouldn't be the questions we're asking.

I worried that in all this glitter, we would miss what is truly gold: that this type of computing can speed up scientific research and solve problems we’d traditionally never dream of tackling. So I'm writing this to introduce a new concept to you, and implore you to think about how to move the human race forward through science and research.

To start, let's answer a question implied in many comments: Why is this important? For years cloud computing has been about paying for what you use, and accessing the compute power you need, when you needed it.

The problem is, today, researchers are in the long-term habit of sizing their questions to the compute cluster they have, rather than the other way around. This isn’t the way we should work. We should provision compute at the scale the questions need. We're talking about taking questions that require a million hours of computation, and answering them in a day. Securely. At reasonable cost.

I call this “utility supercomputing”, that is, provisioning resources at a scale of the Top 500 supercomputing list, for researchers to use for a few hours, then turn off. For example, take the 5 years of back data that you never were able to re-analyze with a new, innovative algorithm because it would take 2 years on your internal compute cluster, and instead get results back in 9 hours. For $12000.  Wow...

Scratch the surface of this idea, and you'll see a world of research the way I see it. No more waiting. No more R&D folks task-switching for days or weeks while compute is run. Only answers at the speed of thought, at the speed of invention, at the scale of the question.

But unlike other innovations, this is not a tool for the have's and not the have-not’s. This tool is equal opportunity and an extreme competitive advantage. You pay for what you use with no up-front investment. We've already enabled individual researchers to rent computing equivalent to all of the Top 5 Pharma's HPC clusters combined, for a couple thousand dollars an hour. And all this without the researcher needing to know IT, or scripting, or APIs.

Now, on behalf of the Cycle team, I'd like to answer an additional question we’ve been asked repeatedly over the past several weeks: Will you go bigger?

Absolutely. We will. There is no more thinking that having a single, fixed-size internal cluster is the right idea. You might buy internal compute capacity for day-to-day usage. But the answer isn't there. Not anymore.

This is truly exciting. And not exciting simply because technologically Cycle can now create 30000 core clusters in minutes, leveraging robust automation, performance analytics, and moving data securely across continents to answer un-askable questions.

It's exciting because someone is going to take these clusters and cure cancer, or Alzheimer's, or my personal affliction, Type-1 diabetes. And hopefully cure them faster because they have better tools.

So there's no more running. We aim to misbehave. We aim to wreck the status quo of HPC clusters and computational science. We will enable those crazy questions from the misfit geniuses, the ones so big that you would never even ask them, you might even joke about how silly they are, and we will help answer them. Efficiently. Cheaply.

It is with this in mind that I'd like to ask you, what's your question? What is the analysis or simulation that could change your field? What is the work that would bring you, your university, your research organization, your company, or all of humanity to the next level?

Because these are the questions we should be asking.

And to put our money where our mouth is, we're going big or going home by offering the equivalent of 8 hours on a ~30000-core cluster (specifically $10000 of CycleCloud time), for free, along with free support from Cyclers, to help a single researcher with an un-askable question. It's the CycleCloud BigScience Challenge 2011.

If you have one of these questions please fill out the form, with brief entries, describing your career, the work you do, why you believe in it, and how you want to use 10000 or more cores to push us forward. Proposals are due Nov 7th, 2011, and we'll announce the Finalists at Booth #443 at Supercomputing 2011.

We want the runts, the misfits, the crazy ideas that are normally too big to ask, but might, just might, help humanity.

Jason Stowe
CEO, Cycle Computing

New CycleCloud HPC Cluster Is a Triple Threat: 30000 cores, $1279/Hour, & Grill monitoring GUI for Chef

Update: Wow, we've gotten tremendous feedback from this run on Arstechnica, Wired, and others, and man has it been a busy few days. We did have many people ask a quesiton that we wanted to clarify:
Q: How long would the run-time take in-house vs. in-cyclecloud?
A: The clients indicated the workload would never have happened in-house because it would have used everything they had for week(s). The in-cyclecloud run time was 7-8 hours.

=========================

In more ways than one, the Nekomata cluster is three times as impressive as our last public mega-cluster. A few months ago, we released details of the Tanuki cluster, a 10,000-core behemoth launched within AWS with the click of a button. Since then, we have been launching large clusters regularly for a variety of industries. We kept our eye open for a workload large enough to push us to the next level of scale. It didn’t take very long.

We have now launched a cluster 3 times the size of Tanuki, or 30,000 cores, which cost $1279/hour to operate for a Top 5 Pharma. It performed genuine scientific work -- in this case molecular modeling -- and a ton of it. The complexity of this environment did not necessarily scale linearly with the cores.

In fact, we had to implement a triad of features within CycleCloud to make it a reality:
1) MultiRegion support: To achieve the mind boggling core count of this cluster, we launched in three distinct AWS regions simultaneously, including Europe.

2) Massive Spot instance support: This was a requirement given the potential savings at this scale by going through the spot market. Besides, our scheduling environment and the workload had no issues with the possibility of early termination and rescheduling.

3) Massive CycleServer monitoring & Grill GUI app for Chef monitoring: There is no way that any mere human could keep track of all of the moving parts on a cluster of this scale. At Cycle, we’ve always been fans of extreme IT automation, but we needed to take this to the next level in order to monitor and manage every instance, volume, daemon, job, and so on in order for Nekomata to be an efficient 30,000 core tool instead of a big shiny on-demand paperweight. Our new Grill monitoring GUI app for CycleServer helped show what's cooking with our Opscode Chef environment that helped with cloud infrastructure automation for all the instances for this cluster.

The Stats

Before we step through these enhancements one by one, let’s take a moment to sit back and contemplate the sheer scale of this compute environment.

Resource	Count
c1.xlarge instances	3,809
cores	30,472
RAM	26.7-TB
AWS Regions	3    ( us-east, us-west, eu-west )

 

• Where Tanuki certainly encompassed a significant number of physical racks of compute hardware in the AWS datacenter, this scale meant that our footprint was a significant number of aisles. Certainly, the power, cooling, and floorspace that we occupied during this run were larger than what one would find at many moderately sized datacenters.
• Our system placed individual staged requests to AWS that eclipsed the size of entire clusters that we considered to be large at the time (such as Oni). These requests were made relentlessly to Amazon and were fulfilled with impressive efficiency.
• The amount of RAM available on the compute nodes was roughly 30TB, which means that it could hold the entirety of the current raw Wikipedia database 5 times over in memory (citation needed -- get it?)

The workload that was run for the client in this case was a software package that consisted of  molecular dynamics approaches applied to molecular structure and functioning that were run against millions of compound targets. The expected run time on the internal cluster environment was expected to be at least a week worth of run time consuming the entire cluster.

To manage the high throughput workload of the jobs, we used the condor scheduler for load distribution and CycleServer to monitor progress of workload and job scheduling. The table below provides a synopsis of the total work that was done in this massive scale analysis.

Metric	Count
Compute Hours of Work	95,078 hours
Compute Days of Work	3,961 days
Compute Years of Work	10.9 years
Number of condor jobs	154,116
Avg. Run Time per job	37 minutes

New Multi-Region Support

Because of the scale of this run, we opted to go across multiple regions to spread out the spot instance requests and increase our probability of getting the core counts we required. Drilling down a little deeper into the numbers shown in the table above, here is what the breakout across the three regions looked like. We were very pleased that provision times were very consistent across the regions.

Region	Server Count	Core Count	RAM
US-East	2,825	22,600	19.8-TB
US-West	484	3,872	3.4-TB
EU-West	500	4,000	3.5-TB

Massive Spot Instance Support

Nekomata also marks our first use of Spot Instances. The Spot Market provides a huge discount over traditional on-demand instances which means we can do more science for less money. We’ve added support for Spot Instance bidding for CycleCloud execute hosts. Nekomata made use of over 3700 c1.xlarge Spot Instances at an average cost of 0.286 USD / instance / hour (0.036 USD / core / hour). Compare that to the 0.68 USD / instance / hour for the same On Demand instance. That’s 57% savings!

Massive Telemetry and Monitoring: CycleServer & Grill

We used Nekomata to give a workout to some of the new tools we’ve been working on that provide unprecedented views into large-scale computing environments. We’re extremely excited about this, as this is the first time that we are leveraging the plugin architecture of CycleServer to enable both the collection and visualization of diverse types of data. One such tool is a new offering that we have named Grill for CycleServer.

Cycle has professed its love for Opscode’s Chef project in the past on our blog, and we continue to use it to provision even our largest clusters--Nekomata is no different. When you have hundreds of nodes converging each minute, you need a way to see precisely which converges are failing so that you can resolve the issue(s).

Using the Handler feature of open-source Chef, we wrote a Ruby gem that reports converge information to our flagship telemetry engine, CycleServer. Grill shows the result of every converge, as you can see in the screenshot below.

For Nekomata, we built a Grill view that shows a timeline of all converges and a window showing drill-down to individual hosts and their converge history. Important information such as converge duration, number of resources updated and an exception stack trace on failure are all just a click away. On the right side of the display is a stacked histogram of converges per minute with successful in green and failed in red. We have many more features planned for our upcoming release of the chef dashboards.

Along with the chef dashboards, we have also extended our telemetry engine and visualization dashboards to include  heat maps to succinctly highlight servers that are running host as well as including the data captured by ganglia to see resource utilization trends over time as illustrated with a point-in-time screenshot below from the Nekomata run.

 

We’ll be releasing more information in the coming weeks about these tools, so please stay tuned.

Next time around...

While all parties involved are absolutely thrilled at the achievement of the Nekomata cluster, we would be remiss if we did not share a couple of lessons learned during such a large run. You can be sure that when you run at massive scale, you are bound to run into some unexpected gotchas. As a result, it’s critical that you always execute a post mortem after such an event to identify what went well, what didn’t go well and how to do it better next time.

In our case, one of the gotchas included such things as running out of file descriptors on the license server. In hindsight, we should have anticipated this would be an issue, but we didn’t find that in our pre-launch testing, because we didn’t test at full scale ( another lesson learned....keep reading ). We were able to quickly recover from this bump and keep moving along with the workload with minimal impact. The license server was able to keep up very nicely with this workload once we increased the number of file descriptors.

Another bump that we hit were EBS volume and byte limits on EBS-backed volumes as outlined in their documentation here: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?Concepts_BootFromEBS.html. Once again, once identified, necessary limits were raised and we kept moving right a long.

The Future?

What’s next from CycleComputing’s large scale clusters? Well, we already have our next use-case identified and will be turning up the scale a bit more with the next run. While we are excited about the ability to scale clusters to larger and larger core counts and forging new frontiers in cloud computing, we are most excited about the possibilities we are creating by enabling scientists to spin up such large clusters with minimal effort to answer questions that they would never have considered asking a year ago because they didn’t have access to resources that could answer those questions. Today, we have enabled them to have access to massive computing scales in minutes which in turn gives them the ability and opportunity to ask these sorts of complex questions, not once or twice, but multiple times with multiple different perspectives and multiple different variables leading to improvements in their discovery research efforts. We understand that ultimately, it’s not about core counts or TB of RAM or PB of data. Rather, it’s about how we are helping to transform how science is done.

If you’re interested in talking about high volumes of calculations of compute with Cycle Computing, or using CycleCloud, CycleServer, and Grill, then please contact us.

Fast and Cheap, pick two: Real data for Multi-threaded S3 Transfers

Gentleman start your uploads! They're free now but how fast can we do them? Lately we’ve been working with clients solving big scientific problems with Big Data (Next Generation Sequencing analysis is one example) so we’ve been working hard to transfer large files into and out of the cloud as efficiently as possible. We’re optimizing two costs here: money and time.

Lucky for us, Amazon Web Services continues to drive down the costs of data transfer. We were excited to see that all data transfer into AWS will be free as of July 1st! They’re also reducing the cost to transfer data out of AWS. Less money, more science, yes!

We still need to optimize for time, however. The scalability of the Elastic Compute Cloud (EC2) means we can throw as many cores at a scientific problem as we can afford in a very short time. But what if our input or result data is so large that the time to transfer it far outweighs the time to analyze it?

Our previous work has shown that file transfers often do not fill the pipe to capacity, and are often limited by disk I/O and other factors. Therefore, we can speed transfers by using multiple threads to fill the pipe.

 

As shown above, this work involved moving data directly to a file system using rsync. But since that time, we’ve begun to rely upon the Simple Storage Service (S3) as both a staging area and long-term storage solution for input and result data. S3’s availability and scalability are far superior than even striped Elastic Block Store volumes running on an instance. Besides, using S3 over an instance serving an EBS file system means we have a smaller support footprint. So how can we move data quickly using multiple streams into and out of S3?

Multi-stream Uploads

S3 supports multiple stream data transfers differently depending on the direction of the transfer. If you move data into S3, you can make use of the multi-part upload feature. This enables you to break up a large file into chunks and upload them separately. When you’ve uploaded each chunk, you issue another call to S3 to stitch the chunks together. So we can take a large file, chunk it up, and transfer the chunks simultaneously over many threads to fill our upload pipe.

Multi-stream Downloads

When downloading S3 objects, you can make use of the ranged GET feature of S3. The Hyper Text Transfer Protocol (HTTP) defines a Range header you can add to GET requests. This feature allows you to specify a byte range and only transfer that part of the resource. Because S3 supports the Range header, we can create a list of chunks - specified by byte ranges - and download them simultaneously. After all chunks are downloaded, we can concatenate them and checksum the resulting file to verify integrity.

Test Environment and Results

We tested a new transfer tool by repeatedly uploading and downloading a 2.2 GB NGS file using different numbers of threads. Tests were run on an m1.large EC2 instance. The time to upload the entire file and the time to download it from S3 (including time to reassemble and verify the checksum) were recorded and averaged. As you can see in this first graph, the total bandwidth used in both directions increased with an increasing number of threads.

By using many threads, the time to transfer decreased dramatically. This graph shows the percent increase by number of threads. In the best case, we were able to decrease the time to upload the file to S3 by nearly 2.5 times!

Multi-stream Challenges

Even with a multi-part transfer tool, you still run into the following issues:

• You need to track the transfers and retry any that fail
• You can be too efficient with your transfers and overload your data pipe
• Some Internet proxies drop the Range header from requests silently, resulting in many copies of the entire S3 object to be downloaded
• The time to concatenate downloaded chunks can outweigh some of the transfer gains
• Many S3 tools assume the etag header of the object contains the MD5 checksum of that object (this is not the case with mutipart uploaded objects!) causing integrity checks to fail
  
  

Coming soon: A JetS3t app update

We’ll be addressing these challenges in the coming weeks. Unfortunately, the most commonly used S3 transfer tools have limited support for parallelization. We are big fans of s3cmd, but it doesn’t support multiple threads, and it lacks support for multipart transfers. JetS3t is another option. It’s Synchronize application has support for multiple threads, but it’s focused on moving lots of smaller files in parallel rather than one big file in chunks. Because the JetS3t library has support for multi-threaded transfers, and because it’s open source, we used it as the basis for our own multipart data transfer tool. It’s this tool that generated the results you see above.

We’ll be releasing an application to do this automatically in the next week or so, to enable anyone to easily do parallel uploads.

Why Baking Your Cluster AMI Limits the Menu: DevOps for HPC clusters

You may have read our last blog post about Tanuki, the 10000-core HPC supercomputer we built to predict protein-protein interactions. We’re back to tell you a little bit about how we provisioned the 1250 c1.xlarge instances that made up Tanuki. In fact, it’s the same technology that builds all of our CycleCloud instances whether you select a single-node stand-alone Grid Engine cluster or a super kaiju Condor cluster like Tanuki. But before we get into how we do things today, lets talk about where we’ve been and what we’ve learned.

 

Pre-Built Custom Images: Basic Cloud Cluster “Hello World”

It seems every one's first foray into building HPC clusters in a public virtual cloud (like Amazon’s EC2) involves baking a specialized image (AMI) complete with all the tools required to handle the workload. The most basic architecture includes software for pulling work from a queue or central storage location (e.g. Amazon’s SQS or S3), running the work, and pushing the results back. If you’re feeling especially clever, you may even use a scheduler like Condor, SGE, or Torque.

This first cluster comes up fast, but like all first attempts, probably has some bugs in it. Maybe you need to fix libraries to support your application, add an encrypted file system, or tweak your scheduler configuration. Whatever the case, at some point you’ll need to make changes to it. If you’ve got just one cluster with a handful of nodes, making these changes manually can be done but it’s a pain. Alternatively, you can make your changes, bake new images and restart your cluster with the new images. This is a slow development cycle. It’s painful, but it can be made less so with automated build and test scripts.

With a little time and some perseverance you can get yourself to a point where you’ve got a couple of clusters up and they are working hard for you. Maybe you’ve got a few different image types to handle different types of workloads with different scheduler configurations for each cluster. Maybe you’ve even created images for different operating systems. But then there’s a critical OS security update, or you realize you need to change encryption keys on all your nodes. What do you do?

Figure 1: Explosion of AMI versions

Baked AMIs create problems for everything but the most trivial of use cases: How long will it take you to make new versions of your images and replace the old with the new? How do you keep track of what image versions are affected? Wouldn’t it be easier if we could just change all these instances on the fly?

Absolutely! And you can make that happen with automated configuration management.

 

Base Images Configured with Chef

Having experienced and learned from many of the nuances of building AMIs, configuration management and scaling clusters over the years, we have chosen to use Chef, the open-source project from Opscode, to configure our CycleCloud cluster images. We maintain a small number of base images and layer cluster and role-specific software and configuration on top of the base images using Chef cookbooks. We find the ability to describe the configuration of an instance with a simple Ruby domain specific language to be invaluable. Why?

• Cookbooks are easy to write, read and understand
• Cookbooks are version-controlled in our git repository
• Cookbooks can be modified and tested without baking a new image

Chef allows us to make changes to running clusters. We can patch operating systems, change scheduler configuration, grow file systems and more just by changing a few lines of code and waiting for the scheduled Chef client run.

Figure 2: The DevOps update procedure is much more efficient

So the “get coffee” part is cool! But baked AMIs, for anything but trivial usage, wastes your time, users time, and $$$. DevOps configuration methodologies are a key part of good cloud management.

 

Scaling Chef

In a typical datacenter environment, the life cycle of a machine can range from weeks to years.  In general, bringing new hardware online is staggered, allowing a configuration management system such as Chef to breathe. It tends to only do the heavy lifting when performing the initial configuration of a brand new machine.  In the world of cloud computing, the life cycle tends to be from minutes to weeks.  In addition, the launches tend to be stacked right on top of each other, meaning the demands on the Chef infrastructure are immediate and requests come one after another in rapid succession causing immediate and significant load for Chef.

When we started regularly launching clusters that contained hundreds of machines (thousands of cores) instead of tens of machines (hundreds of cores) - our 4000-core cluster Oni is a good example - it became immediately apparent that we needed to work hard to scale our Chef infrastructure.  It was obvious to us that we had to be able to handle the spike of activity as bare bones EC2 instances came online as naked as the day they were born. Once these systems are converged within Chef, the configuration management load is significantly less, both because there are fewer changes that need to be made and because we decrease our polling interval as a result.

We couldn’t publish a blog post without including some technical details, so here’s how we initially scaled our Chef servers to handle the Tanuki load. We took inspiration from one of the only blog posts we could find on the subject: Joshua SS Miller’s Chef Tuning Part 1. The following tips are for CentOS 5.5.

First, start with a larger EC2 instance, one with quite a few cores to handle the additional application server processes. Then, turn off your chef server and edit /etc/sysconfig/chef-server. Change the OPTIONS line to start as many processes as you wish. In this example, we are only starting two.

# Configuration file for the chef-server service

#CONFIG=/etc/chef/server.rb
#PIDFILE=/var/run/chef/server.pid
#LOCKFILE=/var/lock/subsys/chef-server
#LOGFILE=/var/log/chef/server.log
PORT=4005
#ENVIRONMENT=production
#ADAPTER=thin
#CHILDPIDFILES=/var/run/chef/server.%s.pid
#SERVER_USER=chef
#SERVER_GROUP=chef
# Any additional chef-server options.
OPTIONS="-c 2"

Now start chef-server back up and verify that you have the right number of merb worker processes.

-bash-3.2# ps -ef | grep merb
root      7285     1  0 May11 ?        00:00:02 merb : chef-server (api) : spawner (ports 4005)

chef      7287  7285  0 May11 ?        00:11:46 merb : chef-server (api) : worker (port 4005) 

chef      7288  7285  0 May11 ?        00:11:38 merb : chef-server (api) : worker (port 4006) 

root     12859 12830  0 09:39 pts/4    00:00:00 grep merb

You will now proxy chef-server using Apache. Install apache, then create a new configuration file in /etc/httpd/conf.d called chef_server.conf. Add these lines:

Listen 4000

ServerName devchef.cyclecomputing.com
DocumentRoot /usr/lib64/ruby/gems/1.8/gems/chef-server-api-0.9.12/public/

<Proxy balancer://chef_server>
    BalancerMember http://127.0.0.1:4005
    BalancerMember http://127.0.0.1:4006
</Proxy>

LogLevel info
ErrorLog /var/log/httpd/chef_server-error.log
CustomLog /var/log/httpd/chef_server-access.log combined

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://chef_server%{REQUEST_URI} [P,QSA,L]

You will need to add more BalancerMember entries if you set /etc/sysconfig/chef-server to start more than two processes. You may also have to modify the DocumentRoot to point to the location and version of your chef-server-api gem.

Reload apache with /etc/init.d/httpd reload and test with a knife command.

 

What’s Next

In general, we were universally pleased with the performance of CycleCloud as we began to launch these “mega scale” clusters within cloud environments.  However, there are still improvements to be made.  More specifically, we are always looking to maximize the amount of useful compute work that can be done for the compute time that our clients have paid for.  We want to see clusters of arbitrary scale launch as quickly as possible, and we want the configuration of the cluster to proceed quickly as well.  Stay tuned for our architectural solutions to these challenges. Rest assured that Cycle will not stop driving scalability by generational leaps while making the process of launching a cluster easier and quicker than ever.

As always, if you have questions, or comments, or also love big clusters, please contact us.

Single click starts a 10,000-core CycleCloud cluster for $1060/hr

Update: This cluster received great coverage, including Amazon CTO Werner Vogel's kind tweet, customer commentary on this Life Science cloud HPC project, & results from our EC2 HPC Cluster.

Meet our latest CycleCloud cluster type, Tanuki. Created with the push of a button, he weighs in at a hefty 10,000 cores.

Yes, you read that right. 10,000 cores. Tanuki approximates #114 on the last 2010 Top 500 supercomputer list in size, and cost $1060/hr to operate, including all AWS and CycleCloud charges, with no up front costs.

Yes, you read that right. 10,000 cores costs $1060/hr. Here are some statistics on the cluster:

Scientific Need =	80000 Compute Hour
Cluster Scale =	10k cores, 1250 servers
Run-time =	8 hours
User effort to start =	Push a button
Provisioning Time =	First 2000 cores in 15 minutes, All cores in 45 minutes
Upfront investment =	$0
Total Cost (IaaS & CycleCloud) =	$1060/hr

This historic supercomputer, built completely in the cloud, drew its first breath minutes after the push of a button. Tanuki started operations through a completely automated launch using our CycleCloud^SM service. It ran for 8 hours before the job workflow ended and the cluster was shutdown. The 8-hour run-time across 10000 cores yielded a treasure trove of scientific results for one of our large life science clients.

The ability to run a cluster of this size for $1060/hr, including AWS and CycleCloud charges, is mind-boggling, even to those of us that have been in the cloud HPC business for a while. When Tanuki was first mentioned within Cycle, its scale was thrown out partly as a joke and mostly as a gauntlet. We certainly aren’t laughing anymore. In fact, we have multiple clients already poised to run on 10k+ core clusters in the weeks following this blog post. To find out more, you can also contact us or request an account. All CycleCloud clusters benefit from:

• Zero upfront capital costs, you only pay for what you use
• Tera-scale computing available on-demand in minutes
• Engineered, intelligent configurable data encryption
• Current generation compute instances that include NVidia GPU-accelerated machines and 8-core, Nehalem machines
• 10 GBit ethernet network speeds
• Large, secure data transfers in to the cloud at 400GB/day over enterprise WAN connections
• Pre-configured, secure, maintained, ready-for-launch clusters for most HPC schedulers (SGE, OGE, Condor, Torque are the most popular)

So, how did this particular capability come about?

Scientific Need

Computational modeling for life sciences, especially for protein binding and structure prediction calculations, is a useful tool for finding areas of interest that then need to be examined in the lab. The goal of these computations is to find targets of interest that might eventually lead to treatments for disease.

For this use case, the computational work is attempting to simulate the interactions of real proteins and molecules in order to help solve the combinatorial needle-in-a-haystack problem of finding these potential treatments. So needless to say, the faster this discovery process can be completed the better. In this case we'll use 10000 cores to shoot for hours instead of weeks/months for a very large Life Sciences company.

History of HPC Motivation

Basked in the eerie glow of LCD monitors, teams of Cycle engineers have been working on this capability for quite some time. Our goal: to run a client workload at a scale that rivals flagship clusters at supercomputing centers. The path to Tanuki took considerable effort to travel. In order to fully tell the story, we’ll start at the very beginning:

Nearly all of us at Cycle Computing are part of the diverse heritage of HPC. It doesn’t matter what industries we came from: academia, life sciences, financial, entertainment, and then some. Building things bigger is in an HPC person's blood. Increasing utilization, decreasing runtimes and improving scientific and computational output has been our goal long before we became colleagues at Cycle.

The cluster environments where we all first flexed our muscles were bound by typical budgets allocated for internal/private clusters. These types of computing environments are thoughtfully designed and purpose-built. Cycle still manages a significant number of compute grids like this today.

But, fixed sized clusters are too small when you need them most, and too large at every other time. Valleys are wasteful, and peaks are painful. This simple realization is what is driving the majority of Cycle’s business towards cloud environments such as AWS and internal VM deployments.

One of our core tenets is to leverage established scheduling environments with their easy submission interfaces, user priority, and other facilities, coupled with shared file-systems and standard open source toolsets. In a previous blog post, you might have met Okami, a 2048-core Torque cluster built within Amazon EC2 or seen our benchmarks using clusters of GPUs on the CG1 instances. We doubled this with the 4096 core Oni Cluster, a cluster which taught us a lot about the point at which traditional HPC architectures require more sophisticated techniques in order to scale in the cloud. When our EC2 clusters started to surpass the size of some of the larger internal environments that we manage, it was clear that we were onto something. It wasn’t certain whether it was Cycle or a few of our clients that were more excited when we announced that we were aiming at a 10,000-core cluster, a scale previously reserved for supercomputing centers and large enterprise customers.

This is the first in a series of four blog posts covering the gory details of Tanuki.

So, consider this post the overview. Our goal with Tanuki was a massive cluster, launched simply with a single click via our CycleCloud front-end service and easily familiar as a cluster environment, so it could easily accommodate workloads that were already running on internal supercomputing environments. There is absolutely no industry that will not be touched by the implications of this cluster, the first of many mega-clusters that Cycle will enable in 2011.

Launching Tanuki

CycleCloud as a cloud cluster management environment has been around since 2007. In a nutshell, we use it to automate every aspect of launching clusters quickly and easily. As CycleCloud began to grow, we needed a way to provision a wide variety of cluster types and machine types, all with a common base. We knew from benchmarking our client's application which servers were the most cost-effect to run the jobs. And this cluster used a lot of servers:

Cluster Role	Count	AWS Specifications
Execution Machines	1250	c1.xlarge: 8 core, 7GB RAM each
Condor Collector, Condor Negotiator, and CycleServer	1	m1.xlarge: 4 core, 17.1GB RAM
Primary Scheduler and Disk Filer	1	c1.xlarge: 8 core, 7GB RAM
Auxiliary Schedulers	2	m1.large: 2 core, 7.5GB RAM
total	1254	10014 core, 8.6TB RAM, 2PB disk

We quickly became hip to the ways of chef. Still, chef configuration is a single tool, and it can be challenging to scale even an elegant configuration management environment like chef when dealing with hundreds or thousands of concurrent machines launching. We are proud to say that we met this challenge.

In the series of blog posts to come, we will share some of the challenges and secrets of how we were able to achieve some of this scalability. We will provide some cold hard facts about chef converges based on our intensive monitoring of Tanuki. We’ll describe a few failure scenarios within AWS and explain how CycleCloud transparently recovered from them. We’ll also explain how we plan to go even further in scaling clusters larger than ever considered before in a cloud environment.

Scheduling: Condor

Launching 10,000 cores is only a small part of the challenge in a cluster like this. Scheduling actual jobs to the cores can be extremely difficult. We like to use the same HPC scheduler (Condor, GridEngine, Torque) that a researcher might have in house. We pushed against some scalability boundaries with Torque on our 4096-core cluster. For this cluster, we were positive that we needed an environment that was massively scalable and fault-tolerant.

Thankfully, the Condor schedule is astoundingly scalable and extensible, out of the box, and deals well with cloud architectures where nodes come and go. We put our Condor knowledge to use in order to build a fully automated, horizontally scaled scheduling environment that was tuned precisely for maximum responsiveness on a cluster of this size. A future post in this series will describe in detail how we managed to scale condor to handle both the data requirements and the challenge of 10,000 concurrently running jobs.

Monitoring and Performance Analysis

When launching a cluster of this magnitude, it is imperative to closely monitor it. We had extensive telemetry capture enabled during the run which required a robust, scalable tool to both store and visualize the monitoring telemetry that we captured. Not coincidentally, we leveraged Cycle’s flagship product CycleServer to do just that.

The upcoming version of CycleServer (version 4.0) will include two enhancements that made keeping tabs on Tanuki much more easy than it would have been otherwise. For one, the scalability of CycleServer was frankly astonishing; as we threw every bit of scheduler and OS-level data that we could at it. Also, the new plug-in subsystem allowed us to extend the amount of metrics that were being captured and visualized to include practically everything of interest.

The result was a single pane of glass where we were able to monitor jobs, aggregate the performance of every machine in the cluster, and provide real-time readouts of progress as the workload progressed. In another blog post, we will dive into the monitoring capabilities of CycleServer and how it can be used to monitor HPC environments in an environment such as Tanuki or in your own datacenter.

10000-cores could be yours for $1060/hr

Look, we will admit that there was a time when the idea of replacing an internal cluster with a cloud environment was bleeding edge. If Tanuki teaches us anything, it is that clusters in the cloud (even massive clusters in the cloud) have proven time and again to be viable alternatives to running in-house clusters. The scientific results that our clients have gotten from these cloud clusters have proven this over and over. We have now run hundreds of clusters in cloud environments. Many of these clusters are larger than typical academic or enterprise clusters.

As we said before, running a cluster even an order of magnitude larger than that for less than $1060/hr is mind boggling. You can be sure that Cycle is already working on the next generation of mega-elastic cloud clusters. We won’t stop until any individual scientist or organization with a need can spin up a cluster quickly, submit their jobs easily, monitor the workload closely, and cut it all loose when they are done. Whether you need 10 cores or 10s of thousands, we’d like to help harness the HPC compute power you require to succeed with your research. To find out more, please contact us or request an account.

CondorWeek 2011 T-Shirts

Every year it seems like the Condor community will run out of ideas for CondorWeek t-shirts, but, we're happy to say that still hasn't happened! Check out list of t-shirt ideas, and please if you have a new idea, please comment!

Suggestions so far for 2011:
1. I knew I had a problem when I found myself looking at my phone thinking, "I could run Condor on that!"
2. JPMC uses Condor, BearStearns didn't. hmmmm...
3. My Condor pool's so big, my pool has a pool (hdfs), and even my pool's pool is bigger than your pool...
4. Ray. Ray! the next time someone asks if you're running Condor, you say YES!
5. I don't often run 10,000 core clusters, but when I do, I use Condor. Stay compute-thirsty my friends.
6. "Look at your cluster, now back to me, now look at your cluster, now back to me. I'm Condor, the scheduler your cluster could schedule like."
7. That's no moon, that's Purdue's Condor cluster!
8. Condor: Slots that actually pay out!
9. Dr. Condormatch (Or How I Learned To Stop Worrying And Love my Cluster)
10. Condor: "ZKM" WTF?
11. There Can Be Only One condor_master
12. Condor: finding particles, curing disease, and encoding mp3's since 1988.
13. You want the slots? You can't handle the slots!
14. Kirk: Khhaaaaaaaannnnndooooorrrr!

EVERY VOTE COUNTS! If you like one of these over the others, please comment on our blog with your preferences. You don't have to use your name, but please use your e-mail so we can tell if we're using your preference/idea, and get your size!