Compute Cycles

Make the Most of Your AWS Instances: Using open-source Condor to Harvest Cycles, Part 2

How To - Harvest Cycles From Your AWS App Servers, Part 2

In Part 1 of this series I introduced you to AmazingWebVideo Inc. They’re a successful, Amazon EC2-based, application provider who wants to get more out of their rented processors. Specifically they want to harvest unused compute cycles from various application servers in between bursty, end-user traffic. We introduced them to Condor in Part 1 and helped them move three classes of background processing jobs from a simple queuing system to Condor in preparation for cycle harvesting. Now lets take a look at how Condor, installed on their application severs, can help them accomplish this goal.

In our existing Condor pool, our machines are set to service jobs always. Since the only processing load these machines experience comes directly from running Condor jobs this setup is fine. But our application servers won’t be running under Condor’s control. Condor needs to pay attention to load outside of Condor’s control and only run jobs when this load is suitably low. We’ll use Condor’s START attribute and ClassAd technology to write an expression that controls when these machines should run jobs. But first lets decide how we want the jobs to run on these machines. There is a whole spectrum of choice here and it helps to think about it advance of writing your run-time policies in Condor configuration files.

Policy Time

There are four state changes around which we need to develop policy:

1. “When can Condor run jobs on this machine?”;
2. “When should Condor suspend jobs it may be running?”;
3. “When should Condor resume running suspended jobs?”; and
4. “When should Condor kill a job it may be running?”

The first question, “When can Condor run jobs on this machine?”, is certainly the most interesting to us at this point in time. We’d like to use these machines to do Condor work whenever there’s a lull in primary app server work. Condor tracks the overall load average for the machine and the load average of any processes under Condor’s control. With these two values we can compute the non-Condor load average:

NonCondorLoadAvg = (LoadAvg - CondorLoadAvg)

For the sake of convenience we’ll define two new ClassAd attributes for our machines called CPUBusy and CPUIdle. CPUBusy is true if the CPU of the machine is busy doing non-Condor work and CPUIdle is true if the CPU of the machine is quite low:

CPUBusy = ($(NonCondorLoadAvg) >= 0.8)
CPUIdle = ($(NonCondorLoadAvg) <= 0.3)

We can adjust that 0.8 to suit our specific tastes.

Are we ready to define our START policy now? Lets see:

START = $(CPUBusy) =!= True

That looks pretty good. LoadAvg in Condor is a 1 minute average so you’re reasonably safe from thrashing due to quick dips in your app server work load. The machine will have been calm for some time before the START expression is changed to True.

Suspend or Kill?

With Condor to the choice to suspend or kill a job when the app server load spikes is not mutually exclusive. You can build a flexible policy that suspends first and vacates the job if it can’t be resumed within a defined window of time. Policy can even be built around job classes or attributes, suspending some jobs and vacating others when app server load begins to increase.

For this post we’ll build a policy that just kills the jobs when the non-Condor load begins to go beyond 0.5 again. We want to turn off suspension on the app server machines:

WANT_SUSPEND = False
SUSPEND = False
CONTINUE = True

And we want to preempt the job when the CPU is no longer idle:

PREEMPT = $(CPUBusy)

We can give our job a window of time to respond to a soft kill signal when we enter the preempting state before we actually hard kill the job processes. Lets provide our jobs with a thirty second window to vacate the machine before we move to hard kill them:

StateTimer = (CurrentTime - EnteredCurrentState)
ActivityTimer = (CurrentTime - EnteredCurrentActivity)
WANT_VACATE = True
KILL = $(StateTimer) > 30

We created a macro called StateTimer to make our policy a little more succinct. Once a machine passes from the Claimed state to the Preempting state a job has thirty seconds to complete before a hard kill signal is sent to terminate the process.

We can observe our app server and dedicated server policies in effect through the CycleServer management interface. In the graph below, machines being used by the primary application are not available to run jobs and are represented by a dark grey area in the graph. The available machines show what work they were processing at the time, with render (dark blue), mine(green), and clean(orange) jobs executing.

Now that we have this working, how can we make it so jobs on condor don't run if certain executables are running (like backup.sh)? Can we prioritize render jobs so that thumbnails for important content renders first? Can we not preempt jobs just because the node they run on gets busy for a minute or two?

In Part 3 of this series we’ll explore some more complicated Condor policy setups, including the ability to suspend and resume certain jobs.

Experience with Data Transfer into EC2

Maximizing data throughput: Multi-stream data transfer into Amazon EC2

It is common for cloud computing articles to talk at length about the abundant hardware resources the cloud can offer the modern researcher or analyst, but little is typically said about the back end data store available with cloud computing. Before any research in the cloud can take place, data must be staged in a manner that is accessible to your cloud-based compute resources. It becomes non-trivial to perform the staging portion of your cloud use if your data sets are large.

The Amazon EC2 cloud provides large quantities of hardware suitable for high speed, high throughput scientific computing. Coupled with the AWS storage and Amazon S3 system it makes a formidable platform for anyone looking to do large scale, scientific computing on quantities of file-based data. In this post we’ll explore data ingress to EC2 as download speeds out of EC2 are typically much higher, both from consumer grade and enterprise level Internet connections, as we typically see more upload than download in the practical use of AWS services. Large data sets get transferred in to EC2, working data stays on cloud-local storage, and summarized, compact results are brought back from the cloud.

Let’s look at a few common cases for moving a large data set into AWS-hosted storage and explore the transfer rates, benefits and drawbacks of each approach.

Case #1: Consumer Grade Internet Connection

It is trivial to saturate the upstream network pipe of a consumer grade cable or DSL Internet connection transferring using only a single stream to EC2. With strong encryption on the ingress stream, CPU demands are insignificant for modern personal computing hardware. Given a 512 kilobit upload speed, it is possible to upload 5.1 gigabytes in one day with no compression.

Case #2: Mid-Range Business-Class Internet Connection

Among our clients we see average business-class Internet connections with six to eight megabits of upload performance. Even with the more capable six megabit upload connection, it is still possible to saturate the line with a single, encrypted rsync into EC2. With no compression, it is possible to transmit 64 gigabytes a day from a mid-range, business-class Internet connection.

Case #3: Enterprise-Level Internet Connection

With our larger clients it is not uncommon to see Internet connections with sustained data-out capabilities of 100 megabits or more. In this class of Internet connection WAN principles such as frame size, packet order and retransmissions start to come in to effect and require consideration to optimize the ingress rate. With little to no optimization, we have seen upper bounds on a single Internet-bound network stream of around 18 megabits. Even without compression or multiple streams, an Internet connection operating at this outbound rate can transfer nearly 200 gigabytes in a 24 hour period. There is, of course, potential for the single-stream transfer to be further optimized but often enterprise network teams have a philosophical problem of a single application leveraging more than 50% of an Internet connection shared by potentially tens of thousands of people.

Special Case #1: Parallel S3 Streaming

If the inbound files are relatively homogeneous and reasonably large, we have seen instances where a parallel upload into S3 can be advantageous. S3 is globally distributed with massively scalable ingress. When file sizes are similar, it is possible to saturate an outgoing network pipe without worrying about one stream of data lagging behind the others because it is larger. It should be noted that once data is in S3, it can be presented via s3backer to a large number of EC2 instances at high speeds using the same principles of S3’s massive scalability.

Special Case #2: Parallel Throttled rsync

There are cases where a single-stream approach out of an enterprise-class connection is insufficient for the data transfer needs. On a massive, shared network connection two undesirable conditions can be observed:

1. A single upload will sometimes fall very short of the available network bandwidth of a particular site due to the network topology; and
2. Multiple uploads will sometimes overwhelm a shared network connection causing complaints from other users and potential business disruption.

We have seen very good results packaging rsync transfers into multiple parallel streams, where each stream is limited to a rigid amount of bandwidth consumption, in situations where sustained throughput needs to be consistent and relative peak-free, but overall throughput should not user more than the network administrators are comfortable with. This approach allows us to push the limits of a shared network connection without saturating it. We often come as close as 90% of the available network bandwidth, meaning that we can perform a significant amount of data transfer into the cloud in a minimal amount of time, with no perceivable disruption to the other users and applications on the network.

Final Thoughts

The EC2 ingress rates are quite capable of handling large data transfers from connections with significant upstream bandwidth available to them. Parallel transfers with capped parallel streams can help take maximum advantage of enterprise-class outbound connections without imparting any serious latency issues on other users within your organization.

If the data is compressible it is possible, using tools such as rsync, to greatly decrease the number of bits transmitted over the wire. In our experiments we were able to increase upload rates by a factor of 3 on heavily compressible ASCII text data used for genetic computations. We have pushed data set uploads of 250 gigabyte to EC2 in 22 hours by using a single encrypted and compressed rsync stream on a 8 megabit upload link.

It is unlikely that the ingress performance into Amazon for even a single EC2 instance will ever come into play unless you have an outbound Internet connection with sustained rates of a gigabit per second or more. Nevertheless, we recommend running at least an m1.large machine for data reception on the EC2 side of the transfer to handle the encryption demands. We have seen inbound transfer rates of encrypted data surpass the capabilities of the basic m1.small instance type.

When performing HTTPS based transfers a potentially severe impact on transfer rates is traversing corporate infrastructure for HTTP proxy and DNS filtering. These HTTP security architectures tend not to be designed with sustained data transfer performance characteristics in mind but rather control over interactive web browsing. If at all possible, avoid these pieces of infrastructure in your data transfer path.

Finally: if your ability to transfer exceeds your available Internet connection, fear not. There is a solution. Simply mail Amazon a USB disk drive or drive array with your data and they will promptly load it into an S3 bucket for you. As amazingly low tech as this approach sounds, for multi-terabyte data transfer it can be surprisingly quick. After all, you should never under-estimate the bandwidth of a truck full of drives...

Make the Most of Your AWS Instances: Using open-source Condor to Harvest Cycles, Part 1

How To - Harvest Cycles From Your App Servers, Part 1

It’s a common problem: you run a successful, cloud-based application business in Amazon’s EC2 cloud with bursty traffic. In order to handle the bursts you have to keep a minimum number of EC2 application servers up and running. Would it not be nice if you could do something with these servers between handling the bursty requests? After all: you’re paying for that time, and there’s thumbnails to generate, analytics to calculate, and batch applications to run.

Enter Condor.

Condor is a high throughput distributed computing environment from the University of Wisconsin, Madison (http://cs.wisc.edu/condor/) that can be configured to steal unused cycles from your application severs when they aren’t serving your main business applications to your customers. Condor provides advanced job scheduling, quota management, policy configuration, support for virtual machine based work loads, integration with all the popular operating systems in use today. And: it’s free.

In the next three posts I’m going to show you how to use Condor to harness the wasted compute power on your application servers and how Cycle Computing’s CycleServer can help make this process simple and manageable.

The Setup

Throughout this series of posts I’m going to talk about a fictitious web application company: AmazingWebVideo Inc. They offer video hosting services and their business has been growing rapidly over the past twelve months. They already run all of their web application components in Amazon’s EC2 cloud, but the nature of their business still requires that they keep a base number of web app servers constantly running to handle the start of any bursts in usage.

They also are currently having to waste the partial hours of compute time that occur when an app server is done serving traffic and could be shut down, or when the machine is far less than 100% utilized. We’d like to capitalize on any idle time a machine has, including periods where it is serving traffic, but not busy enough to top out the machine.

In addition to the application front end they have three other types of work loads that they currently handle on dedicated hardware in the cloud. They use a simple queueing system, perhaps based upon Amazon SQS, and three pools of machines to process each type of job.

render

This type of job performs some CPU-bound work on data from the application database on an as-needed basis. Run time varies from a few seconds to a few minutes. If they have free cycles they want render jobs to run first because timely completion impacts the end customer experience.

 

clean

This type of job performs a small amount of database maintenance. It can be run on an ad hoc basis and wipes away application data that is no longer being referenced and optimizes the database. The more frequently these jobs can be run, the longer the gap between critical maintenance outages.

 

mine

This type of jobs mines application data for internal analysis and planning. It’s very rare that it ever really needs to run, but nice when it can run.

The existing approach to running this work is to have worker instances running each type of job in the cloud. A simple queue for each type of job farms out jobs to these pools of machines and using AWS Auto Scaling can spin instances up or down automatically, if demand ebbs and flows.

Introducing Condor to The System

The first step to bring Condor in to this picture is as a replacement for the in-house simple scheduling system. Condor can provide a single queue for all three types of jobs and a unified view of the pool of cloud machines available to run those jobs. Using group quota technology in Condor it’s a simple matter to set policies on job types and begin making maximum utilization of the pool.

Our run-time policy for the machines in this Condor pool is simple: we always run jobs as long as their are jobs in our queue to run. In Condor configuration speak:

START = True

The machines are setup with no preference towards which type of jobs they will run. Instead jobs are entered in to the Condor queue tagged with a custom group based on their job type. The relevant submit ticket syntax is:

+AccountingGroup = “<job_type>.<owner>”

Group quotas are set at a system level by system administrators:

GROUP_NAMES = render, clean, mine
GROUP_QUOTA_render = 8
GROUP_QUOTA_clean = 3
GROUP_QUOTA_mine = 1

And jobs are entered in to the queue, assigned to groups. In a Condor submit ticket the following assigns a job to group “render” as user “jdoe”:

+AccountingGroup = “render.jdoe”

We can now make optimal use of their background task machines. When render type jobs are not present in the queue more mine and clean jobs will start to fill up the unused processing power. As render jobs reappear in the queue we can configure our policy to either immediately preempt running non-render jobs to make space for them, or to simply not run any more non-render jobs on machines until render jobs have once again consumed their quota of execute slots in the pool.

We know, we promised to show you how to harvest cycles from your idle application servers. But we had to tackle the first step: getting your jobs in to a Condor-managed pool.

By running a Condor pool, we get free access to tremendous features we’d otherwise not have: including resource quotas, priority scheduling, configurable policies, being able to run parallel jobs, fairsharing between users/applications, and detecting the state of the execute nodes for policy purposes. These are things you can't easily get from SQS-based scheduling systems.

Now that we have a Condor pool with dedicated background task processing CPU power in it we can move on to the next step: harvest cycles from our idle application servers.

And that step we’ll deal with in Part 2.