Compute Cycles

Servers Are Not Houseplants: Servers Are Wheat!

Once upon a time, in the early evolution of computing, it was a time of great opulence, and all was well with the kingdom. This was a time when when every computer server had a title and they were loved and anthropomorphized by the humans. The elder humans would give them mawkish little names and take tender loving care of each and every one of them, talking to them, hugging them, feeding them individually and treating them with great care, understanding the uniqueness of each and every one.

Ah, history! It teaches us so many things! These primitives were not alone, even today the British Royalty have been known to talk to their house plants! Hands up how many of us have named our servers?  We come up with all sorts of names much inspiration has come from Starwars, names of planets, names of beverages and other popular motifs. Our friend Peter Cheslock started a lovely twitter thread, which also included replies from Pinterest. Think about it, Pinterest's entire business is about being social! Even their catchphrase is "A few (million) of your favorite things!", but still these folks don't name their servers, a UUID is just dandy!

So why is this?  Why don't we give cutesy names to our servers any more? Our answer here at cycle is:

Scale has changed everything!

The ubiquity of cloud computing and utility supercomputing clearly changes all of this. The amount of science we need to execute on a daily basis changes this. We don't have time to nourish and look after individual computer servers, or houseplants for that matter! Accordingly:

Servers Are Wheat!

Now we metaphorically plough the fields, we sow the seeds, we reap the wheat, and then we consume it. When we are done, we start all over again. The emotional attachment to wheat is not the same as houseplants. Wheat; yeah it has a job to do, just like our servers! We don't our "hug servers", servers are wheat, why would you want to hug wheat?

So then, we are now all modern day computational farmers! We have had to move from the "hunting and gathering" stage of computing into a brave new era of advanced technologies, we have had to adapt.  Because we adapted (like our ancestors) we can now grow more, with higher capacity and increased yield with considerably less effort.  Our traditional scythes have been replaced with combine harvesters! 

We are clearly out of the stone age :-)  

So, if you are an experienced farmer, why not come join us as we literally tend to fields packed with hundreds of thousands of awesome cereals! 

 

 

HowTo: Save a $million on HPC for a Fortune100 Bank

In any large, modern organization there exists a considerable deployment of desktop-based compute power. Those bland, beige boxes used to piece together slide presentations, surf the web and send out reminders about cake in the lunch room are  turned on at 8am and off at 5pm, left to collect dust after hours. Especially with modern virtual desktop initiatives (VDI), thin clients running Linux are left useless, despite the value they hold from a compute perspective.

Fortune 100 Bank Harvesting Cycles

Today we want to educate you about how big financial services companies use desktops of any type to perform high throughput pricing and risk calculations. The  example we want to leverage is from a Fortune 100 company, let's call them ExampleBank, that runs a constant stream of moderate data and heavy CPU computations on their dedicated grid. As an alternative to dedicated server resources, running jobs on desktops was estimated to save them millions in server equipment, power and other operation costs, and London/UK data center space, thanks to open source software that has no license costs associated with it!

Cycle engineers worked with their desktop management IT team to deploy Condor on thousands of their desktops, all managed by our CycleServer product. Once deployed, Condor falls under control of CycleServer and job execution policies are crafted to allow latent desktop cycles to be used for quantitative finance jobs.

Configuring Condor

Condor is a highly flexible job execution engine that can fit very comfortably into a desktop compute environment, offering up spare cycles to grid jobs when the desktop machine is not being used for its primary role. Our client wanted a policy that would make machines available after hours and on weekends, but only if the machine wasn't performing computational or interactive work for its owner when the execution window opened. Condor tracks mouse and keyboard activity as well as non-Condor initiated CPU load making it possible to craft execution policies that meet complex requirements such as this case.

The first step was to define a policy section that managed the execution windows based on time and day of the week. Two macros were created to simplify the final START expression. The first macro, WEEKDAY_CAN_START, tested to see if the date and time fall on a weekday execution window, the second macro, WEEKEND_CAN_START, tested to see if the date and time fall on a weekend execution window. If either is true we know the machine can run jobs. We combined them in the RUNWINDOW_SCHEDULE_OBEYED macro for simplicity. If this macro is true, the job window is open.

# When, during a weekday, should Condor begin to run jobs and stop

# running jobs?

# 18:00 start time (=1080 minutes in to the day)

WEEKDAY_START_TIME = 1080

# 08:00 end time (=480 minutes in to the day)

WEEKDAY_END_TIME = 480

# When, during the weekend, should Condor begin to run jobs and stop

# running jobs?

# 00:00 start time (=0 minutes in to the day)

WEEKEND_START_TIME = 0

# 23:59 end time (=1439 minutes in to the day)

WEEKEND_END_TIME = 1439

# Boolean expression that returns true if it's a weekday and we're

# in the weekday start window.

WEEKDAY_CAN_START = \

( \

( \

$(WEEKDAY_END_TIME) > $(WEEKDAY_START_TIME) && \

( \

ClockMin >= $(WEEKDAY_START_TIME) && \

ClockMin <= $(WEEKDAY_END_TIME) \

) \

) || ( \

$(WEEKDAY_END_TIME) < $(WEEKDAY_START_TIME) && \

( \

ClockMin >= $(WEEKDAY_START_TIME) || \

ClockMin <= $(WEEKDAY_END_TIME) \

) \

) \

)

# Boolean expression that returns true if it's a weekend and we're

# in the weekend start window.

WEEKEND_CAN_START = \

( \

( \

$(WEEKEND_END_TIME) > $(WEEKEND_START_TIME) && \

( \

ClockMin >= $(WEEKEND_START_TIME) && \

ClockMin <= $(WEEKEND_END_TIME) \

) \

) || ( \

( \

$(WEEKEND_END_TIME) < $(WEEKEND_START_TIME) && \

( \

ClockMin >= $(WEEKEND_START_TIME) || \

ClockMin <= $(WEEKEND_END_TIME) \

) \

) \

) \

)

# This returns true if the schedule for running jobs is in a

# run window. Otherwise false.

RUNWINDOW_SCHEDULE_OBEYED = \

( \

((ClockDay > 0 && ClockDay < 6) && $(WEEKDAY_CAN_START)) || \

((ClockDay == 0 || ClockDay == 6) && $(WEEKEND_CAN_START)) \

)

With time-of-day windows now being computed we turned our attention to monitoring external-to-Condor CPU load and user activity on the console to ensure that the job window only opens up when the machine is really not being used off hours. Configurable macros were set up to determine when external-to-Condor CPU was deemed to be unacceptably high. The parameterized approach used meant the  IT staff at ExampleBank could easily tune the policy. The CPUIdle macro indicated when a machine has become quiet and the CPUBusy macro indicated when a machine had become loaded down with non-Condor work.

# These are the load values that determine if non-Condor

# load is acceptably low enough to run jobs.

BackgroundLoad = 0.3

HighLoad = 0.5

# This is the non-Condor load average

NonCondorLoadAvg = (LoadAvg - CondorLoadAvg)

# These macros return true or false and answer load questions

# about this machine.

CPUIdle = ($(NonCondorLoadAvg) <= $(BackgroundLoad))

CPUBusy = ($(NonCondorLoadAvg) >= $(HighLoad))

A final macro to detect keyboard and console use was put in place. The ConsoleNotBusy macro will be true if the machine has been free of mouse of keyboard activity for at least one hour.

KeyboardBusy = (KeyboardIdle < 60*60)

ConsoleBusy = (ConsoleIdle  < 60*60)

ConsoleNotBusy = ($(KeyboardBusy) == False && $(ConsoleBusy) == False)

All three pieces were combined to define the START expression for the machine, which if true, indicates the machine is available to run jobs:

START = ($(ConsoleNotBusy)) && \

($(CPUIdle)) && \

($(RUNWINDOW_SCHEDULE_OBEYED) =?= True)

The other side of the policy was to determine what happened to jobs when the execution window closed. It can close for any of three reasons:

1. The console may become busy in the form of mouse or keyboard activity;
2. The external-to-Condor CPU load may go beyond the threshold set for acceptably low;
3. The time of day may pass in to a time where jobs are not supposed to execute.

When reason 1 and 2 were encountered, ExampleBank wished to run jobs that suspended for a period of 1 hour before being returned to the queue  to relocate CPU resources to use. For reason 3, it was desired the job be returned to the queue immediately when the time-of-day execution window closed since it known the job would have no chance to run on this machine again for many hours.

Condor provides suspend, resume and vacate settings to tailor the behaviour of an execute node. Cycle engineers modified them as follows to achieve the policy ExampleBank desired:

SUSPEND = ($(CPUBusy)) || ($(ConsoleBusy))

WANT_SUSPEND = SUSPEND

CONTINUE = ($(CPUBusy) =!= True) && ($(ConsoleNotBusy))

PREEMPT = (Activity == "Suspended") && ($(ActivityTimer) > 3600) || \

(($(RUNWINDOW_SCHEDULE_OBEYED) =?= False))

The SUSPEND statement tells Condor to suspend a job if the CPU or console becomes busy. The CONTINUE statement tells Condor to resume running a suspended job if the console or CPU stops being busy. The PREEMPT statement says put the job back in the queue if it's been suspended for more than an hour or if the run window time period has closed.

The Outcome

Today, the desktop pools like this perform financial instrument pricing, as well as software build and test. The following picture taken from CycleServer shows their desktop pool availability over a 24 hour period, with risk management tasks (light blue) running at night and daytime unavailability marked by the machines being in the Owner state (dark grey). The light grey area represents machines unused by staff during the day that do not match to work in their queues.

The dip in available machines that become available just after 7:00 pm occurs as power saving software turns some desktop machines off for the night. A next step in this project is to integrate desktop hibernation and wake-on-LAN features with Condor's power management facilities to turn machines on or off depending on the needs of computation load.

The result of rolling out this harvesting was quite a success, even for only one set of desktops. This initial set of desktops provides an additional 3200 execution slots, with approximately 2400 of slots providing enough memory, disk and CPU horsepower to satisfy the demands of the grid jobs. Operating for a 12 hour period during weekdays and a 60 hour period every weekend(we could do much more here), ExampleBank has gained an additional (((12 hours x 4 days) + (60 hours)) x 52 weeks) x 2400 cores = 13,478,400 CPU hours, or ~1537 CPU years every calendar year!

Benchmarks for the brand new Cluster GPU Instance on Amazon EC2

A Couple More Nails in the Coffin of the Private Compute Cluster

Update: We're getting an overwhelming response to this entry, if you have questions come to booth #4638 at Supercomputing 2010
======
Cycle Computing has been in the business of provisioning large-scale computing environments within clouds such as Amazon EC2 for quite some time. In parallel, we have also built, supported, and integrated internal computing environments for Fortune 100s, universities, government labs, and SMBs with clusters of all shapes and sizes. Through work with clients including JPMorgan Chase, Pfizer, Lockheed Martin, Purdue University, among others, we have developed a keen sense for use cases that are most appropriate for either internal or external computing. More and more we see the lines blurring between internal and cloud case overall performance. This is good news for end users that want to have the flexibility to consume resources both internally and externally.

During the past few years it has been no secret that EC2 has been best cloud provider for massive scale, but loosely connected scientific computing environments. Thankfully, many workflows we have encountered have performed well within the EC2 boundaries. Specifically, those that take advantage of pleasantly parallel, high-throughput computing workflows.

Still, the AWS approach to virtualization and available hardware has made it difficult to run workloads which required high bandwidth or low latency communication within a collection of distinct worker nodes. Many of the AWS machines used CPU technology that, while respectable, was not up to par with the current generation of chip architectures. The result? Certain use cases simply were not a good fit for EC2 and were easily beaten by in-house clusters in benchmarking that we conducted within the course of our research.

All of that changed when Amazon released their Cluster Compute offering. This took the massive flexibility and scale of “classic” EC2 and married it with current-generation HPC sensibilities.

The CPU cores? Blazingly fast Nehalems.

The virtualization technology? Near bare-metal speed HVM architecture.

The interconnect? 10 gigabit with the added bonus of being able to specify “placement groups” to automatically situate your compute farm in one tidy spot.

As supercomputing nerds, we were in heaven. Workloads that previously bottlenecked within “classic” EC2 were now running as fast, or faster, than they were on in-house systems. Now, Amazon has dropped another bombshell on the scientific computing community:

We can now provision Cluster Compute instances with a pair of Nvidia M2050 GPUs.

Yet another bastion of in-house supercomputing, GPU-accelerated programming, and perhaps the last excuse that many in-house users have to continue to maintain a private cluster in their data center, has fallen.

Do you want the ability to rent tens or even hundreds of these GPU-enabled nodes for as long as you want and then toss them back? Want GPU accelerated supercomputing without the floorspace, cooling, electricity and staffing costs? Yes, please!

SHOC and Awe

So, is it really as good as in-house hardware? In a word, yes. But you don’t have to take our word for it.

When we began early access to the GPU-enabled instances our primary concern was that the virtualization technology used in EC2 would interfere with the acceleration. Hampering throughput because of given bus speeds, memory bandwidth, hypervisor interactions and other virtualization bottlenecks. We ran a well known benchmark, SHOC (Scalable HeterOgeneous benChmarking suite), created by Jeremy Meredith at Oak Ridge National Laboratory’s Future Technologies group, both on non-virtualized in-house hardware owned by a well known client and on EC2 Cluster Compute hardware with GPU acceleration. Both types of hardware were using dual Nvidia M2050 GPU units.

So SHOC is our benchmark, but we need an Awe-some HPC cluster to run the tests against several machines. At Cycle, in 2007 we built CycleCloud the first Cluster as a Service on EC2, so now you can create fully-secured clusters on the cloud, with GPUs, in 10 minutes, instead of weeks or months.

As the live video below demonstrates, Cycle has created Condor/Torque/SGE clusters on Amazon EC2 using the new GPU instances, with shared file-systems of arbitrary size, encryption, and images that, through CycleCloud, we patch and maintain so you don’t have to!

Demo of starting GPU Clusters on Amazon EC2:
 

With a GPU Cluster in hand, we can run tests on many machines concurrently, to get stats on the performance of these new nodes! So we gave the SHOC tests and AWEsome CycleCloud GPU Clusters a spin with the following results:

PCIe Bandwidth

Repeated transfer of data of different sizes from hardware (H) to the GPU device (D) and back to measure the overall PCIe bandwidth available between the GPU and the motherboard. Note the capped PCIe speeds in the client hardware are likely due to motherboard and chipset differences.

Max FLOPS

Determines the peak FLOPS by executing a kernel composed of a large number of multiply and multiply-add operations.

Memory Bandwidth

Memory bandwidth measurements for global, constant and shared memory.

FFT

Performance measurements on a 1-D fast Fourier transform and inverse transform. FFT size was 256 MBytes.

MD

Performance measurements for a simple simple nbody pairwise computation. Dataset size was 73728 atoms.

Reduction

Performance measurements for a large sum reduction operation using single precision floating point data. Data size was 64 MBytes.

Scan

Performance measurements for an exclusive parallel prefix sum of single precision floating point data. Data size was 64 MBytes.

SGEMM

Performance measurements for device versions of the SGEMM BLAS routine, on a square matrix. Data set size was 16 kBytes.

Sort

Performance measurements for a radix sort on unsigned integer key-value pairs. Data set size was 96 MBytes.

Stencil 2D

Performance measurements for a 2D, 9-point single precision stencil computation. Reported times include PCIe transfer.

Triad

Performance measurements for a large vector dot product operation. Reported times include PCIe transfer. This is an OpenCL version of the stream triad benchmark.

Real World Application Benchmarking

Based on the positive results of our SHOC benchmarking, we approached a Fortune 500 Life Science and a Finance/Insurance clients who develop and use their own GPU-accelerated software, to run their applications on the GPU-enabled Cluster Compute nodes.

For both applications, the applications perform a large number of Monte Carlo simulations for given set of initial data, all pleasantly parallel.

The results, similar to the SHOC result, were that the EC2 GPU-enabled Cluster Compute nodes performed as well as, or better than, the in-house hardware maintained by our clients.

GPU Clusters in 10 minutes

With this new AWS offering, the line between internal hardware and virtualized, cloud-based hardware for high performance computing using GPUs has indeed been blurred!

CycleCloud provides the lowest cost, fastest entry point to accessing the new GPU-enabled Compute Cluster instances in EC2. Let us help you benchmark your GPU-accelerated HPC workloads in this new environment within EC2. You’re likely to be pleased.

If you’ve wanted to experiment with GPU-accelerated supercomputing here is your chance to access state-of-the-art hardware at prices anyone can afford. Risk-free assessment of GPU-accelerated computing is now a reality. Talk to Cycle Computing about our ultra-fast entry point for using these new machines in EC2. A few hours of research could be worth thousands of hours of saved computation time for your workloads.

Can you tell we’re excited about this?

If you are too, please contact us.

Experience with Data Transfer into EC2

Maximizing data throughput: Multi-stream data transfer into Amazon EC2

It is common for cloud computing articles to talk at length about the abundant hardware resources the cloud can offer the modern researcher or analyst, but little is typically said about the back end data store available with cloud computing. Before any research in the cloud can take place, data must be staged in a manner that is accessible to your cloud-based compute resources. It becomes non-trivial to perform the staging portion of your cloud use if your data sets are large.

The Amazon EC2 cloud provides large quantities of hardware suitable for high speed, high throughput scientific computing. Coupled with the AWS storage and Amazon S3 system it makes a formidable platform for anyone looking to do large scale, scientific computing on quantities of file-based data. In this post we’ll explore data ingress to EC2 as download speeds out of EC2 are typically much higher, both from consumer grade and enterprise level Internet connections, as we typically see more upload than download in the practical use of AWS services. Large data sets get transferred in to EC2, working data stays on cloud-local storage, and summarized, compact results are brought back from the cloud.

Let’s look at a few common cases for moving a large data set into AWS-hosted storage and explore the transfer rates, benefits and drawbacks of each approach.

Case #1: Consumer Grade Internet Connection

It is trivial to saturate the upstream network pipe of a consumer grade cable or DSL Internet connection transferring using only a single stream to EC2. With strong encryption on the ingress stream, CPU demands are insignificant for modern personal computing hardware. Given a 512 kilobit upload speed, it is possible to upload 5.1 gigabytes in one day with no compression.

Case #2: Mid-Range Business-Class Internet Connection

Among our clients we see average business-class Internet connections with six to eight megabits of upload performance. Even with the more capable six megabit upload connection, it is still possible to saturate the line with a single, encrypted rsync into EC2. With no compression, it is possible to transmit 64 gigabytes a day from a mid-range, business-class Internet connection.

Case #3: Enterprise-Level Internet Connection

With our larger clients it is not uncommon to see Internet connections with sustained data-out capabilities of 100 megabits or more. In this class of Internet connection WAN principles such as frame size, packet order and retransmissions start to come in to effect and require consideration to optimize the ingress rate. With little to no optimization, we have seen upper bounds on a single Internet-bound network stream of around 18 megabits. Even without compression or multiple streams, an Internet connection operating at this outbound rate can transfer nearly 200 gigabytes in a 24 hour period. There is, of course, potential for the single-stream transfer to be further optimized but often enterprise network teams have a philosophical problem of a single application leveraging more than 50% of an Internet connection shared by potentially tens of thousands of people.

Special Case #1: Parallel S3 Streaming

If the inbound files are relatively homogeneous and reasonably large, we have seen instances where a parallel upload into S3 can be advantageous. S3 is globally distributed with massively scalable ingress. When file sizes are similar, it is possible to saturate an outgoing network pipe without worrying about one stream of data lagging behind the others because it is larger. It should be noted that once data is in S3, it can be presented via s3backer to a large number of EC2 instances at high speeds using the same principles of S3’s massive scalability.

Special Case #2: Parallel Throttled rsync

There are cases where a single-stream approach out of an enterprise-class connection is insufficient for the data transfer needs. On a massive, shared network connection two undesirable conditions can be observed:

1. A single upload will sometimes fall very short of the available network bandwidth of a particular site due to the network topology; and
2. Multiple uploads will sometimes overwhelm a shared network connection causing complaints from other users and potential business disruption.

We have seen very good results packaging rsync transfers into multiple parallel streams, where each stream is limited to a rigid amount of bandwidth consumption, in situations where sustained throughput needs to be consistent and relative peak-free, but overall throughput should not user more than the network administrators are comfortable with. This approach allows us to push the limits of a shared network connection without saturating it. We often come as close as 90% of the available network bandwidth, meaning that we can perform a significant amount of data transfer into the cloud in a minimal amount of time, with no perceivable disruption to the other users and applications on the network.

Final Thoughts

The EC2 ingress rates are quite capable of handling large data transfers from connections with significant upstream bandwidth available to them. Parallel transfers with capped parallel streams can help take maximum advantage of enterprise-class outbound connections without imparting any serious latency issues on other users within your organization.

If the data is compressible it is possible, using tools such as rsync, to greatly decrease the number of bits transmitted over the wire. In our experiments we were able to increase upload rates by a factor of 3 on heavily compressible ASCII text data used for genetic computations. We have pushed data set uploads of 250 gigabyte to EC2 in 22 hours by using a single encrypted and compressed rsync stream on a 8 megabit upload link.

It is unlikely that the ingress performance into Amazon for even a single EC2 instance will ever come into play unless you have an outbound Internet connection with sustained rates of a gigabit per second or more. Nevertheless, we recommend running at least an m1.large machine for data reception on the EC2 side of the transfer to handle the encryption demands. We have seen inbound transfer rates of encrypted data surpass the capabilities of the basic m1.small instance type.

When performing HTTPS based transfers a potentially severe impact on transfer rates is traversing corporate infrastructure for HTTP proxy and DNS filtering. These HTTP security architectures tend not to be designed with sustained data transfer performance characteristics in mind but rather control over interactive web browsing. If at all possible, avoid these pieces of infrastructure in your data transfer path.

Finally: if your ability to transfer exceeds your available Internet connection, fear not. There is a solution. Simply mail Amazon a USB disk drive or drive array with your data and they will promptly load it into an S3 bucket for you. As amazingly low tech as this approach sounds, for multi-terabyte data transfer it can be surprisingly quick. After all, you should never under-estimate the bandwidth of a truck full of drives...

Make the Most of Your AWS Instances: Using open-source Condor to Harvest Cycles, Part 1

How To - Harvest Cycles From Your App Servers, Part 1

It’s a common problem: you run a successful, cloud-based application business in Amazon’s EC2 cloud with bursty traffic. In order to handle the bursts you have to keep a minimum number of EC2 application servers up and running. Would it not be nice if you could do something with these servers between handling the bursty requests? After all: you’re paying for that time, and there’s thumbnails to generate, analytics to calculate, and batch applications to run.

Enter Condor.

Condor is a high throughput distributed computing environment from the University of Wisconsin, Madison (http://cs.wisc.edu/condor/) that can be configured to steal unused cycles from your application severs when they aren’t serving your main business applications to your customers. Condor provides advanced job scheduling, quota management, policy configuration, support for virtual machine based work loads, integration with all the popular operating systems in use today. And: it’s free.

In the next three posts I’m going to show you how to use Condor to harness the wasted compute power on your application servers and how Cycle Computing’s CycleServer can help make this process simple and manageable.

The Setup

Throughout this series of posts I’m going to talk about a fictitious web application company: AmazingWebVideo Inc. They offer video hosting services and their business has been growing rapidly over the past twelve months. They already run all of their web application components in Amazon’s EC2 cloud, but the nature of their business still requires that they keep a base number of web app servers constantly running to handle the start of any bursts in usage.

They also are currently having to waste the partial hours of compute time that occur when an app server is done serving traffic and could be shut down, or when the machine is far less than 100% utilized. We’d like to capitalize on any idle time a machine has, including periods where it is serving traffic, but not busy enough to top out the machine.

In addition to the application front end they have three other types of work loads that they currently handle on dedicated hardware in the cloud. They use a simple queueing system, perhaps based upon Amazon SQS, and three pools of machines to process each type of job.

render

This type of job performs some CPU-bound work on data from the application database on an as-needed basis. Run time varies from a few seconds to a few minutes. If they have free cycles they want render jobs to run first because timely completion impacts the end customer experience.

 

clean

This type of job performs a small amount of database maintenance. It can be run on an ad hoc basis and wipes away application data that is no longer being referenced and optimizes the database. The more frequently these jobs can be run, the longer the gap between critical maintenance outages.

 

mine

This type of jobs mines application data for internal analysis and planning. It’s very rare that it ever really needs to run, but nice when it can run.

The existing approach to running this work is to have worker instances running each type of job in the cloud. A simple queue for each type of job farms out jobs to these pools of machines and using AWS Auto Scaling can spin instances up or down automatically, if demand ebbs and flows.

Introducing Condor to The System

The first step to bring Condor in to this picture is as a replacement for the in-house simple scheduling system. Condor can provide a single queue for all three types of jobs and a unified view of the pool of cloud machines available to run those jobs. Using group quota technology in Condor it’s a simple matter to set policies on job types and begin making maximum utilization of the pool.

Our run-time policy for the machines in this Condor pool is simple: we always run jobs as long as their are jobs in our queue to run. In Condor configuration speak:

START = True

The machines are setup with no preference towards which type of jobs they will run. Instead jobs are entered in to the Condor queue tagged with a custom group based on their job type. The relevant submit ticket syntax is:

+AccountingGroup = “<job_type>.<owner>”

Group quotas are set at a system level by system administrators:

GROUP_NAMES = render, clean, mine
GROUP_QUOTA_render = 8
GROUP_QUOTA_clean = 3
GROUP_QUOTA_mine = 1

And jobs are entered in to the queue, assigned to groups. In a Condor submit ticket the following assigns a job to group “render” as user “jdoe”:

+AccountingGroup = “render.jdoe”

We can now make optimal use of their background task machines. When render type jobs are not present in the queue more mine and clean jobs will start to fill up the unused processing power. As render jobs reappear in the queue we can configure our policy to either immediately preempt running non-render jobs to make space for them, or to simply not run any more non-render jobs on machines until render jobs have once again consumed their quota of execute slots in the pool.

We know, we promised to show you how to harvest cycles from your idle application servers. But we had to tackle the first step: getting your jobs in to a Condor-managed pool.

By running a Condor pool, we get free access to tremendous features we’d otherwise not have: including resource quotas, priority scheduling, configurable policies, being able to run parallel jobs, fairsharing between users/applications, and detecting the state of the execute nodes for policy purposes. These are things you can't easily get from SQS-based scheduling systems.

Now that we have a Condor pool with dedicated background task processing CPU power in it we can move on to the next step: harvest cycles from our idle application servers.

And that step we’ll deal with in Part 2.

Multiple Condor Schedulers on a Single host

So Cyclers have run multiple schedds per host since 2004/2005 when Jason did a Disney movie with Condor, running 12 schedds per submit server with Condor 6.6/6.7 and using software to load balance jobs between the schedulers. Given the interest in this area, we thought we could help explain how to do it in detail.

Often, when dealing with scheduling jobs at a large scale in Condor, it can sometimes become useful to simultaneously run more than one condor_schedd daemon on the same server. On modern, multi-core architectures, this technique can bring about several improvements: scheduler bottleneck avoidance, improved job startup times, improved condor_q query times, improved job submission times, and enhanced overall throughput.

Today, our guys wrote up both the new school (condor 7.4 or later) and the old school (Condor 7.2 or older) ways of implementing multiple schedulers. Since 2006 CycleServer has done load-based, job distribution between multiple schedulers, so that won't be covered. This post will show how to set up multiple schedulers on a single host, and name the schedds in question. Hope this helps:

http://www.cyclecomputing.com/wiki/index.php?title=Running_Multiple_Condor_Schedds