Considersations for Financial HPC Applications in the Cloud

Julio Gomez had a great post in WallStreet & Technology about considerations for Building a Cloud Strategy that was created by one of the Innovation Councils including the following:

1. Prepare to educate vendors
2. Liability and indemnification are a major disconnect
3. Nail down your authentication and federated i dentity capability
4. Identify lowest risk, lowest value areas for initial forays
5. Get your physical infrastructure organized

This is a great piece, and the points mentioned by the council are spot on. It is important to consider security and authentication management up front, as well as applications that have low risk or downside from a business perspective. As companies start deciding the types of workloads that make sense to move into the cloud, there are a few steps that can make tremendous sense in this environment.

Based on my experience in this area with high performance computing (HPC), there are a few areas to consider to ensure that applications are suitable for the cloud. Most of them revolve around data and security. When looking at hedging, pricing, trading and other quantitative applications, I would recommend the standard steps for technology adoption:

• Assessing entry points/applications
• Launching a proof-of-concept (POC) with an initial application
• Rolling that application into production
• Planning wider adoption based upon lessons learned from the POC

In assessing potential HPC applications for cloud, I would recommend reviewing criteria including the authentication, risk factors, and regulatory requirements from the council meeting, plus the following:

• Latency requirements - External environments don't lend themselves to low latency at this point in time relative to internal InfiniBand environments
• Data volume and source - If the data for this application is generated externally, then running the calculations might be easier externally as opposed to applications where the data is internal and would have to be migrated
• Security level - HPC applications that don't contain individual customer data are likely easier to migrate from a regulatory perspective
• Burst usage - The cloud can generally best help applications that require lots of compute for limited periods of time as opposed to fixed numbers of internal servers
• Internal costs - In addition to organizing internal infrastructure, it is important to quantify the costs of internal infrastructure including capital equipment, power, real estate, staff, and utilization over time so you can compare with external offerings
• Data to compute ratios - HPC applications that have medium to high computation relative to lower data are well suited for cloud infrastructure
• Internal model software / open source / 3rd party licensing - HPC applications that utilize internally developed software or open source are easier to migrate and simpler to run at large, elastic scales, without having to worry about how many cores you have licensed from a 3rd party. Even using schedulers like Condor/Hadoop can significantly impact costs over Platform Computing and Tibco/Data Synapse
  ... and others ...

Once HPC applications have been assessed using the criteria above, there are generally a few that commonly become suitable for the cloud. These include back-testing of trading models, as the historical market data is generally externally produced and the model tweaking is compute intensive, while not involving individually identifiable customer data. Additionally, these trading models generally are internal software that runs on off the shelf Linux OS's, like Ubuntu/CentOS/OpenSuse, use open source schedulers like Condor/SGE/Hadoop, and therefore don't have licensing entanglements to 3rd parties. As corporations assess applications and plan a POC, besides talking to peers in other industries that have adopted cloud computing in greater numbers, like life sciences, talking with experts in the industry that have tremendous depth in these areas can save tremendous time and avoid potential pitfalls immediately.
 
The Innovation Council definitely outlined the major policy issues, and 2010 will be an exciting year to see how these will be addressed by the market.

Comments

[Myinnervoice]

Jason, see my comment on the original article http://bit.ly/beNb5S. I signed as Anonymous.

My view is to add a sine-qua-non pay-per-use for all services delivered from the data center, to get a cloud.

Miha Ahronovitz
Cloud Consultant
myinnervoice@gmail.com