Kmesh Presents: Supercomputing in the Cloud by Jeff Kim, CEO


Hi. My name is Jeff Kim. I’m the CEO of Kmesh. I’ve been in the distributed Cloud infrastructure space for over 20 years, first with web, then with video, then mobile, cybersecurity, and now I’m really focused on Big Data and HPC.

About Kmesh

The thesis or vision behind Kmesh is that the past 20 years have been focused on centralized data lakes. All enterprises have put all of their data in a single location, usually in a physical data center. We think that the next 20 years is really going to be about distributed data ponds across Onprem, Multi-cloud and Edge. There are three major forces that are propelling this.

  1. One is the adoption of hybrid and multi-cloud.
  2. The second being edge computing, which is just starting to light up; where pieces of data are going to have to live on the Edge, in the Cloud and Onprem,
  3. and the last piece is more of a geopolitical one called data sovereignty where different regions and countries and even cities around the world are mandating, by law that data for their citizens get stored in country.

Kmesh provides software to transition from centralized data lakes to distributed data ponds.

So what is HPC?

High performance computing, HPC, or super computing is everywhere. What do we use this for in today’s world? Practically every heavy computational mathematical operation, from things like weather mapping, semiconductor chip design, brain scanning, DNA sequencing are probably the most heavy use of supercomputing is in physics, quantum mechanics, to map out the, the stars in the universe or going down to quantum mechanics to understand the fundamental laws of the universe.

Is it possible to have supercomputing in the Cloud?

At Kmesh, we think so. We started out three years ago with the technology called Lustre. When you actually open up a supercomputer, you have Lustre underneath. We ported or took Luster out of the fastest supercomputers on the planet and we made it work in multi-cloud environments. Now you can have it on AWS, on Azure, on IBM etc. You can move data from Onprem up to whatever Cloud you’d like to get supercomputer type functionality out of a Cloud environment.

So what are the challenges of moving HPC into the Cloud?

Recently… actually in fact, yesterday I talked to one of the foremost experts of HPC at AWS, and I posed the question to him: What is the challenge? Why aren’t we moving more HPC workloads into the Cloud? He gave a very crisp and concise answer that I loved. From a technology perspective, there’s nothin that’s blocking this. The Clouds are ready: from a compute, from a network, from a storage perspective, they’re ready.

The biggest obstacle is actually organizational and behavioral changes. Lots of sysadmins like buying boxes, wiring them up, walking through the data center and seeing flashy blinky lights as they go along. What they don’t understand is that you can see the same flashy blinky lights on a Cloud portal. Technology wise, the Cloud is ready to support supercomputing workloads. We’re just at the infancy of this coming online.

At Kmesh we’re facilitating that transition by helping Onprem HPC customers to easily move their data up to whatever Cloud they like, and then use the individual capabilities of the Cloud to perform whatever applications that they’re working on; be it quantum physics, be it mapping the stars, be it modeling brains.

Kmesh recently ran a bunch of different performance benchmarks. We used VDBench and IO500, which are the leading kind of benchmarking tools out of the HPC industry. We ported workloads up to AWS and Azure, and we ran performance benchmarks with plain vanilla services that you can get from the Clouds. Right now without any special tuning, without any special interoperating with the Clouds, we were able to get basically the 20th fastest supercomputing performance out in the Cloud today.

Now we’re just getting started. Once we start interacting with more of the Cloud vendors and getting their special HPC services, we think that we can close this gap. HPC, supercomputing in the Cloud is ready to go… as the industry grows.

Is high performance computing coming to the Cloud?

The answer is YES. The technology is ready. Our benchmarks show it. The vast majority of HPC customers are still in data centers Onprem, but it is inevitable that they will be using a multi-cloud environment.

The Cloud vendors are providing the compute power and the networking. One of the biggest problems is the actual movement of data. Kmesh is providing the software to easily move that data across all of these Clouds. HPC right now is kind of relegated to big government institutions and education facilities. That’s not going to be how it’s going to be in the future. Enterprises with IoT and machine learning and with all of the new problems that are going on will absolutely need to embrace high performance computing. There’ll be a lot more calculations to do, a lot more application workloads to go through and Kmesh is here to be able to support that transition.

WordPress Image Lightbox Plugin