If you were unable to attend LUG 2019, the Lustre User Group conference hosted by OpenSFS, you missed an insightful presentation given by Vinay Gaonkar, Co-founder and VP of Products at Kmesh.io. Vinay’s presentation detailed the ‘ins and outs’ of Multicloud Lustre-as-a-Service. Fortunately, you can watch the replay of his presentation here.
In the presentation, Vinay covered lots of ground in under 10 minutes, and this post discusses some of what you will see in the video replay.
Why Lustre-as-a-Service is Needed in Today’s Cloud-Based World
Vinay reviewed the three main drivers behind the need to have a high-performance file system in the cloud and, as you likely already know, Lustre is the highest performing of them all. The need for cloud-based Lustre, he explained, is driven by both technological and business factors.
On the business side, many enterprises need to comply with a wide array of new data sovereignty (data localization) regulations that compel organizations to decentralize their data, having country-specific or region-specific data hubs. At the same time, the increasing adoption of multi-cloud strategies and edge applications (think Kubernetes) further push organizations to decentralize their cloud data. In the end, this all adds up to a trend in which the cloud world is moving from heavy use of centralized data lakes to a much more flexible and responsive architecture of many smaller, distributed data ponds.
Vinay then showed the Lustre users how Kmesh developed its dataplane architecture in response to this new world of distributed data ponds and the need to orchestrate data across them.
As part of the service overview, Vinay highlighted the need for a highly performant file system. In essence, once an HPC user recognizes the need to make data accessible and moveable across multiple clouds and locations for HPC applications like EDA, the choice of Lustre as the core of the Kmesh architecture seemed an obvious choice to make. In fact, the entire Kmesh engine is built on Lustre.
Removing the File System and Cloud Management Burdens for HPC Users
Vinay went into detail regarding what it takes to perform HPC in the cloud and demonstrated the steps Kmesh took to make Lustre in the cloud a turnkey, enterprise-ready solution. Lifting up the covers from the Kmesh architecture, Vinay explained how it was made enterprise-ready by creating a unique HA model for Lustre. Kmesh figured out how to make multi-cloud data cloning and sync a reality. This lets HPC users take data created in cloud and bring it wherever they want, such as for hybrid HPC such as ML, data analytics, and EDA. One of the key features that makes it all possible is the use of Lustre-based data nodes spread throughout the customer’s choice of clouds and locations.
Vinay further highlighted the massive effort HPC teams will save when using Kmesh SaaS rather than trying to develop their own Lustre-in-the-cloud solution. Instead of having to figure out how to install Lustre, prepare AWS instances for every single use case, and figure out what storage strategy is best-suited to each different cloud, it all comes turnkey with Kmesh. All the customer has to do is make data orchestration selections within the Kmesh portal. They never need to deal with calculating which underlying technologies or capacities are required. Kmesh handles that in the background.
Vinay also covered the various features of Lustre-based data nodes within Kmesh—there are node options for performance, balance, and costs. So, a performance node special for an AWS EC2 instance can be picked to match the customer’s specific size and performance, along with granular control over how to manage storage, snapshots, tiering, etc. with all the bells and whistles.
DataFlows™ Make Numerous Use Cases Possible
Kmesh DataFlows are, essentially, a data sync technology between two Lustre instances. They can also sync object storage (like NFS) to Lustre instance, or even sync from a traditional Lustre instance to a Kmesh Lustre instance. As a result, Vinay showed, HPC users can use Kmesh for multiple use cases, including real-time sync between Lustre clusters; sync between a data source (like object storage) and a Lustre cluster; and remote data access using the Kmesh intelligent caching.
Vinay packed a lot of insights into his 10-minute LUG presentation, and the Lustre users learned how easily they can now shift their HPC workloads to the cloud. To learn what they learned, you can view Vinay’s presentation here.