Linux containers are everywhere, and HPC is no exception.  The last 10+ years have seen accelerating momentum in every space of Information Technology (IT) for this convenient but often misunderstood software paradigm.  Fundamentally, Linux containers offer lightweight application virtualization capabilities by encapsulating all software dependencies for a given solver or algorithm.  This improves portability and eliminates the need to “install” software on systems, greatly reducing operational complexity when dealing with large numbers of applications.  Developers and vendors are free to package their entire stacks, with assurance they will run as intended when finally deployed – regardless of target system.

Containers are often misunderstood because up until recently, they were primarily used to package and deploy very different types of workloads than what the HPC community is accustomed to.  For example, containers are popular in web-service and “micro-service” application stacks.  But the underlying technology has no restrictions on the types of work it can encapsulate, and containerized HPC continues to gain momentum as the market gains a deeper understanding of its capabilities and benefits.

Solving Application Packaging, Distribution, and Deployment Challenges

Decades ago, developers relied on compilers and a runtime library (or two) to build their software.  Increasingly over time, libraries evolved into frameworks composed of a multitude of individual software packages, all having to provide compatibility and integrity to any algorithm leveraging them.  With the advent of “copyleft” licenses starting in the 1980’s and becoming ubiquitous in the 90’s and aught’s, statically binding to specific libraries and components became impossible for most vendors because it would have meant being forced to expose their valuable source code on request to the community.  What in the past may have meant a single binary, modern software applications usually consist of dozens or even hundreds of shared objects and other dependencies, all having to follow a precise manifest for interface compatibility.  Operating system package managers stepped in to help improve the installation of dependencies, but still come up short when multiple software packages must coexist on the same systems with different prerequisites, often resulting in conflicts and what IT types affectionally refer to as “dependency hell”.  Other techniques such as “modules”, popularly used in HPC, have similar limits and create a high maintenance burden for system administrators.  Containers offer a strong alternative to managing and installing complex, often conflicting software stacks.

Convergence of HPC and Other Forms of Computing

Next to increasing application complexity, the convergence of HPC with other types of workloads, such as AI (artificial intelligence), further popularized containers.  Many of these non-traditional HPC codes are “container-native” to begin with, and despite benefitting from HPC system architectures, require different software distribution paradigms.  Additionally, containers allow self-service for end users to manage their own applications, eliminating bottlenecks on HPC system administrators to unlock further innovation.  A properly secured, container-friendly HPC environment can support orders of magnitude more codes and software versions without increasing demand on system administrators and operators.  While it’s true that container runtime environments require some setup to be scalable and secure, once in place, day-to-day use is largely self-service.  Container environment topics such as security and performance are mature, well understood industry topics, with plenty of publicly available patterns and expert advice to draw from.

The Linux Container Ecosystem

The container ecosystem includes runtimes, registries, and orchestration platforms.  Runtime formats range from OCI (Open Container Initiative), popularized by Docker, to technologies utilizing “single image” formats such as Singularity.  Virtually all runtimes are free/open-source software, with commercial support available from various vendors.  Registries offer repositories, both public and private, to securely create, update (or “push”), and download (or “pull”) containers from a centralized location.  Most container registries provide important security mechanisms such as signing and automated vulnerability scanning.  In many cases, the software powering these services can also be deployed and managed privately, giving Enterprise and research organizations a multitude of choices and flexibility in authoritative application hosting.  Finally, orchestration platforms range from highly evolved “infrastructure-as-code” stacks based on technologies such as Kubernetes (also free/open-source), to manual or scripted execution of containers from schedulers for HPC work.  Increasingly, Kubernetes-based technologies are providing strong capabilities for HPC as well, including a vendor-neutral “on-ramp” to public cloud infrastructure which is also becoming more HPC-friendly for appropriate use cases.Containerizing applications simplifies their movement across different types of platforms, and HPC applications are no exception.

Regardless of format, orchestration, or consumption model, containers are revolutionizing HPC codes as they have other types of software, offering a game-changing paradigm shift for end-users and system operators alike.  Organizations are free to construct their own strategy around containers, leveraging community or vendor support from a range of options depending on policies and budgets.  This flexibility and diversity continues to drive adoption of one of the most exciting software trends in HPC we’ve seen in a long time.