Student Technical Paper
Technical Paper
Evaluation of Docker Containers for Scientific Workloads in the Cloud
Event Type
Student Technical Paper
Technical Paper
Facilitation Tags
HPC Facilitation
Technical Paper Tags
Technical Paper
Student Technical Paper
TimeTuesday, July 2411:15am - 11:30am
DescriptionThe HPC community is actively considering emerging cloud-based environments to run scientific applications. Containers are at the epicenter of these efforts. Docker containers are the leading platform of choice for the industry because it offers flexibility, portability of applications, and reproducibility. Singularity is another solution that is drawing interest in the HPC community since it targets scientific applications. However, the adoption of containers within the HPC community is currently limited due to the lack of performance analysis and benchmarking of containerized applications as well as lack of standard methods of running multi-node HPC applications in the containers. Our research focuses on (1) performance evaluation of Docker and Singularity on modern architectures (2) mapping elements of parallel workloads to the containers, and (3) optimal resource management with container-ready orchestration tools on academic clouds such as Jetstream and Chameleon.

Performance analyses show that scientific workloads for both Docker and Singularity based containers can achieve almost native performance. The advantage of Docker, over Singularity, is that it provides overlay networking, an intuitive way to run MPI applications with one container per rank for fine-grained resources allocation. However, like Singularity, it is possible to directly use the underlying network fabric from the containers for coarse-grained resource allocation. We present a policy-driven approach for optimal resource management on the cloud for parallel and distributed containerized jobs. We hope that our findings will help application developers make informed decisions on choosing the container technologies that are suitable for their HPC workloads on cloud infrastructure.