by Laurence Horrocks-Barlow, Technical Director of OCF
In my previous blog, I talked about some interesting new developments in the High Performance Computing (HPC) market coming to light that will become more prominent in the coming months. Particularly, containerisation, cloud and GPU based workloads are all going to dominate the HPC environment in 2020.
Containerisation and storage developments
There is a big push on the parallel file system BeeGFS, now available on open source which is seeing some extremely positive bandwidth results within HPC compute clusters. There are storage vendors who are now looking at containerising BeeGFS, so it can be included on embedded devices in the storage system to ensure faster deployment and configuration management.
Containerisation for a file system in a virtualised environment is becoming increasingly popular, notably IBM is looking at it for its IBM Spectrum Scale storage solution to ease the deployment of their IBM ESS product.
Containerisation allows you to put your applications or file systems in a ‘wrapper’, so they become very mobile, with the ability to tie them into standard configuration management. By designing components of the cluster as a container in the lab, it allows for faster deployment, ease of management and upgrading on-premise.
A lot of research institutions are using containerisation to containerise their scientific applications and experiments, as it enables a researcher use of the entire HPC environment with all the libraries and applications for an experiment. The researcher can then replicate the experiment multiple times around the cluster (emulating a 100 node job), running their experiment within this containerised environment, with very little dependencies on the host operating system or the administrator’s configuration of the cluster.
Once the experiment is complete, the researcher can archive the container which can then be easily reloaded multiple times on different occasions, making re-configuration much simpler and data retrieval more cost effective.
The ability to restrict the containers and section up the memory, to avoid any memory leaks, is certainly becoming more prominent in recent months. Some providers are starting to limit access to the same system, via a total encryption multi-tenant approach, which secures part of a memory between containers and virtual machines (VMs), so they aren’t able to see each other’s memory maps.
One of the major security aspects of cloud computing and containerisation is the concern that other users or tenants on the system are able to start looking at memory maps and leaking information of research that is confidential for example, medical research using non-anonymised data. Having new security technologies coming onto the market whereby you are limiting the scope of the container or how the VM is able to access the memory goes a long way to reducing that worry.
GPU computing has become more significant with the rise in deep learning, used by artificial intelligence, data mining and data analytics.
NVIDIA’s support for Arm-based HPC systems combined with its CUDA-accelerated computing is giving the HPC community a boost to exascale. ARM‘s ability to produce incredibly low powered CPUs has incredible benefits in an HPC environment.
With many new technology developments and positive uptake of cloud and containerisation, 2020 will herald exciting times for the HPC market.