The dust is just beginning to settle on a busy few months integrating a new cloud HPC system for hundreds of world-leading biomedical researchers working at eMedLab. You might have read about the successful launch on Computing or Computer Weekly.
The system is one of the first in the UK and is technically innovative, hence such great media interest. It uses a combination of technologies from our world-leading partners; such as Red Hat [Enterprise Linux OpenStack Platform], Lenovo, IBM, and Mellanox, enabling researchers to maintain a central store of data whilst ‘spinning up’ virtual HPC clusters bespoke to their needs.
We’ve integrated similar cloud technology using OpenStack Juno at the Cloud Infrastructure for Microbial Bioinformatics (CLIMB) and we integrated a ‘traditional’ HPC server cluster at the Wellcome Trust Centre for Human Genomics last year. We’ve certainly learned a lot along the way about the intersection of HPC and genomics and I can share some tips for other customers considering similar deployments:
- Firstly, there aren’t many people out there who can claim experience of integrating a cloud HPC system – it is the leading edge; that makes accessing skills tricky. To fill the void, customers should look for integrators with their own proven people, skills and expertise. References are essential.
- It’s not essential, but customers should also try to identify a technically gifted project lead too – to help keep the project moving along internally
- Make sure that you or your integrator has good relations with the hardware and software vendors involved. Again, in the case with eMedLab and CLIMB, there were multiple vendors involved that needed to be managed to deliver kit on time and on specification. This process can be simplified when relationships are proven
- From a technical perspective, if customers want to use OpenStack, we would recommend Red Hat simply because the firm can provide the relevant support for bugs and other issues quickly. If customers want to go down the open source route (because often there are cost implications for a vendor supported solution, as well as benefits) we would support that, but proactively we would recommend Red Hat.
- It is also useful to have configuration management tools on the system, to ensure that the system is consistent. At eMedLab we’re using Satellite in conjunction with Puppet which helps us with this.
- Make sure that the integrator you choose is flexible and agile. When you’re working with cloud HPC machines, you’re going to run into problems. Integrators need to get their hands dirty, and engage with the right people to get a project moving again.
If you would like to learn more about HPC cloud and deployment best practice or have questions, feel free to leave a comment below.