For HPC temporary scratch storage there are a few prominent technologies (Spectrum Scale previously known as GPFS, Lustre and increasingly BeeGFS) and all offer very good performance in a scratch storage environment. However, for me, Spectrum Scale is still the hands down winner.
Its data management features, which actually stretch to support more than just scratch storage, are extremely valuable. These features include Spectrum Scale’s Policy Engine which enables migration of data between tiers, integration with IBM Spectrum Protect (TSM) enabling efficient back up and HSM to tape. Also, increasingly important for our customers is its Cinder driver for integration into OpenStack.
However, perhaps the biggest challenge to Spectrum Scale is not from another file system, but object storage. Although scalable file systems have made it possible to store huge quantities of data (OCF has customers with 10s of PBs of data), it doesn’t really enable organisations to get value from data. For example, how do they quickly find all documents created by a single researcher, or all images in the file system taken in a particular place? In my opinion this is where object storage, combined with a rich-metadata data management layer, will help people get value from their data rather than just store it.
So what’s the right answer? File systems or object storage? Truth be told [and I appreciate this could be perceived as sitting on the fence] it’s difficult to compare object storage and file systems directly as they are completely different technologies. I don’t believe there is one clear winner in terms of storage technology. Generally, every customer and their requirement is different, even two customers who run the same applications and have similar performance and capacity requirements might have vastly different in-house skills, for example, requiring very different solutions. Although there are many use-cases that would suit object storage, there are just as many that won’t.
Let me know your views. Why would you choose Spectrum Scale or another file system over object storage? How are you using these technologies today and what are your experiences?