Cloud storage: Leveling the playing field
The ideal solution is a cloud storage software package that is 100 percent S3 API compatible. This can then be deployed by enterprises for private clouds or by service providers to provide their own branded cloud storage service. It would allow existing Amazon S3 applications to be simply re-pointed to the new system.
There are many challenges to building this type of solution, but we’ll note three general challenges here. First, this system must provide the extreme reliability and scalability of a cloud storage platform. The prime directive of cloud storage is that the data must never be lost. For scalability, the solution must work for small systems of a few nodes up to large systems of hundreds of nodes across multiple geographical sites.
Second, it’s difficult to build and maintain S3 API compatibility. For example, Openstack Swift offers an S3 compatible layer, but it supports only the most basic operations lacking important and widely used functions like multi-part upload and access control lists. And Amazon S3 engineers are not standing still; they are continuing to develop new API features, forcing any compatible system into a constant catch-up mode.
Third, operations and maintenance of the system must be simple and low-cost. Administrative functions like adding/deleting users, reporting, and billing must be easy to use and reliable. System operations like adding capacity to the system, detecting and replacing dead nodes, backup and restore must all be provided.
The recent development of commercial-grade “NOSQL” databases like Cassandra, MongoDB, and HBase has been a key factor in making the ideal solution possible. These storage solutions are designed from the ground-up to support scalability, high-availability, and high performance. Typically, they offer simple read/write operations only, trading off query complexity for scalability/availability/performance.
For example, SQL-type transactions are generally not supported. For a cloud storage system, these NOSQL databases give the distributed systems foundation to scale from a few nodes to hundreds of nodes while maintaining high-availability and performance. Most importantly, this includes algorithms on how to manage multiple replicas of data across a set of distributed nodes.
However, NOSQL databases are not well-suited for object stores for similar reasons that SQL databases are not well-suited. Objects in object stores are essentially immutable blobs. Once they are written, they don’t need to be changed or have contents indexed or queried against. Storing the blobs in a database is overkill in terms of functionality, even for a NOSQL database. The databases are not designed to efficiently handle large objects. At some crossover point, it becomes more efficient to store the objects simply as files on the file system. This leads us to the ultimate solution: A hybrid approach of using the distributed systems algorithms of NOSQL databases to manage objects stored on a file system (instead of a database).
The hybrid NOSQL and file system approach only addresses the first of the three challenges above. The other two challenges above are also significant, but different in that they are largely not technical challenges, but require good engineering execution.
biography
Gary Ogasawara is the VP Engineering at Gemini Mobile Technologies. He has worked on large scale mail systems for service providers and other high-performance, high-volume software systems. Gemini’s Cloudian™ product is an S3-compatible storage software package.