The Future of Enterprise Data Storage is
Key-Value over Fabrics
Written by Ken Grohe
Published on March 11, 2020
The Key-Value Store concept has been around awhile, but it has enjoyed a revival as a scalable way to manage large collections of data . Big-name companies such as Facebook, LinkedIn, and Amazon have made extensive use of the paradigm, and for a good reason: when you need to scale quickly without losing performance, it works.
How Key-Value Stores Work
The basic concept is simple. A key, which is a number or string, is used to look up a value, which can be any kind of data. The software converts the key to a hash value, which maps to a storage address. This approach works efficiently, no matter how big the store gets.
A key then maps to a single value, but it could be a list, a JSON object, an XML description, or something else. The value can hold keys to other values, so a value can be as complex as necessary. When applied consistently, it yields a graph database that a query can walk through quickly.
Stores use caching to improve performance. They can cache hashes, values, or both. Hashes are small, and a cache can hold many of them, but caching an entire value provides a more significant saving.
Big data tends to be sparse, and some fields have no value. The key-value store approach handles unstructured data efficiently since there’s no need to store a null pair. Deletion is reasonably efficient.
Use in NoSQL databases
What broadly is called a “NoSQL database” includes several approaches to storage; the key-value store is one of the simplest and most popular. SQL queries degrade in performance as tables grow larger. A Key-Value Store scales better with size, making it a great fit for organizations dealing with massive amounts of unstructured data.
Key-Value Store databases can scale up (i.e., move to bigger storage volumes) or scale out (i.e., add more storage servers). With this in mind, we created the Key-Value Store over Fabric architecture that is the foundation of the Stellus Data Platform , which enables the “scale-through” concept. Stellus’s scale-through architecture allows a database to efficiently scale up and out at the same time, independent of one another. The decoupling of capacity and performance are critical when dealing with increasing amounts of unstructured data. This is especially true when data collections get into the terabyte range and higher because this scalability keeps the operation going without having to rework the architecture.
Examples of key-value services
RocksDB is a key-value engine from Facebook, inspired by LevelDB. It’s intended for databases that are terabytes in size, and it’s optimized for fast, low-latency storage systems.
PalDB is LinkedIn’s opensource key-value engine. It was designed for “side data” or information that doesn’t change often or that specific pieces of code need.
Amazon S3 is a key-value storage service available to AWS customers.
Key-Value Stores are powerful when dealing with large amounts of data, and particularly when dealing with unstructured data (Media & Entertainment is a prime example).
The paradigm is simple, and implementations are highly scalable. Scaling can consist of increasing a single storage device’s capacity (scaling up) or adding additional devices (scaling out) — and better yet, scaling through, where they can scale up and out at the same time independently. The Stellus Data Platform is the only solution available based on Key-Value over Fabric architecture that enables this record-setting throughput.
As companies continue to face exponential data growth, the key-value store approach is steadily gaining ground as the future of enterprise storage.