Hollywood Has a Velocity Problem
Written by Erik Weaver
Published on April 7, 2020
Hollywood has a “velocity” problem. Let’s talk about why Key-Value Stores are critical.
What do Amazon Web Services and Facebook have in common? Of course, they are both fantastically successful, but they also share something else. Technologically, they are both back-ended by Key-Value Stores. So what is a Key-Value Store (KVS)? In technical terms, it’s a distributed hash table. It’s a way for you to store very simple objects or BLOBS (binary large objects) across a cluster of machines. The cluster provides a robust failover system and a way to store a large number of records with extremely low latency.
Benefit of Key-Value Store
The benefit of a Key-Value Store over a block store is that a KVS is much better suited for more substantial and varying data sets. Block stores were built to accommodate the much smaller 4K blocks associated with databases and ERP (enterprise resource planning) applications. Most are built for a data set with the sector size of 4K—which is what DB and ERP applications typically write.
Unstructured data, especially media file data, has relatively huge file sizes compared to the DB/ERP data sets. Key-Value Stores are designed for larger and varying sized data sets. One example, one of these is the Stellus Data Platform, which uses an atomic size of 1MB. It is significantly more efficient to write a 20MB file into 20 x 1MB chunks than hundreds of thousands of 4k chunks. This means that substantially fewer resources are needed to serve application requests with KVS.
KVS and NVMeOF
Key-value SSDs and NVMe Over Fabrics (NVMeOF) were ratified by SNIA in the 1.0 API standard last April.
There are two additional reasons why modern architectures need to move toward a Key-Value Store: scalability and speed. In terms of scalability, Key-Value Stores handle large data sets well and are good at processing a constant stream of read/write operations. This makes it highly scalable. As far as speed, Key-Value Stores use simple operational commands such as get, put, and delete. This makes them highly efficient at processing constant streams of read/write operations.
This combination of benefits brings us into a profoundly more efficient future.
The world of SAN/NAS will shrink over the coming years. Also, the future is moving towards Object and Non-Volatile Memory Express over Fabrics (NVMeOF) for Velocity and the demands of modern content.