Maps can be great- Until They’re not
Written by Lynn Orlando
Published on November 20, 2019
Maps can be great.
- Local maps can take you from Point A to Point B when you don’t exactly know where you’re going.
- Foreign maps can show the lay of the land in places you’ve never seen or might never visit.
- Ancient maps can provide insight into long-lost civilizations.
- Fantasy maps can provide geovisualizations of places that have never existed except in the cartographer’s imagination.
There are blogs devoted to maps, clubs for map enthusiasts, and even public radio broadcasts for those who like to listen to people talk about maps. One might caution you against inviting those people to your next party, however. (Just saying….)
Nevertheless, how in the world can well-constructed maps ever cause problems? Well, they can. In the world of data storage, global maps can be great—until they’re not.
Just about every traditional data storage system has a map. Most scale-out systems use global maps to keep track of data’s locality in the system. As data is changed, added, or deleted, the maps need to change so that they stay current and in sync in real time across all nodes. When a system is small or sparsely populated, global maps work pretty well. However, as the size and amount of the data grows, the map grows, and as the size of the cluster grows, the resources required to keep the global map in sync grow.
This growth translates to the start of the trouble with the global map. As your data storage system fills up and you try to scale out the system, taking control of the global map, managing that map, compacting and pruning that map becomes increasingly resource intensive for the system. When the system reaches scale and capacity, significant CPU time and bandwidth are consumed by keeping the global maps in sync across the nodes, known as the “east/west” traffic.
In fact, in enterprise and hyperscale data center systems, map control can be so intensive that the system often spends more of its resources in trying to keep that map in sync rather than in actually answering the application requests and delivering data to the applications, known as north/south traffic, for which the storage systems were was built. Ultimately, all of the resources required to keep those maps in sync are keeping the systems from being able to serve and protect the data more quickly for the respective applications.
Okay, so now that we’ve established that maps can be great—until they’re not—what’s the answer? After all, data needs a clear and known path between its source and its destination for safe placement and easy retrieval.
The good news is that Stellus has the answer. Stellus provides a revolutionary system that is based on a key-value store, which has enabled it to get rid of the map and build a very performant system—all as software.
How did we do this? Stellus has implemented an algorithmic locating system within its Stellus Data Platfom (SDP) that enables it to ask the algorithm where to store the data, so the Stellus Data Platform does not have any maps to manage and keep in sync. The Stellus algorithm computes storage locations and calculates addresses for placement of all objects across all nodes and SSDs. This calculation is done on the SDP Data Management (DMs) and it is performed on all data included in the SDP.
After the address is calculated and the Stellus Data Platform receives a request for the data, the SDP can ask that same algorithm where the data lives. The algorithm can compute the data location easily and efficiently. Most importantly, it can retain the resources required for map management in traditional storage architectures and use them to serve up the data to the applications, which means shorter application run times, greater performance, and faster access to the data.
Maps can be great, but save them for small and sparsely populated storage systems—and the times when you need to find your way to a business, to a friend’s house, or to a great party, one without the people wearing paper hats made from folded maps. (Just saying….)