Maps can be great- Until They’re not

Written by Lynn Orlando

Written by Lynn Orlando

Published on November 20, 2019

Maps can be great.

  • Local maps can take you from Point A to Point B when you don’t exactly know where you’re going.
  • Foreign maps can show the lay of the land in places you’ve never seen or might never visit.
  • Ancient maps can provide insight into long-lost civilizations.
  • Fantasy maps can provide geovisualizations of places that have never existed except in the cartographer’s imagination.

There are blogs devoted to maps, clubs for map enthusiasts, and even public radio broadcasts for those who like to listen to people talk about maps. One might caution you against inviting those people to your next party, however. (Just saying….)

Nevertheless, how in the world can well-constructed maps ever cause problems? Well, they can. In the world of data storage, global maps can be great—until they’re not.

Just about every traditional data storage system has a map. Most scale-out systems use global maps to keep track of data’s locality in the system. As data is changed, added, or deleted, the maps need to change so that they stay current and in sync in real time across all nodes. When a system is small or sparsely populated, global maps work pretty well. However, as the size and amount of the data grows, the map grows, and as the size of the cluster grows, the resources required to keep the global map in sync grow.

This growth translates to the start of the trouble with the global map. As your data storage system fills up and you try to scale out the system, taking control of the global map, managing that map, compacting and pruning that map becomes increasingly resource intensive for the system. When the system reaches scale and capacity, significant CPU time and bandwidth are consumed by keeping the global maps in sync across the nodes, known as the “east/west” traffic.

In fact, in enterprise and hyperscale data center systems, map control can be so intensive that the system often spends more of its resources in trying to keep that map in sync rather than in actually answering the application requests and delivering data to the applications, known as north/south traffic, for which the storage systems were was built. Ultimately, all of the resources required to keep those maps in sync are keeping the systems from being able to serve and protect the data more quickly for the respective applications.

Okay, so now that we’ve established that maps can be great—until they’re not—what’s the answer? After all, data needs a clear and known path between its source and its destination for safe placement and easy retrieval.

The good news is that Stellus has the answer. Stellus provides a revolutionary system that is based on a key-value store, which has enabled it to get rid of the map and build a very performant system—all as software.

How did we do this? Stellus has implemented an algorithmic locating system within its Stellus Data Platfom (SDP) that enables it to ask the algorithm where to store the data, so the Stellus Data Platform does not have any maps to manage and keep in sync. The Stellus algorithm computes storage locations and calculates addresses for placement of all objects across all nodes and SSDs. This calculation is done on the SDP Data Management (DMs) and it is performed on all data included in the SDP.

After the address is calculated and the Stellus Data Platform receives a request for the data, the SDP can ask that same algorithm where the data lives. The algorithm can compute the data location easily and efficiently. Most importantly, it can retain the resources required for map management in traditional storage architectures and use them to serve up the data to the applications, which means shorter application run times, greater performance, and faster access to the data.

Maps can be great, but save them for small and sparsely populated storage systems—and the times when you need to find your way to a business, to a friend’s house, or to a great party, one without the people wearing paper hats made from folded maps. (Just saying….)


Related Post

Optimizing Data Locality for Efficient Data Management

Optimizing Data Locality for Efficient Data Management

Data locality optimizing algorithms work to reduce the time it takes to retrieve data from a network. This post looks at what role they can play in your enterprise data management. Data locality is a basic computing principle that
Post-Production Data Pipeline

Post-Production Data Pipeline

An effective post-production data pipeline is the best way to accelerate a project, protect the data, and maintain top quality. Generally speaking, a data pipeline facilitates the automated flow of data from one point to another. Generally speaking, a data pipeline
What is Software-Defined Storage?

What is Software-Defined Storage?

Software-defined storage, or SDS, is a type of storage architecture that separates storage software from the storage hardware. It manages and unites all storage-area network (SAN) and network-attached storage (NAS) devices in your data center.

Media & Entertainment

The Stellus Data Platform (SDP) sets a new standard for media storage performance, empowering M&E companies to support more workloads, more simultaneous playback streams, and faster render times. Unlike architectures that waste resources on tasks irrelevant to modern storage, the SDP is an entirely new file system, built from the ground up for unstructured data and solid-state media.

Read Solution

Life Science

Stellus is rewriting the rules for life sciences computing. The Stellus Data Platform (SDP) file system is built from the ground up for storing and processing unstructured data. It enables organizations to dramatically accelerate application performance for genomic analysis. Researchers can now process more workloads in far less time and take concrete steps to enable personalized medicine therapies.

Read Solution

Stellus Data Platform

Stellus provides a modern, highly performant file storage platform that enables hyperscale throughput for digital enterprises. Based on key-value store technology and exceptionally suited for unstructured data.

Learn More

Solution Brief- Genomics

Unlock the Targeted Therapies of the Future

Read more

Solution Brief- M&E

Transform Media Economics with Breakthrough Capacity & Performance

Read more

Solution Brief- Cryo-EM

Break the Backlog for High-Speed Microscopy

Read more