Surviving the IoT Flood

Last month’s blog addressed edge computing and how it supports the Internet of Things.  Now, let’s look a bit further into the storage demands of IoT. IoT storage can’t possibly be covered in a blog, but here are some thoughts.

We’ve all been introduced to an IoT-embellished future. Smart homes, buildings, cities, and cars. Industrial, transportation, environmental, and scientific sensors. Sensors that tell us what’s in the soil, what’s in the air, and what’s in the water. Sensors in our clothes and, eventually, even ourselves. A 2017 white paper by IDC forecast that by 2025, IoT devices worldwide will generate some 40 zettabytes of real-time data ( For those who are counting, that’s 40 billion terabytes. The data onslaught will only intensify.

These data will need to be moved, stored, and shared. And, of course, value must be extracted from them. Otherwise, why waste the time and money to collect them in the first place?

Where will zettabytes of data reside? Where should your IoT data reside? It all depends on the data and how they’re used. Complicating things, there’s a vast diversity of IoT data, ranging from tiny file logs to huge video surveillance files.

Some data needs to be processed immediately for safety or well-being. Think critical avionics or data exchanged between smart cars approaching an intersection. Similarly, healthcare providers must know immediately when a device detects that an at-home patient is suffering from a medical crisis.

Some data need to be processed soon, such as sensors in an industrial device that can indicate an impending failure. Some data can be processed later. An oil exploration firm’s geological data, for example, needs to be analyzed carefully over time. Or a vending machine can send pings to the datacenter just whenever a purchase is made or inventory needs to be restocked. Evaluating data for insights into consumer preferences and behaviors can be done periodically at the datacenter.

Data that is analyzed in real time may not need to be stored, other than in a temporary cache. Other data need to be retained for set periods of time, such as the aforementioned video surveillance files. Some data can be discarded right after analytics. When the value of the data is to reveal outliers, for example, data points that indicate otherwise can be discarded after analysis. Do you really need to store the sensor reading in your refrigerator informing you that you need milk?

The nature and purpose of the data will determine where they are stored and for how long. It would make sense to store video surveillance files in object storage in a cloud, but large data streams can congest pipelines to cloud repositories. This is also true of data streamed across substantial geographical distances to the enterprise datacenter. Consequently, with large and ongoing data streams, it can make sense, as discussed in last month’s blog, to adopt an edge computing strategy and store these data in mini-datacenters located in general proximity to the data’s sources. Processing can then be done locally.

For critical data that demand real-time analytics, edge computing is a practical option. What would be forwarded to the enterprise datacenter are just the results of the analytics. Of course, you still need to determine if these data must be retained after processing.

For critical data collection and transactions, fast solid-state storage is a must. How much latency is tolerable between two cars communicating with other as they approach an intersection?

The point is you may have to address many kinds of data with many kinds of processing and storage needs. Presently, your storage options basically are clouds, edge computing, or your datacenter, where the value and insights from IoT data will probably be ultimately realized. But does your datacenter have the resources to house a steady flow of IoT data? Is your WAN up to the task without impeding business operations? Resorting to clouds is cost-effective for some IoT data, but moving data from many disparate sources to clouds still takes time.

Once you figure out your IoT management and storage needs, you must address security requirements. How important is each IoT data stream and what is a commensurate level of security?

Finally, you can start pondering what IoT data to backup and how. Once you figure all of this out, go home and spend some time with your family.

As you do so, carry this thought. The IoT certainly presents a host of challenges, but it is also a revolution in the making that offers unprecedented opportunities for prosperity and well-being.