We can all agree on one thing: nobody likes to wait. When we go to the airport, we like our flights to be on time, and when we head to the grocery store, we want our ripe avocados to be in stock.
Businesses don’t like to wait either: time is money. If a store is running low on avocados, the business wants to know as soon as possible, so they can get the restock process moving right away, ideally without missing a sale. Sounds simple enough, right? Just have the manager call the distribution center. Problem solved!
Except, what happens when operations get to scale, and a business is dealing with millions of transactions a day? This becomes a data problem. Traditional data warehouses were built to store data in batches, and report on it after the fact, like if the store manager called the distribution center to tell them about what happened last week. Yet modern businesses need to take action on new information as soon as it arrives, and they’re increasingly finding themselves blocked by unacceptably stale results.
Enter the streaming data warehouse. This new class of analytics technology delivers up-to-the-second analysis as data streams in. Before the store manager can even dial the phone, the streaming data warehouse has received and analyzed the transaction data, and put the replenishment process in motion.
Since none of us like to wait, the streaming data warehouse is poised to play a big role in driving our economy. Let’s take a look at a few key facts about streaming data warehouses so you can get up to speed.
What’s the matter with my traditional data warehouse?
A traditional data warehouse is designed to store batches of data, made available for analysis after the fact. However, enterprises are no longer focused primarily on batch data because data-driven business decisions must be made in real time.
Businesses are now being asked to incorporate streaming, location, and machine learning data into their data warehouse architecture. Traditional data warehouses were not designed for these data pipelines. Moreover, the traditional data warehouse was not designed for real-time query and data analysis. The data warehouse today must not only store data for analysis later, but also convert multi-dimensional data into immediate analytical insight for business consumption.
What differentiates a streaming data warehouse from a traditional data warehouse?
Whereas the traditional data warehouse is focused on the first mile of ingesting and storing data for analysis, the streaming data warehouse both ingests and stores data, and analyzes that data in real time as it is received. The key difference is that traditional data warehouses can perform analytics, but cannot run them in real time and are limited in the type of analytics they support. With the streaming data warehouse, businesses can transform data into immediate, usable insight.
How does a streaming data warehouse work in the wild?
Organizations from telecommunications to financial services, the public sector to retail, are all facing challenges that a streaming data warehouse could tackle, delivering up-to-the-second analysis that incorporates all of their data.
For instance, the Kinetica streaming data warehouse is used across the public sector to process and analyze IoT and edge data at scale. The streaming data warehouse can identify national security threats in real time, model complex disaster risks to determine resources allocation in the moment, proactively predict cyber threats, and enable dynamic logistics and supply chains, from mail to medical.
Elsewhere, telecom providers can blend complex geospatial and business datasets to understand the demands on their 4G networks as they occur, and accelerate their planning and rollout of 5G to high-traffic areas, for example. While banking and capital markets organizations gain real-time, on-demand results for trade and risk decisioning to match the speed of the market.
One of the world’s largest retailers uses a streaming data warehouse to make real-time inventory replenishment decisions, coordinating its distribution centers and stores to ensure they are always stocked with the products consumers want. With thousands of stores and hundreds of thousands of SKUs, the retailer sells thousands of dollars worth of products every second. At this scale, inefficiencies in the supply chain have an impact measured in the billions of dollars, both in terms of lost revenues and inventory misalignment.
With a streaming data warehouse, the retailer can quickly pose complex questions to diverse sets of data in order to make real-time stocking decisions, visualize inventory as it moves through the supply chain and route it accordingly, and turn replenishment operations into a dynamic process to react to fluctuations in demand. This helps ensure only optimal quantities of products are on the shelves at any time, to avoid shortages, overstock, spoiled inventory, and inventory-demand misalignment that lead to lost revenue.
A streaming data warehouse must be complicated to work with, right?
Building analytical applications typically requires patching together different tools for different data needs — graph tools, location tools, and so on. Each different type of analytics requires its own component, resulting in a complex data architecture that is more brittle, harder to troubleshoot, and difficult to evolve over time.
By combining streaming, location, and machine learning analytics into a unified platform, the streaming data warehouse actually untangles a data architecture. Users get off the ground more quickly without the hassle of diagnosing problems between various components, and the resulting simplified architecture more easily evolves to fit future needs.
The wait is over
With up-to-the-second results that incorporate all of an organization’s data, the streaming data warehouse is built for the dynamic environment we find ourselves in today. From responsive supply distribution of personal protective equipment to accelerating identification of trash in polluted waterways, our ability to quickly make sense of complex information will be crucial to providing better services and building better communities — and for keeping our ripe avocados in stock, too.