Harnessing Data Streams and Lakes with Data Grids
Jason Kolb, Chief Technology Officer
Originally published on the Uptake Blog
Without a doubt, electricity changed the world.
But it wasn’t until an efficient delivery network that electricity went from being a niche solution to becoming a new way of life for the developed world.
In the late 1880s, the standard for distributing electricity was Thomas Edison’s direct current technology. A few fundamental flaws limited the adoption of electricity: Power was lost when transmitted over a great distance, multiple lines were required to transmit different voltages for different purposes and the adoption of electricity required an additional distributed power generation infrastructure.
It wasn’t until a group of European scientists developed the competing alternating current standard that the fully realized electric future arrived. Direct current created the market and drove awareness. But alternating current enabled it by providing efficient distribution of energy across a grid, allowing for things like light bulbs and electric stoves to tap into energy generated hundreds of miles away. This ultimately became the ecosystem that powers our appliances, computers and transportation today.
The Promise: Making Everyone Better
Many successes in technology are game-changers because they solve distribution problems and build ecosystems around promising technology. Some of the most critical infrastructure in the world consists of the distribution networks for moving media, entertainment, electricity and data.
Today’s predictive analytics infrastructure is in a similar place as Edison’s direct current technology for electricity was: It’s crude, inefficient and doesn’t realize the promise of a truly connected and data-driven world.
The true promise of the Internet of Things — a connected everything — is that harnessing and transforming data into relevant information can make people and entire organizations better at what they do. By synthesizing the data from sensors, workflows and the entire business, we can reduce downtime, forecast important events and predict profit and loss based on what is happening in the real world.
The Next Shift
This isn’t a new idea. The technology world has been working towards this idea for decades and so far has gone through two major iterations of how we view and analyze data. We’re about to undergo a third.
The first major shift happened when organizations started buying software to generate data around specific workflows. There was little awareness that nearly everything could generate useful data, but people with vision realized the potential of valuable information in data that could be used to manage a business.
The second shift happened when organizations began valuing data and actively looked to generate more of it. Existing systems were upgraded and extended to create more data, and care was taken to preserve and make data more useful. The first crude predictive systems were developed. Business intelligence applications were built to expose that data, dashboards to share it and scheduled reports to stay on top of it.
The third shift is just beginning. It takes us to a time where data and predictions are at the very core of data-driven businesses. In this phase, all critical workflows in a company are plugged into a common backbone. This backbone is similar to the electrical grid, except that feedback is as important as distribution. Similar to the development of the electrical grid, a better, more sophisticated version than we have today is needed to bring us into the future.
The Data Grid
At Uptake, we believe effective and leading organizations need to rethink their approach to analytics and connectivity. Leaders need to go from being data-aware to data-driven, and we need a piece of infrastructure that distributes data intelligently between data producers and data consumers. Once that is in place, our usage patterns will undergo the same sort of radical shift that electricity did when the electrical grid was converted from DC to AC.
The key is to look at data as a raw resource. Data is often referred to as oil that can be converted into gasoline to drive a business. But that’s the wrong analogy. Data is much more like water — rather than refining it and using up the output, we need to put a hydroelectric dam in front of it and use it to power a grid of interconnected nodes. Then, the data lakes and streams we collect can be transformed into answers and plugged into a grid that runs the entire organization. This is the critical infrastructure that covers the last mile and takes us from answers to actions. This is the genesis of the data grid concept. This is at the foundation of Uptake Core.
The true promise of the Internet of Things — a connected everything — is that harnessing and transforming data into relevant information can make people and entire organizations better at what they do.
The grid itself is smart connectivity. It knows about the nodes that live on it, what they need and what they’re doing, and it delivers the answers they need at the right time. It is both a transmission medium and an ecosystem. Similar to how the electrical grid allows you to choose which toaster or microwave you want to plug in, a data grid allows us to choose solutions in an elegant way.
Nodes on the grid are a combination of data producers and data consumers. By erasing the distinction between producers and consumers, nodes can both take information off the wire as well as put more back in. Now an application that does work can report back on what happened. Nodes can feed efficacy data to the grid and report to the platform how effective they are.
Here’s a snapshot of how it works: One node on the network is a predictive model using data from the grid to recommend a procedure that prevents failure of a machine. Another node is a model predicting the exact same thing, but with a completely different algorithm and better efficacy. Another node is a shop tool application for the foreman, showing the best recommendations coming from the grid. In the shop, another node is on the technician’s iPad as he follows, step-by-step, the most effective procedure for the recommendation. And yet another node watches the outcome of those actions, monitoring the health of the machine after it leaves the shop and reporting efficacy information back to the entire grid.
Survival of the Fittest
Because the grid knows how effective each node is, an ecosystem is created where the most effective solutions rise to the top. Predictive models, workflow applications, and automated services all feed from and contribute back to the same backbone, and predictive models will only be consulted if they are performing better than their competition. Workflows will only be used if they are the most effective for a given facility. A worker has a choice between multiple workflow tools so that it’s easy to find the right one to fit his or her specific role.
Applications and services plug directly into the data grid, harnessing the power of all of the nodes to instantly start running in new and exciting ways. No longer do applications or services need to do the heavy lifting of converting data into actionable predictions, now they can focus on what they set out to do: Deliver user experiences built around predictions, answers and work that needs to be done.
Uptake Core is our unique implementation of a data grid. We have developed not only the underlying connectivity but also an diverse set of nodes that plug into it. Infrastructure is only as valuable as the ecosystem that lives on it, so we have jump-started our ecosystem by investing in building out critical nodes. Bundled with Uptake Core are nodes that plug in predictive models to forecast failure, recommend procedures, monitor productivity, assist with fixes, display data visualizations, monitor fleets and much more. We develop custom nodes for specific uses and industries, and the nodes make Core ever more powerful.
As all of these nodes participate on the grid, the grid itself becomes smarter. The grid collects data on all of the activity that takes place on it, and each new node and application that plugs in can contribute value to every other application that builds on it, much in the same way that houses with solar panels can put energy back into the electrical grid. Applications plugged into Uptake Core can contribute information back to the grid that is constantly turning data into predictions, answers and actionable information that can be used to power a whole new host of use cases.
Uptake’s Promise: Simple, Elegant, Flexible
What does Uptake Core look like? From the end-user’s perspective it looks like beautiful, user-friendly software–applications that fit the user’s needs, elegantly provide information, and simply prompt action. It looks like the most intelligent tool they’ve ever used because it’s simply telling them the answers without confusing interface flows; it just makes recommendations and brings useful information to their attention.
Behind the scenes is incredible horsepower — smart connectivity and horizontally scalable computing that can crunch massive streams of data in real-time. Uptake Core supports nodes that evaluate hundreds of thousands of decisions per second to decide what should be done next. It supports thousands of predictions per second and route the output to the nodes that need it.
Uptake Core brings tools and data together in a way that is simple and elegant, and flexible enough to deliver massive value quickly while adapting to the complexities and subtleties of an individual business. We’ve developed a set of common interface toolkits and paradigms and all of the nodes on our grid speak the same language and bring important information to users quickly, easily, and in a way that encourages action.
At Uptake, we’ve realized that we have to seed the data grid with critical infrastructure and functionality before it reaches an inflection point that enables technology to just “work,” and that’s what we’ve done. We have developed a platform that serves as the connective tissue to take connected data, route it to the apps and functions that need it, and then route the answers to the people who need them. And we’ve begun building out an ecosystem of nodes and applications on that infrastructure to pull the future towards us as quickly as possible.
It took 100 years from the time the electrical grid was started until it was widely in use. It took mobile phones only 25 years to reach 100 percent adoption. It’s easy to see the promise of new technology, but it’s not immediately obvious what it will take to get there. It has to reach that inflection point. Then it simply becomes a necessity of life.