What Do You Need to Stream?

By Roger Rea

Getting the data is sometimes the hardest part of continuous intelligence.

Recently Jim Sharpe, CEO of Sharpe Engineering and I were talking about creating three applications for a customer. They were really very simple applications, and the partner said he’d have all three in prototype mode early the next week. But ONLY if he had access to the data. Otherwise, it could take several weeks or even months.

Unfortunately, that’s often the case for continuous intelligence (CI) applications. Data access issues can include legal, political, expense, and invention. Data may be the new oil, but unfortunately, just like oil, data is sometimes hard to get. So, as you begin your journey to CI, make sure you consider access to data.

With the advent of the General Data Protection Regulation ( GDPR) in Europe, using personal data requires consent. And furthermore, use is limited to the purpose granted. Suddenly a whole class of CI apps for marketing and customer care is much more difficult to put into use. Many countries also have laws that require data to be processed only in their country, further restricting data access. Either of these might delay or even stop your ability to create the CI apps you want cost-effectively.

It’s strange to think that getting access to data could be political, but often it comes up. One part of an organization owns the data, and they want to prevent or control access to ‘their’ data. Never mind that it’s the organization’s data, not their particular corner of the organization. They spent money collecting and storing their data and don’t want just to give it away. Often, they are concerned that if they pass out their data, someone else will use it to tell a different story. They want to control the story, so they don’t pass along the data.

One hospital spent weeks arguing that data from intensive care unit (ICU) monitors could not be sent to a central location within the hospital for analysis. Why, you ask? One faction was concerned about adding traffic to the hospital networks. What if that added bandwidth disrupted critical communications? What if someone could access the network and see data they shouldn’t see? That might be a HIPAA violation! Some might call this just political maneuvering to block access to data. But I’d suggest these are legitimate questions that must be addressed.

Just as oil is hard to find, it’s not free. And getting data can be expensive, too. With departmental turf wars, the people who own and curate their data might charge you when you want their data. One telco company doing customer care applications estimated that its costs were only 10 percent of the total project expense. The remainder was paying for access to the data, transporting the data, training users of the applications, reporting, and support.

Many sources of data, such as government data, are public and free. For example, the U.S. government continues to provide more and more free data at its open data site. However, the number of its sites collecting weather data is far smaller than The Weather Company, so it may be worthwhile to pay for the additional granularity available from The Weather Company.

At one company, the data they wanted came from transactional databases. While technology exists from multiple vendors to read the log files to capture this data, the company’s implementation was so large that the change data capture solutions couldn’t provide just the updates without re-doing the database and table structures. This involved not only costs for the change data capture software, but also changes to many applications using the existing database organization.

In many situations, the data simply might not be available. One car manufacturer wanted to perform telematics on its cars-using location data to know where they were always and information from the hundreds of onboard sensors. Many issues came up. Who owned the data from the car? The manufacturer or the car owner? Changes to purchase contracts could change that. How would the data be transported to the manufacturer? Wireless in the car, of course, but who would pay the wireless bill? And then if they were building out such a CI app, why limit it to only new cars? Why not create a dongle to plug into the car and sell it to existing car owners? They would appreciate the extra service of knowing exactly when maintenance might be required or when additional safety issues come up. It was nearly a year to develop the new dongle, delaying the implementation of the new CI applications.

In other situations, no sensors existed to collect the kind of data thought to be useful. In manufacturing, for example, audio and visual data can help determine when equipment like conveyor belts or bearings are going bad. Sensors can be built to get this data, but that’s the point-they must be built! Or use other existing technology like smartphones to do the data collection. After all, even 4- or 5-year-old smartphones have cameras and microphones.

Summary

CI apps can help companies improve profitability, as described in The Case for Continuous Intelligence. But without data to feed the apps, they can’t even be built. As you embark on your CI journey, consider the legal, political, expense, and possible invention hurdles that you may need to overcome to get that data in your overall business case. Get started early to ensure access to the data-it can take far longer than building the apps!

Originally published at https://www.rtinsights.com.

For more articles and resources on continuous intelligence, visit our Continuous Intelligence resource center here. To get the latest CI news and insights in your inbox, subscribe to our biweekly Continuous Intelligence newsletter.