How We Made 1B+ Time/Depth Drilling Data Points Available in Real-Time in 4 Weeks
Concho Resources’ Experience Deploying the Tempus WITSML Accelerator
by Justin Shivers and Kelly Kohlleffel
I recently had the pleasure of visiting with Justin Shivers, Solution Architect for Analytics and BI at Concho Resources, after completing a project together to implement Hashmap’s Tempus IIoT Framework for Drilling Data and Analytics.
Concho Resources has been a major E&P operator in the Permian Basin for many years, and Justin and I discussed the data challenges they face as they continue to transform their company by leveraging digital technologies and real time data most effectively. We also talked about the formula they used to derive initial business value and some of the short-term results and their longer-term objectives.
Kohlleffel: Justin, you’ve been at Concho since 2011 working across a variety of BI and Analytics roles, but last year really started taking a serious look at real time data streams for business analytics enablement — what were you seeing across the company or within the industry that caused you to start looking at delivering real time insights to the business?
Shivers: This WITSML drilling data project was really our first step into the real-time analytics space and was predominantly driven by a diverse set of business requirements across our drilling, completions, and geotech teams. We had already started implementing a corporate data lake with Cloudera Enterprise Data Hub and it became apparent, as we discussed use cases with our business owners, that real time data was going to be critical to meeting their expectations long term.
Our drilling teams were looking for ways to continue to improve a number of existing workflows including MSE Calculations, Rig State Evaluation, BHA Performance, along with other metrics, and at the same time, our completions and geotech groups had future needs for leveraging machine learning and predictive models on real time streaming data. We also wanted better ways to visualize our real-time data streams and easily customize views for various groups and users.
Kohlleffel: As you were looking at delivering on the use cases you described, what difficulties presented themselves when dealing with your WITSML data streams?
Shivers: Several primary issues come to mind: #1 — WITSML data streams are fairly challenging to deal with in and of themselves — getting the data parsed, serialized, and made useable for our business and analytics teams was a big challenge. #2 — Getting the data into a useable format only solved part of the equation — to make it truly relevant within the content of our business and operational systems, the WITSML data stream needed to be enriched with information from other core operational systems, in real time. Both issues were going to require technical and domain experience along with software technology and development that we didn’t have in house at the time. #3 — Additionally, our business units required some custom calculations and computations delivered in real time that were outside the scope of our EDR service provider.
Kohlleffel: The areas that you described seem to be common themes across operators regardless of company size, operating basin, or company history. Can you talk about how you decided to address your real time data requirements?
Shivers: It’s actually an interesting story about how Concho and Hashmap got connected. Our CIO had reviewed a WITSML blog post from Hashmap and asked me to followup with the company and their approach.
As a organization we have trended more towards quick, high value engagements that leverage open source software approaches to solving technical and business challenges, and Hashmap was well down the road to developing an Industrial IoT framework, Tempus, that was open source, could be deployed quickly, and would seamlessly integrate into our existing operational systems such as Pason, as well as Cloudera, our Big Data platform, and other enterprise systems that we already had in place.
Last year, Hashmap had the Tempus WITSML Accelerator ready to install (connecting real time WITSML data streams from our EDR provider with a WITSML Client and SDK) and we deployed that as a first step by connecting our Pason WITSML data stream to Cloudera providing a nice parsed and serialized WITSML data stream that we stored in Apache Kudu. We did that in about 4 weeks.
We still had a need for a UI that would allow us to visualize these data streams in real time as well as configure business units, assets, and devices. Hashmap’s open source Tempus UI became available early in 2018, so we deployed the Tempus Application Server, Rules Engine, and Real Time UI in Q1. We did this is about 4 weeks as well.
Kohlleffel: It sounds like there were some benefits that cut across both the business and IT — can you describe what you experienced?
Shivers: The key thing for us was the ability to move quickly. In just 4 weeks, working together, we completed the Tempus WITSML Accelerator including ingesting, processing, parsing, and serializing all Pason historical data as well as all the real-time ”drilling” or active wells effectively capturing 1B+ time-indexed and depth-indexed points and making those available in real time.
Some of the initial business value that Tempus provides as a real time surveillance, alerting, and monitoring solution includes an ability to do ”look-backs” and to assist in decision making on drilling new wells along with enabling enrichment of well attributes (this is a big deal for us — enriching our real time streams with well master data) and joining of WITSML data with other key operational datasets.
In addition, we can provide custom drilling calculations and computations not provided by our EDR provider along with drilling specific dashboards.
From an IT perspective, we like the cost savings of going the open source approach versus licensing domain specific software while still making WITSML data enrichment and actual useability in real time a reality.
Also, we can take Tempus in a number of directions including to our Production group for OPC-related real time datasets which sit outside Drilling, Completions, and GeoTech. Lastly, we are seeing that customizations, modifications, and enhancements are much quicker and easier to achieve.
Kohlleffel: Were there other ancillary benefits that you saw as part of the deployment?
Shivers: Connecting our operational systems to our enterprise systems such as Cloudera was a critical for us — until we deployed Tempus, we didn’t have a way to join data in the field with our data and systems in the data center — operational data and enterprise data were not connected.
Hashmap also helped us ramp up performance on Apache Kudu (ultimately where we are persisting all of the real time data for long term, low latency storage) and we’ve seen a significant performance benefit especially retrieving data in large volumes as we’ve been able to do intelligent routing of data to Kudu via Spark. Additionally, they assisted in connecting our corporate visualization tool, Tibco Spotfire, to Kudu for historical data analysis and visualizations.
Kohlleffel: Justin, do you have any lessons learned as you’ve forged a path at Concho for real time data and analytics?
Shivers: First and foremost, engage your key business stakeholders throughout the process — do it early on with use case workshops and requirements discussions and then throughout the deployment to get critical feedback on the way the real time solution will be used and how it can best fit into existing workflows.
Also, don’t take for granted the value of enriching your real time data, whether you use that enrichment during a real time process or later on for historical analysis. Lastly, look across your business lines for opportunities to affect value with real time data streams. At Concho, we started with drilling data, but there are an equal number opportunities to assist our production team as well with real time data and analytics.
Kohlleffel: Are there any other Tempus engineering developments that are interesting to you for the future?
Shivers: As stated earlier, we really like the fact that Tempus can be used for any real time data stream — drilling, production, or other. Also, as part of our engagement with Hashmap, they provided ESRI ArcGIS integration with Tempus so that we could leverage ESRI maps as a widget within the Tempus dashboard framework. Lastly, we are eager to see some of the drilling specific enhancements that the engineering team is working on as well as what is going on in the data quality area and how that development can be extensible to our broader application and system ecosystem.
Kohlleffel: Justin, I really appreciate your time today, your ongoing support and partnership, and your thoughts and perspectives on the initial work that you’ve done at Concho with real time drilling data.
Feel free to share on other channels and be sure and keep up with all new content from Hashmap at https://medium.com/hashmapinc.
Justin Shivers is a Solutions Architect for Analytics and BI at Concho Resources and Kelly Kohlleffel has responsibility for sales, marketing, and alliances at Hashmap. Feel free to reach Justin or Kelly on LinkedIn.
Be sure and catch Hashmap’s weekly IoT on Tap podcast for IoT discussions with a developer focus.
