I Focused on WITSML Data for a Week and Here’s What Happened
Making WITSML 1.3.1.1 and 1.4.1.1 Data Streams Analytics Friendly and Consumption Ready
by Chris Herrera
As a Big Data / Connected Data / Open Source architect and consultant, I am working with clients daily to help make complex file types easier to deal with and more useable.
Bill Schmarzo, CTO at Dell EMC Services, in a recent blog outlined the significant difference between Big Data and the Internet of Things, stating that managing real-time streaming data and making real-time decisions at the edge are critical for IoT analytics.
To this end, WITSML data can be especially challenging to ingest, process, operate on, and work with in a Big Data context, so I’ve developed an Apache 2.0 open source, Java-based WITSML Objects Library SDK that simplifies working with both WITSML 1.3.1.1 and 1.4.1.1 data.
What is WITSML?
By way of background, WITSML is a standard maintained by Energistics, a global consortium (Hashmap is a member organization) that facilitates the development of standards for the Upstream Oil & Gas industry. WITSML itself is an XML based specification that is used to transmit, and sometimes archive data that was generated throughout the construction phase of a well.
Questions My Peers Are Asking When Dealing with WITSML Data
So, what are some of the questions and requests that we are hearing from customers dealing with WITSML datasets?
· Why isn’t there a Java-based, open source toolkit purpose built for processing WITSML files and allowing me to parse, enrich, and make my WITSML data more useable?
· How can you help me take some of the tedious effort out of parsing and serializing the massive number of objects and types in both 1.3.1.1 (318 total) and 1.4.1.1 (380 total)?
· Is there a way to avoid parsing tons of XML but still allow me to expose my WITSML data in a domain centric way?
· How do I deal with 2 different versions of WITSML and seamlessly convert between them?
Use the SDK to Start Working with WITSML Data
If you have WITSML data sitting on a drive, in a shared folder, or on your WITSML servers, and want to start ingesting, processing, operating on, and working with that data in a Big Data context, this library will help you accelerate that process.
The WITSML Objects Library SDK is…
· Open source (Apache 2.0) and Java-based
· Fully documented
· Tested with commercial WITSML servers (support both the 1.3.1.1 and 1.4.1.1 versions of WITSML)
· Ready to be used within your Apache NiFi, Apache Spark, Apache Flink, Apache Storm, and HDFS based applications or anywhere WITSML data needs to be parsed and handled
Deliver Business Value with your WITSML Data
There is a variety of potential end users and consumption patterns for WITSML data within your organization:
· Data Scientists and Business Power Users — equipment optimization and maintenance, lessons learned (what do I know, when did I know it), threshold detection (ROP optimization, stick-slip monitoring/detection, efficiency KPIs and alarming, etc.)
· Business Analysts, Drilling Engineers, Reservoir Engineers — offset well analysis and characterization, drilling hydraulics, etc.
Get Started
This SDK is a sub-project that will support our continuing work of developing a WITSML client for use in applications such as Apache NiFi and the ongoing development of the real-time data collection and transmission pipeline for Hashmap’s Tempus Industrial IoT Accelerator.
I’d encourage you to download the SDK and start working with WITSML data to address specific use cases within your organization.
We’d also be glad to assist you in this effort — we have a quick 4 week WITSML data accelerator consulting engagement that will get the SDK setup as a Spark job with a variety of output options and ways to start structuring your WITSML data for analytics.
Happy WITSMLing!
Feel free to share on other channels and be sure and keep up with all new content from Hashmap at https://medium.com/hashmapinc.
Chris Herrera is a Senior Enterprise Architect at Hashmap working across industries with a group of innovative technologists and domain experts accelerating the value of connected data for the open source community and customers. You can follow Chris on Twitter @cherrera2001 and connect with him on LinkedIn at linkedin.com/in/cherrera2001.
