Building an Oilfield Production Surveillance Dashboard
…. from scratch to deployment
Introduction
In simple terms, production surveillance (oilfield context) means monitoring changes in well parameters for each well in a producing oilfield. Well parameters include rates, pressures, choke size, basic sediment & water, gas-oil ratio, uptime among others.
This is done to:
- identify possible anomalous data.
- to explain how changes in one or multiple parameters affect the parameter of most interest (oil rate) and how those parameters can be better optimized to achieve the production target.
For effective production surveillance, I decided to build a dashboard.
Requirements
In building the dashboard, I set out some requirements;
- data needs to be dynamically updated on the dashboard.
- the dashboard can answer some common questions always asked e.g what's the average production over a certain period among others.
- the web framework for the dashboard needs to be deployed so it can be accessed by anyone in the team.
Building Process
The first iteration in the building process is as shown below:
In the first iteration, I had a folder where I kept 92 daily reports (~3 months). The daily report is an excel file with a lot of data in it sent daily from the oilfield production facility.
The python script can be said to be in two parts;
- data wrangling script: this part of the script access the data of interest from each daily report kept in the folder, cleans up the data and organizes the data into a structured format (DataFrame).
- dashboard script: this part creates the dashboard with visualizations and the web framework.
The dashboard output (short video) of the first iteration can be seen here:
The limitations with the dashboard output of the first iteration are that it didn’t meet the requirements I set out;
- I had to manually run the script each time I added a new daily report to the folder.
- The dashboard output only runs locally on my computer.
- I can sure do better with the dashboard output in terms of the layout and presenting the data.
To address the first two limitations above, I had to learn google cloud platform tools and for the 3rd limitation, I had to learn the dash bootstrap components library.
The learnings led to the second iteration which is shown below;
In the second iteration,
I moved the daily reports in the folder in the first iteration to a google cloud storage (gcs) bucket.
The data wrangling script part of the python script in the first iteration was reviewed to access data from the reports in the gcs bucket and then deployed as a function in the google cloud function (gcf).
The beauty of the gcf is that it is event-driven which means the function only runs when there is a trigger (in my case, only when I add a new daily report to the gcs bucket).
The output of the data wrangling script is a DataFrame that was then stored as an excel file in a new gcs bucket (second bucket). Anytime, there is a trigger of the function, the new output overwrites the existing output in the gcs bucket.
The dashboard script part of the python script in the first iteration was reviewed to access data from the second (new) gcs bucket, the web framework improved upon using the dash bootstrap components and was deployed in the google app engine.
The beauty of the dash bootstrap components is that it allows you to structure your layout in rows and columns. This makes it easy for one to have control of the design of the layout.
The google app engine allows for web applications to be deployed in a serverless manner which then means the dashboard output can be shared with any other person.
In other to validate the second iteration, I uploaded 30 additional daily reports into the first gcs bucket (started 1 report at a time, after the 3rd upload, I uploaded the remaining 27 daily reports at once).
The metrics of the google cloud function and app engine can be seen here:
The metrics shows that both the cloud function and app engine ran perfectly fine with no errors.
The dashboard output of the second iteration can be seen here: https://dummy-surveillance-project.nw.r.appspot.com/
- *Note: because of the data privacy policy, the actual wells data have been replaced with randomly generated data.
Conclusion
The dashboard output of the second iteration met the requirements I set out initially. Working on this project made me realize that scale can be multi-dimensional and not one-dimensional.
While the dashboard output works very fine for additional (same well) data from the subsequent daily reports, it sure won’t capture data from an entirely new well (in the event a new well is brought online and data added to the daily report).
Also, because I used integer location-based indexing (.iloc) to access data from the daily reports, any changes (inserting a new row or column) made around the cells of interest in the daily report will affect the output of the dashboard.
In future iterations, I will try to address these concerns among other things.
Thank you for reading.
The repository can be found here: https://github.com/0ladayo/production_surveillance_project
