Journey on My First Web Application for Geoscience. Things I Learned So Far (Part-1)
What Motivates Me, Difficulties Along the Way, and Benefit of Putting It Out Online (www.plotpetrophysics.xyz)
Why Am I Making the Effort?
Over the course of 2021, I sometimes gave lectures to students in university about geoscience, in particular; Petrophysics. It was all theory and some examples, but these data (which we called well-log data) is complex, have multiple units of measurement, and can be daunting to learn at first. Let alone imagining via presentation and theory. I always try to visualize it using commercial software, but there is a low possibility for them to recreate the one I showed them on their own.
There are two big parts in learning petrophysics. Data and Software.
In the beginning, well-log data is not an easy task to find, but the rising of open-source data initiatives like Volve Dataset, and any other resources like KGS, really help to provide a good dataset for people to work with. The second part is the software. Having good software will not fix all of your problems, but “not having it, will”.
Not having a good software is like working on table data but could not create a chart out of it.
Ever since the beginning of the oil and gas industry, graphs, visualization have been the core of the well-log analysis (or petrophysical analysis). One needs to be able to showcase the data, do QC on it, and extract information about the rock (reservoir) and the fluid (hydrocarbon) it contains in a quantitative output (volume of rocks constituent, volume of empty space/pores, and volume of the fluid).
This can be done using commercial software, they are good but expensive. For a company, that might be okay, but for a university student? Not the case.
This is the part where students hit the wall.
Some universities may have the privilege to use this commercial software via some mutual agreement, but many universities do not. Even if they do, they usually have restrictions on features, how many people can have access to it, duration of use, so on and so forth. There has been a big gap in the oil and gas industry in terms of technology and having an additional gap in terms of software will make this gap even larger. I feel like I can do something about it.
“How about making an online plotting website for well-log data?”
With just one python online course via Udemy and google on my back, here is my story in making this comes true.
Let the journey begins!
Start from Jupyter Notebook
10th October 2021. My first attempt is by creating a Jupyter notebook of inputting LAS files, loading them using a library like LASIO, and finally doing some plotting using Matplotlib. Nothing fancy, just trying to make a decent triple combo plot (common plot in petrophysics where you plot the essential data in a “combo” of one resistivity log and two porosity logs, hence “triple”). I am not even doing it on my laptop, all done in Google Collab.
I’ll be honest, I am not convinced I can do it till finished, therefore I took the easiest way I can do it.
For the first one, I primarily watch Andy McDonald's youtube channel and start tinkering with the color, shading, things I want for my final plot.
I was able to survive and finish the notebook, with a somewhat decent-looking plot. I know I can improve it, so I added a future project at the end (“Define Function to Plot TC Plot”).
Creating Functions
15th October 2021. I am still in a jupyter notebook in Google Collab, I know I am close to what should be the ideal solution to this plot, a function to create a single plot by defining the parameters. I know I need to do it because we as Petrophycisist changed the scale, color, depth, all the time. As someone said;
“if you run the same line of codes more than twice, you need to make a function.”
Like always, I am not gonna reinvent the wheel. I start looking at Yohanes Nuwara GitHub to get a pretty rough idea of how to make a good function to fit my needs and add a bit of spice here and there.
I feel like a magician here, as I just have to write a single line of code and set up some parameters, and just like that, I have a plot!
This is I think the time I got hooked up by python
I put it on Linkedin (because that is where most of my colleague is), and give some tutorial on how to use the function by themselves.
There is just one problem, how can I make it available for most people? I know people can fork it from my GitHub, but for most people who are not familiar with Github, they would probably just skip it over simply because they dont know how to fork a repo/ how to run the jupyter notebook. It is not a hard thing to do, but I am trying to remove as many barriers as I can.
So for that, I need to do two things:
- I need to make it online, and free of use.
- I need to make a pipeline, that takes the LAS file and automatically loads the entire setting and people just have to adjust what they want on their parameters.
Meeting Streamlit
27th October 2021. I realized that I need to deploy the app to the internet, and for that, I need a library to do so. Some libraries may help, but so far I have been into Streamlit. Streamlit is one of the new python libraries, where it was designed specifically to make it easy to make an app works. Their tagline is “The fastest way to build and share data apps” — and spoiler alert, they are not lying.
As always, Youtube is the best place to start. I know what I want to build in mind, I just don’t know the “how”. For this, I watch Mısra Turp's explanation on youtube, which pretty much covers all I need to make an app. I highly recommend this to everyone interested in making one.
It was scary at first, cause I have to run it on my local computer (installing VSCode, learning git, etc), but I am glad I took the risk, as I learned a lot during this time. Dealing with errors, missing dependencies, heck even incompatible library to another library. Long story short in just a line of code, the app runs smoothly in my local host, and everything seems to work just fine.
The deployment is another struggle (a story for another day I guess). I’ve been bumping between Heroku/ amazon. I found out that streamlit has its own cloud hosting that can be connected directly through GitHub, so there I go.
I don’t need to set up anything, just point to my repo, and run the app directly in the streamlit cloud. Voila! Welcome to the Triple Combo Plotter!
You will see that the error still persists in the welcome window, but once the data is loaded the app functioning properly as I explained in my Linkedin article. I know it is not perfect (first time), and I am okay with that.
I believe in making things work first, rather than chasing absolute perfection.
Once I get the app up and running, then some more feedback, inputs come in. So far my internal connection who knows me directly contacting me to give their inputs. I think this is the best part of putting it out there. You will have a different set of eyes to look at your app and give you a different perspective. Improvement is faster this way.
Virtual Environment
28th October 2021. I am working on fixing the bug when the LAS file has not been uploaded by the user. I read the documentation, search some threads in the community streamlit (they have been awesome), and tried multiple things that come to my mind. Until I found out.
The latest version of streamlit is not supporting LASIO library
In the first version of the app, I did use the latest version of streamlit because it supports nicer UI, and using the latest version is just a no-brainer. But then I found out that any version higher than 0.67 would not be supporting the LASIO library, which is a pretty big headache for me. How I am gonna make this work without the LASIO library?
conda create -n myenvTurns out, you can work it out by using a virtual environment. I can use an older version of the library by installing it inside a virtual environment I set up. With this, I don’t have to worry it will affect my current version on my laptop. With that, I create a new virtual environment using Conda and start working on fixing the bug of my app.
Shortly during this time, I work to improve some bugs, mainly dealing with a different units of the same dataset, adding some more explanations to make things easier for the user, improving the general flow of the app. I even go as far as creating a tutorial in PDF.
Why not add the documentation in the repo you asked?
At this time I have no idea about markdown file, I thought README.md is just a plain text file! Silly me!
The virtual environment works, and I successfully remove the bugs, and everything works perfectly. My app even got featured in the community!
Web App Triple Combo Plotter
Shortly after this, I did something stupid and lost all the updates (git nightmare), able to add more features using plotly (cool stuff) and eventually moving out of streamlit cloud.
Will explain in PART-2, stay tuned!
