Overcoming Data Challenges in A Machine Learning Project
A Real-World Project.
For most people when it comes to “Artificial Intelligence” or “Machine Learning” it brings up memories of sci-fi movies they have watched. Some think of robots and some of the terminators.
The truth and real potential of AI tell a completely different story. AI has the potential to address some of the world´s most challenging problems. The amazing part is that it’s already happening!
What a great time to be alive!
Though this field looks glossy from the outside, the inner workings of it are a different story. Like any other innovative technology, Machine learning is at its very early stage and much of the innovation is still in progress. The results are that there are numerous challenges and bottlenecks.
Let’s dive deep into a real-world project and see what Machine Learning engineers do to overcome the challenges.
The case: Increasing solar energy adoption
In partnership with the Engie factory and Solar AI, 40 Omdena collaborators worked for 8 weeks on detecting rooftops via satellite images to identify rooftop features that are crucial for solar panel installments. The developed solutions could significantly help to facilitate solar energy adoption by reducing the time and costs of the rooftop solar assessment process. We used satellite imagery of Singapore’s Jurong Island and the main island for this project. All the annotations were done manually using QGIS. Read more details about the project in this informative article by a fellow collaborator.
Let´s dive into the challenges
1. Real-world data: The best horror movie?
I wish Harry (Potter) never wasted his time in quidditch and came up with a spell to clean and improve input data for machine learning models. Until there’s a magic spell to do so, we’ll have to stick with getting our hands dirty to clean and improve the raw data.
As mentioned, we got the “.tiff” file as input data of Jurong and the main island of Singapore. We first converted the huge tiff file into smaller tiles by clipping frames of 512x512 dimensions so that the data becomes more manageable.
The resolution turned out to be insufficient for some of the challenges ahead. In Computer Vision projects, we often find input images to be inadequate for finding fine features. We tried to manipulate the data by increasing the resolution using machine learning approaches.
We started by first acquiring data to feed into a machine learning model that can minimize the gap between the supplied resolution images to its desired-resolution counterpart. The gathered high-resolution data was first converted into a low-resolution image by adding noise. This gave us a low-high resolution image pairs dataset that can be used as input data and output labels respectively. We implemented Generative adversarial networks crafted for this task and multiple models on the training dataset to extract high-res images from low-res ones. Winner takes all!
In machine learning implementations, excessive manipulation of input data is not recommended. Over manipulation causes the machine learning model to predict poorly even if there is very little change in the data. Generalization is the key to make your machine learning model sustainable in production.
Concept drift: means that the statistical properties of the target variable, which the model is trying to predict, changes over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.
2. When expectations are not results
Usually, when you see a model’s implementation on Github, you see astonishing results and you think that your struggles are over once you apply this model to your data, only to find out that it didn’t work with your data and the results you are expecting is not what you are getting. Papers and official Github repositories are tailored to data they have experimented upon and sometimes it might work perfectly on your data. In that case, Aladdin just granted you a wish, congratulation! But this is not the case every time.
The data is gathered, the cleaning and pre-processing have been done. The next step is to implement machine learning models to the data. You are not in control anymore, the data is. The data you have prepared chooses which model it wants and you have to go with it. We implemented Neural Networks focused on image segmentation to detect building rooftops and their types in the input images. We compared the results on our data and we found out the model that was the best candidate for the job; hired! Still, the masks were not perfect, we could still see that the masks were undershooting the roof area. This can cause a problem in the later phases of the project.
Suit up, let’s tackle this problem. There are three major ways we can improve a model’s accuracy.
- Increasing the input dataset: The more the machines see, the more it can identify. We took up the task to annotate more dataset by including more rooftops on Jurong island. A time-consuming task but always worthly. We studied rooftop types that are already annotated and tried to diversify roof annotations by including varied roof shapes like industrial rooftops and curved roofs.
- Train and after that, train some more: We implemented the model with 200k iterations and observed the trend in loss functions. We plotted the losses and manually decided on the global minima. We use Occam’s razor to select simple solutions by making simple assumptions. This way we could come up with a solution in a reasonable amount of time.
- Playing with the hyperparameters: If you have worked on any real-world dataset, you already know how crucial parameter tuning is. Consider the model as a radio and hyperparameters as the frequency selector; It gives you the capability to mold the neural network according to your data.
3. The fall of Machine Learning gives rise to your creativity
The state of the art machine learning algorithms does not help you for free. They demand an appropriate dataset, a certain capacity of resources, and computation time. It’s not always necessary that you’ll be able to bear these demands. What do you do then? Leave the project? Well, that’s not an option we consider!
After detecting the rooftops, we were set to find a few final deliverables that can calculate the solar potential of the rooftop. We need to detect obstacles on rooftops so that we can subtract that area from the rooftop area. Segmentation of the roof facets of a hip roof needs to be done to calculate which geographical direction each facet is facing. These calculations were crucial because of two reasons:
- We cannot put solar panels on the roof segment where there is some obstacle like an air vent is present.
- The energy output of a solar panel depends upon the geographical direction it is facing.
The problem:
If we follow our earlier approach of using the neural networks to detect roof facets, we’ll first need to annotate a huge amount of data. Looking at the image on the left, it’s clear that this is next to an impossible task. With the huge number of buildings per image and the inadequate-resolution; the roof facets were not even visible to the human eye. Remember, our goal is not just calculating solar potential, but also to present the solution in a limited period of 8 weeks. This was not feasible, neither we had enough coffee stock to complete this.
Computer Vision to the rescue!
1. Detecting the obstacles
Clearly, the annotation was not feasible, so we moved on to detecting using the pixel intensity itself. We started by canny edge detection technique to find obstacles inside a mask area but due to dynamic threshold values, we had to drop this method. The different buildings used different threshold values due to varied pixel intensities. The auto canny threshold detector didn’t give us the desired results.
Next in line was using the pixel intensity to create an intensity map and decide on which value ranges we consider it to be an obstacle. Interestingly, we found that the highest two bins of intensity values correspond to an elevation in the surface. Assuming that this happens where there is an obstacle on the rooftop, we considered all the elevated pixels to be an obstacle.
2. Segmentation of the roof facets from the rooftops
To divide the roof facets of a rooftop, we used canny edge detection with contour approximation techniques. This implementation failed because a lot of unwanted areas getting detected like tree shadow, air vents, and etc. Another reason for the failure of this experiment was the resolution of the images; even we cannot make any difference between different roof facets of a roof. So we had to drop this idea.
Since we cannot detect the roof facets, we moved on to estimation under the regulation approach to calculate the roof facets. We made assumptions that a hip roof will contain symmetrical shapes like triangles, trapezoids, rectangles, or parallelogram. This helped to estimate the rooftops by combining the edge detection method with estimation regularizations
Final words with some takeaways!
Getting an opportunity to work on a project that can potentially help the planet in the future was a delight in itself. We faced numerous issues during the course of this project. The key thing we found out that there’s a lot to be done in this field. The possibilities are endless and you have a lot of room to innovate.
“That which does not kill us makes us stronger.” — Friedrich Nietzsche
Working on a real-world dataset is somewhat similar to this quote. You’ll face real issues, major setbacks and often you’ll get disappointed because the assumptions you made didn’t work out as you expected. But this is what that will make you a better Machine Learning Engineer and a better problem-solver in general. I say take a leap, enter the dirt and find a first-hand experience by working on a real-world project.
