How to detect construction progress in urban environment
Making machines work intelligently
Satellite imagery, once the rare privilege of a handful of high value customers is now a staple input for the day to day life for most people around the world. Map services provided by Google, Microsoft and Apple caters to personal use for free and business users at a minimal cost. The advantages and simplicity online mapping applications offer are evident — facilitating many solutions, generic and bespoke, to be built on it. However, the availability of very high resolution (sub metre) has brought out a unique and challenging set of problems to tame. It seems only inevitable, with satellite imagery being the source of very large datasets and Machine Learning needing large volumes of data, this would be the perfect ecosystem for solving tasks at scale, more accurately and quicker than with trained human eyes. These include object detection, identification and counting and change detection.
One such problem that we have encountered and that appears in scale is establishing progress of urban construction development on a local, regional and global perspective. Simply put, the challenge is to look at two images of an urban location taken at two different dates and estimate whether there has been a progress in the construction or demolition work or not. Traditionally, to find the difference between two images, two approaches have been used. The first involves the image processing or Computer Vision approach. One of this is a simple method called Image Differencing.
Image differencing is a primitive and rudimentary process but is still in use. Other variants of this method are in vogue too. This is a three-step process as follows:
· Register the image so that corresponding points in the image correspond to the same physical location on ground; an alignment of sorts.
· Subtract one image from the other. The result would be the pixel-wise difference between the two images.
· Evaluate this difference image to a threshold to determine whether there is a change or not.
The second approach is a parametric method involving histograms. If the images are balanced and have minor variations due to the sun and satellite angles, images that have no difference between them would have nearly identical histograms. By thresholding the difference between the histograms, a prediction can be made. Other variants of this approach include segmentation as a pre-processing step. Segmentation reduces fine clutter that is in abundance in very high resolution imagery.
Notwithstanding the fine detail that accompanies very high resolution imagery, there are aspects of satellite image acquisition that complicates this problem. Even with images at the same zoom and scale, the satellite position with respect to the area, dictated by the ‘Off Nadir Angle’ and ‘Azimuth’ result in differing perspectives of the same image; consider it the equivalent of a full front picture as compared to a profile picture of a face. The second aspect involves the position of the sun with respect to the area at the time of acquisition dictated by the ‘Sun Azimuth’ and ‘Sun Elevation’. Differing positions of the sun can lead to varied shadow dimensions and positions. These four parameters also explain why images taken on the same day may cause the illusion that there is change between the images. Moreover, there is no clear and simple way to compensate or correct for these idiosyncrasies.
There is too much data. There is too little data. I am sure many have oscillating opinions between these two extremes when solving data driven problems. For example, at zoom 18, at a scale of 0.6m to a pixel, about 69 billion tiles of 256 x 256 pixels make up the world. Considering that land takes up about 30% of the Earth, it still is approximately 20 billion tiles. Even when the region of interest is localised to a region of the size of a city, acquisitions from multiple vendors and satellites make the numbers staggering. And this keeps growing. This is one end of the spectrum where “There is too much data”.
The satellite and sun angles are influential control factors and much of it is dictated by the way the satellite is exploited. Differing angles for two temporal images can cause complete havoc with solutions based on Image Differencing and Histogram Comparison. The best way to build a dataset is to choose images within a certain limit of these angles. However, when images are selected based on very near sun and satellite angles to offset their effect, we end up with a sparse dataset. Add to this the peculiarities arising from geography and we can end up with disparate images when taken from say Qatar and Indonesia. Rooftops for example, are of a different construction and configuration in Lisbon, Portugal and Nyeri, Kenya. When trying to build a dataset for solving a problem, the problems above therefore leads to a situation we reach the “There is too little data” end of the swing.
The way ahead to solving this specific problem is therefore to build a solution that is impervious (to a large extent) to changes caused by these angles and yet determine whether, given two temporal images, there is change between them. One reliable, non-parametric method to solve this problem is Machine Learning, specifically Convolutional Neural Networks or CNN in short. CNNs work by self-learning the features that are common to each class of input data. Then, at the time of reckoning, when an input image is fed, it makes a prediction from the features it has learnt. But even Machine Learning, is no match for the wide distribution of data when it comes to this problem: angles of Sun and acquisition, differing shapes, orientation and sizes of construction sites all contribute to wide disparity even within the same class. To improve the predictions another improvement is to build an ensemble of differing models. An ensemble of parametric and non-parametric models particularly works well for the bi-temporal, co-registered, very high resolution satellite imagery to detect change.
The case for letting a machine predict is of value only if it can predict better than a human in accuracy and / or in time. The use case of detecting progress in construction sites is difficult for humans especially from imagery taken from the top. Progress immediately apparent when viewed at site might not be captured in the image at all. Material and vehicles moved around might not imply progress at all. This implies that the dataset for training should be chosen carefully to reflect the use case. Given the diversity in the images of interest, this is a daunting and time consuming task. Once the dataset is in place, they are used to build models that learn to differentiate between ‘Change’ and ‘No Change’. At Bird.i, we firmly believe that the black magic complexity and internals should be insulated from the user who should just be concerned with getting results. The models are therefore along with image pre-processing wrapped in a micro service. This way, all that the user is concerned with is requesting a prediction for a pair of coordinates and date range. It is also important for us to let the user know about the accuracy of the prediction. We do this be tagging along a Measure of Confidence with the prediction. A higher measure of confidence simply implies that the models are surer of the prediction, as should the user’s confidence in the prediction be.
An overlap of domain expertise, computer science and Math/Stats makes a great Data Scientist. At Bird.i, we transformed that union of skills to a more abstract level. Our skills and work lies in the intersection of cutting edge technology in Machine Learning, Observation Imagery, Big Data and Computer Vision. It is interesting, challenging and absolutely fast paced. We believe in the adage that nothing is impossible and love solving complex problems; identifying cats and dogs in pictures are passé. The technology behind our products reflects the same complexity. Get in touch with us if you want to be an early adopter of our Construction Progress Tracker or the New Construction Progress Tracker products. Alternately, if you are facing a problem that involves Observation Imagery and scale, we can solve it for you.
