Boat detection with multi-region-growing method in satellite images

9 min readOct 22, 2021

Image satellite is one of the most incredible things we can have in today’s society. It allows us to measure the effects of climate change, help us in agriculture, and have many other applications like google maps. Technology has evolved into a point that allows anyone to observe the earth from space in our phones. That is huge. In this article, we will use satellite images to do boat detection and do a quick summary of the history of satellite images and the involvement of machine learning in the topic.

Before we go more further in detail, do this method return state-of-the-art results? Nah, I doubt it. But it is pretty cool to implement, lets go.

Introduction

This article will apply a multi-region-grow algorithm to obtain all the bulk carriers and container ships of an image. This method will not require supervised learning techniques or big datasets to implement, allowing us to work with non-label images. We will use Kaggle’s “Ships in Satellite Imagery” dataset that contains satellite images from San Francisco Bay from Planet.

This method can automatize multiple applications, from counting the number of ships passing strategic regions to tracking specific vessels and their direction.

Background

Satellite technology has improved much since the space race and the cold war. One of these examples is the Copernicus Program, carried out in collaboration with the European Union and the European Space Agency (ESA). They monitored the earth to study the climate, borders control, and other applications. We can also find a labeled dataset [Helber, Patrick (2019)] (Eurosat) taken by a Sentinel-2 (Satellite from the Copernicus Program) available in TensorFlow datasets. Other interesting datasets can be found in Earth Engine Data Catalog.

The images we obtain from the satellite depend entirely on its equipment. There are two main types of sensors used [Parman, Rhyma(2016); Aroma, Jenice (2016)]:

Active sensors: (radar) they can work with clouds or at night: TerraSAR-X, RADARSAT…
Passive sensors: (optical systems) use cameras to take images from the planet: Ikonos, Landsat…

We can find multiple approaches in the literature for different tasks in satellite images, Xiang Zhu, Xiao (2017) shows a good review of different machine learning methods used in the field.

Method

Boats, in general, have a well-defined shape depending on their class. Bulk and containers ships are no exception. It allows us to use a more direct method without using deep learning models. In this way, we do not need to generate a dataset or use a supervised approach to solve this task. Our method can also be used to detect other sea objects without significant changes or multiple train samples. The code can be found in this notebook. We divided the algorithm into three steps:

Sea segmentation: where we segment the water of the image, creating a binary image.
Multi-region-growing: used to cluster the different shapes in the binary image.
Boat detection: analyze the shape of the different clusters to detect if they are boats or not. This is the only part of the algorithm that needs modification to detect other sea objects.

Sea segmentation

The first we need to do is remove the sea and create binary images. It will allow us to separate each shape into a different set and obtain all the possible candidates. There are multiple ways to segment the sea in machine learning. We are going to make an approach that requires the minimum amount of data to train. To achieve this, we will use the Likelihood of each pixel to decide if it is a part of the sea.

Each pixel is represented with three values that describe the color. We can use this as three coordinates in Euclidean Space. It allows us to cluster pixels by their proximity. The more close the pixels are, the more similar their colors are. If we select a group of water pixels, we can define a Gaussian distribution. In this way, with a small train set, we can segment full images by calculating the Likelihood of each pixel to be inside the distribution.

Initializing our distribution from a sample with TensorFlow Probability.

Calculating the Likelihood of each pixel can be slow. Therefore, we use the GPU. We achieve this by creating a distribution from TensorFlow Probability (TFP). A reasonably recent library allows us to work with probabilistic models in the GPU and combine them with other Deep learning methods. The output of this method will be a binary image. However, it will have some noise. To remove this noise, we can do some morphological operations with OpenCV. This allows us to obtain a more clean segmentation.

Multi-region-growing

The use of region growing for satellite images is not new. We can find some literature with this method in image segmentation [Rodríguez-Cuenca, Borja (2012)]. The standard approach is first to use PCA to reduce the number of channels and then apply a region-growing method with a threshold to do the segmentation. In our case, we do not need a threshold or PCA because we already make a clear separation with our segmented image.

Our multi-region-growing method is divided in four steps:

Set of images: We divide the image into a grid to separate it into a set of images. This step is important because region growing can be slow, and if we work with recursive functions, we will not be able to process the whole image (we will get the error: maximum recursion depth exceeded). Splitting the image allows us to parallelize the algorithm, be more memory friendly, and avoid recursion depth errors.
Region-growing: For each image in the set, we select a random non-zero pixel and add a label. After that, we go recursively to all non-zero neighbors until there are no more. We repeat this process with a different label until there are no more non-zero pixels unlabeled. Once we finish with all the images on the set, we gather all of them in their original positions. If we parallelize this method, we must never repeat the same label in different threads.

Filter: The bigger boat in history was the Seawise Giant, with 458.45 m of length and no longer in service. This means that any labels larger than that can be removed. In this way, we can filter the more prominent groups of pixels that can slow down our pipeline and know that they will not be boats.
Post processing: At this point of the algorithm, we have multiple labeled shapes. Now we have to join them between the images of the set. There are numerous methods to do it. We are going to apply a fast and straightforward approach. The main idea is that we iterate over the set of images in their position on the grid. If two pixels of different images are touching, we add all the pixels of one label to the other. We perform four iterations, ranging: right, up, left, and down. The method is not perfect. It does not always work with complex shapes, as shown in the image below (case 1). However, shapes as case 2 that look like boats work well.

If we compare the original image and the outputs of the previous steps, we can observe that what we are essentially doing is reducing the amount of non-relevant information. We convert a full-color image into a small number of candidates. This allows us to analyze each candidate individually and classified as boats or not boats.

Boat detection

We take the output of Multi-region-growing, and we consider every labeled shape as a possible candidate. Now the last step is using a function C(x) -> [0,1] that will classify each candidate into boat or not boat. This function will use only the shape, and we can change it to classify something else instead without changing anything else in the whole method.

Since we will only use the ship shape, we must have all candidates with the same orientations to simplify the task. We can do this by first obtaining the original direction of each candidate. A method to do this is using Principal Component Analysis (PCA) to get the principal axes of the shape. Because there are 2D objects, the axis will be (x, y), and y will be the longest. We will not enter more in detail in this approach, but we can find more information in Pablo G.Tahoces (2021), where they used it for medical images and explained it in detail.

Once we have x and y, we want to rotate the shape with the y and x in the same direction as the image’s (x’, y’). To do this, we use a rotation matrix with the angle between y and y’ and recalculate the position of every pixel. The rotation matrix is a linear transformation that rotates the image concerning the coordinates (0,0), so it is essential that before we do it, we define the center of the image as (0,0). Another important detail is that it is recommended to use backward mapping with an interpolation function to maintain the original shape in the transformation. We can use OpenCV for this. In my case, I used an old repository I made. If you want to know more about these concepts, I recommend reading Digital image processing (2004).

Results

To test the results, we used a simple classification function that looks at the dimensions of the boat. Because this project was done for fun and as a concept, there are no quantity results, only quality results.

As shown in the image below, the region growing method removes a big part of the image that we need to analyze to obtain the boats. However, a classification function is still necessary.

The results are promising. The region growing method obtain all the bulk carriers and container ships in the images that are in the sea. But a classification by shape is still necessary. In our case, we use the width and the length of the boat for this task. As we can observe in the images below, we manage to detect bulk carriers and container ships, filtering other smaller vessels and artifacts. But more work is needed in the classification function.

Conclusion

The main conclusion is that our method can work in boat detection from satellite images without big datasets or label data. This approach can have multiple applications, as we mention in the introduction. However, more rigorous testing and optimization are needed it.

In any case, it was fun to implement, and feel free to play around with the code. Please add a comment if you have any questions, and I hope you enjoyed the post.