Area Monitoring — Pixel-level Mowing Marker

When an object-based approach is not good enough

Matic Lubej

Follow

Published in

Sentinel Hub Blog

9 min readNov 4, 2021

--

A time-lapse of a parcel where mowing is performed in partial steps. This lead to issues when performing an object-based detection of mowing events.

This is a multi-part series about machine learning and EO data supporting Common Agriculture Policy. Find information about related blog posts at the bottom.

The detection of mowing events is one of the most important aspects of area monitoring and can represent the majority of agricultural activity in countries with a lot of grassland and meadows. In such cases, a good and reliable mowing detection algorithm is crucial for aiding agencies with decision making.

In our previous mowing-related blog post we have described the pipeline for detecting mowing events with an object-based approach, meaning that the algorithm was performing on spatially averaged time-series for a particular meadow-like parcel.

Area Monitoring — Mowing Marker

Is it possible to detect grass being mowed from 792 km away?

medium.com

However, the use of an object-based approach only makes sense when certain assumptions hold, and in our experience, this is perfectly fine for about 90 % of cases, while for the remaining part these assumptions start to break.

More concretely, the object-based approach only makes sense for parcels that are homogeneous, meaning that you would expect meadow-like parcels to contain grass only — there shouldn’t be any trees, crops, houses, or forests on that parcel. Additionally, even if the entirety of the parcel is covered with grass, it is still necessary that also the agricultural activity on the parcel is uniform. These assumptions quickly start to break as soon as the size of parcels starts growing (you can’t expect farmers to process them in a single day!). Below you can find examples when an object-based approach is not the best way to proceed forward.

An example of a field with agricultural activity only on a part of the parcel.

An example of a large parcel at a higher altitude with a lot of mixing from bare surfaces and rocks.

What to do in such cases? Should you fix the parcel geometry and repeat the object-based approach? This can take time and solves only part of the problem, while the solution doesn’t scale when there are thousands (or more) of such cases.

Another possibility is to apply a pixel-based approach. If you’re lucky (or smart), you have designed an algorithm that works on any kind of time series, be it an object- or pixel-based one. If that is the case, you can simply run the process on different inputs and that’s it. In the particular situation of our mowing marker, we have translated the detection algorithm in question to Javascript and pasted it into our Sentinel Hub Evalscript. This way the service does all the work for us and we download only the important stuff!

A proof-of-principle Sentinel Hub Evalscript for detecting mowing events on the pixel level

If you’re interested in the intricate workings of the mowing algorithm itself, be sure to check the object-based mowing detection blog post, as the algorithm is the same and will not be additionally described here.

The Pixel-level Approach

Let’s now take a deeper look at how the pixel-based approach is performed and the benefits it brings. Imagine we ran the object-based mowing detection on the same parcel as in the top example above and obtained the following result.

Mowing detection: NDVI series for an example parcel where no mowing event is detected.

Let’s see… Firstly, we have enough valid Sentinel-2 observations after filtering out the clouds. Secondly, the NDVI series in the period where mowing is expected is relatively flat. One might conclude that this parcel has probably not been mowed in 2019. However, we know for a fact that this parcel has been mowed based on the image chips above, so why doesn’t it show up here? Looking at the satellite imagery again, we see that when the parcel is mowed, it is mowed only partially. Since we work with spatially averaged time series, this means that the signals from parts of the field, where mowing does occur, mix with the signals from the parts where it doesn’t. This is not something you want when you’re using an object-based approach!

An additional example of the RGB time series for the mentioned parcel. In an object-based approach, the signals from the parts, where mowing does occur, mix with the parts where it doesn’t, which produces misleading outcomes.

Since we can’t expect experts to check the satellite imagery for each parcel when a situation like this comes up, it makes sense to go the extra mile and invest in a search for an alternate solution to cover the remaining cases.

As mentioned above, one of the possibilities is the pixel-level approach. We can start by looking at the NDVI series of all the pixels separately.

NDVI time series for the same parcel as above, but drawn separately for each pixel.

The plot above tells a different story altogether! We can easily see that the mowing activity is performed in several (namely two) steps, where parts of the field are mowed at different points in time. The fluctuations are very large, but the most interesting part is that the NDVI series coming from different parts of the field produce a relatively flat averaged NDVI series, leading us to make false conclusions in the end. Sneaky!

With such rich information at hand, we are able to extract many useful things. We can, for example, mark the parts of the pixel-level time series when mowing events are detected, or we can count the pixels for each date when it occurs in order to determine the relative coverage of the parcel being mowed as a function of time.

Similar image as above, but with additional information shown, such as parts of the pixel-level time-series when mowing was detected (green line) or the relative coverage of the parcel being mowed as a function of time (bottom subplot, red dashed line). The true color images correspond to the timestamps of the marked partial mowing events.

We see that at some point towards the end of June more than 70 % of the parcel was mowed — which is far from negligible — but since the two major portions of the parcel were mowed at different times, the average of the time series resulted in a flat signal and we weren’t able to pick the events up.

As a final result, we are also able to plot how many times mowing was detected for each pixel, and we can spatially arrange them so that they represent the original geometry of the parcel.

Pixel-level mowing shown in a counts-per-pixel style. Mowing has been performed on the majority of pixels, in some cases more than once.

The image above shows data only for the pixels inside of the parcel geometry. If we count the pixels with at least one mowing event, we discover that roughly 90 % (30 %) of the whole area has been mowed at least once (twice) during the considered window of time.

Evaluation

Simply running the pixel-based algorithm is not enough. In how many pixels do we need to detect a mowing event to say that mowing occurs? Should it be an absolute or a relative value? What if pixel-level mowing detection is randomly distributed over the parcel? These are all valid questions that need to be addressed.

Since we are dealing with both large and small parcels, it makes sense to include a metric of relative coverage. However, since we are detecting mowing events on Sentinel-2 pixels, it also means that we should apply some minimal absolute threshold on the number of pixels where mowing was detected. With an appropriate rule based on the description above, we should be able to ignore parcels with low relative coverage and a low absolute count of pixel-level mowing events.

To not be affected by random pixel activations, we have opted for an evaluation based on the smallest area of spatially connected pixels with detected mowing events. An example of such a connected area is shown below on the right, with its counterpart on the left. This area serves as the metric for the absolute count of pixel-level mowing events.

A parcel with a lot of random pixel-level mowing events (left) and a pixel-level mowing events connected in a bunch (right). The example on the left should probably not be considered as a mowing event, while the example on the right should.

After the metrics for the evaluation of pixel-level mowing events were established, a semi-empirical process of defining sensible values was performed due to a lack of a good reference dataset. We have converged to the following definitions of valid and invalid pixel-level mowing events:

The conditional criteria used for the evaluation of valid or invalid pixel-level mowing events. The N_conn is the number of pixels of the smallest connected area where pixel-level mowing was detected, the f_cov is the relative coverage of pixel-level mowing events, the ∧ is the logical “and” operation, and ∨ is the logical “or” operator.

For a valid mowing event on the pixel level, we require a minimum count of connected pixel-level mowing events of 3 and their relative coverage exceeding 0.2 for small parcels or a minimum count of connected pixel-level mowing events of 8 for large parcels. Below you can see the 2D distribution of the defined metrics. The distribution is split into a valid and an invalid region via a red line which represents the conditional border described above.

The pictorial representation of the conditional criteria above. For a valid mowing event on a pixel level, we require a minimum count of connected pixel-level mowing events of 3 and their relative coverage exceeding 0.2 for small parcels or a minimum count of connected pixel-level mowing events of 8 for large parcels.

Unfortunately, no reference datasets for evaluating the pixel-based approach were available to us. However, we did have a general-purpose reference dataset used for the evaluation of the object-based mowing detection. Since the latter contained parcels that were also partially mowed, we have included the results from the pixel-level approach to see if any improvement can be made.

The results below show two confusion matrices (CM), where the one on the left (right) represents results without (with) taking pixel-level mowing events into account. As explained throughout this blog post, the object-based results might contain false negative examples (FN, bottom left cell) with only partially mowed parcels where the algorithm failed to detect a mowing event.

By using the pixel-based approach we are now able to pick up such cases and tag them as true positives (TP, bottom right cell), which leads to a decrease of FN and an increase of TP. Since there is no such thing as a free lunch, this also decreases the count of true negatives (TN, top left cell) and increases the count of false positives (FP, top right cell). This only means that the tuning of your algorithm should be governed by how false positive/negative cases are treated from the business rules’ point of view.

The confusion matrix representing the evaluation of an object-based mowing detection algorithm (left), and the version where also the pixel-base results were taken into account (right).

Conclusion

We have shown how to successfully detect mowing events with a pixel-based approach for a parcel that would have otherwise been falsely accused of having no agricultural activity at all.

Such an approach brings value through tackling the last 10 % of cases that don’t respect the assumptions necessary for an object-based approach. However, at the expense of additional computational overhead.