Area Monitoring — Pixel-level Mowing Marker
When an object-based approach is not good enough
This is a multi-part series about machine learning and EO data supporting Common Agriculture Policy. Find information about related blog posts at the bottom.
The detection of mowing events is one of the most important aspects of area monitoring and can represent the majority of agricultural activity in countries with a lot of grassland and meadows. In such cases, a good and reliable mowing detection algorithm is crucial for aiding agencies with decision making.
In our previous mowing-related blog post we have described the pipeline for detecting mowing events with an object-based approach, meaning that the algorithm was performing on spatially averaged time-series for a particular meadow-like parcel.
However, the use of an object-based approach only makes sense when certain assumptions hold, and in our experience, this is perfectly fine for about 90 % of cases, while for the remaining part these assumptions start to break.
More concretely, the object-based approach only makes sense for parcels that are homogeneous, meaning that you would expect meadow-like parcels to contain grass only — there shouldn’t be any trees, crops, houses, or forests on that parcel. Additionally, even if the entirety of the parcel is covered with grass, it is still necessary that also the agricultural activity on the parcel is uniform. These assumptions quickly start to break as soon as the size of parcels starts growing (you can’t expect farmers to process them in a single day!). Below you can find examples when an object-based approach is not the best way to proceed forward.
What to do in such cases? Should you fix the parcel geometry and repeat the object-based approach? This can take time and solves only part of the problem, while the solution doesn’t scale when there are thousands (or more) of such cases.
Another possibility is to apply a pixel-based approach. If you’re lucky (or smart), you have designed an algorithm that works on any kind of time series, be it an object- or pixel-based one. If that is the case, you can simply run the process on different inputs and that’s it. In the particular situation of our mowing marker, we have translated the detection algorithm in question to Javascript and pasted it into our Sentinel Hub Evalscript. This way the service does all the work for us and we download only the important stuff!
If you’re interested in the intricate workings of the mowing algorithm itself, be sure to check the object-based mowing detection blog post, as the algorithm is the same and will not be additionally described here.
The Pixel-level Approach
Let’s now take a deeper look at how the pixel-based approach is performed and the benefits it brings. Imagine we ran the object-based mowing detection on the same parcel as in the top example above and obtained the following result.
Let’s see… Firstly, we have enough valid Sentinel-2 observations after filtering out the clouds. Secondly, the NDVI series in the period where mowing is expected is relatively flat. One might conclude that this parcel has probably not been mowed in 2019. However, we know for a fact that this parcel has been mowed based on the image chips above, so why doesn’t it show up here? Looking at the satellite imagery again, we see that when the parcel is mowed, it is mowed only partially. Since we work with spatially averaged time series, this means that the signals from parts of the field, where mowing does occur, mix with the signals from the parts where it doesn’t. This is not something you want when you’re using an object-based approach!
Since we can’t expect experts to check the satellite imagery for each parcel when a situation like this comes up, it makes sense to go the extra mile and invest in a search for an alternate solution to cover the remaining cases.
As mentioned above, one of the possibilities is the pixel-level approach. We can start by looking at the NDVI series of all the pixels separately.
The plot above tells a different story altogether! We can easily see that the mowing activity is performed in several (namely two) steps, where parts of the field are mowed at different points in time. The fluctuations are very large, but the most interesting part is that the NDVI series coming from different parts of the field produce a relatively flat averaged NDVI series, leading us to make false conclusions in the end. Sneaky!
With such rich information at hand, we are able to extract many useful things. We can, for example, mark the parts of the pixel-level time series when mowing events are detected, or we can count the pixels for each date when it occurs in order to determine the relative coverage of the parcel being mowed as a function of time.
We see that at some point towards the end of June more than 70 % of the parcel was mowed — which is far from negligible — but since the two major portions of the parcel were mowed at different times, the average of the time series resulted in a flat signal and we weren’t able to pick the events up.
As a final result, we are also able to plot how many times mowing was detected for each pixel, and we can spatially arrange them so that they represent the original geometry of the parcel.
The image above shows data only for the pixels inside of the parcel geometry. If we count the pixels with at least one mowing event, we discover that roughly 90 % (30 %) of the whole area has been mowed at least once (twice) during the considered window of time.
Evaluation
Simply running the pixel-based algorithm is not enough. In how many pixels do we need to detect a mowing event to say that mowing occurs? Should it be an absolute or a relative value? What if pixel-level mowing detection is randomly distributed over the parcel? These are all valid questions that need to be addressed.
Since we are dealing with both large and small parcels, it makes sense to include a metric of relative coverage. However, since we are detecting mowing events on Sentinel-2 pixels, it also means that we should apply some minimal absolute threshold on the number of pixels where mowing was detected. With an appropriate rule based on the description above, we should be able to ignore parcels with low relative coverage and a low absolute count of pixel-level mowing events.
To not be affected by random pixel activations, we have opted for an evaluation based on the smallest area of spatially connected pixels with detected mowing events. An example of such a connected area is shown below on the right, with its counterpart on the left. This area serves as the metric for the absolute count of pixel-level mowing events.
After the metrics for the evaluation of pixel-level mowing events were established, a semi-empirical process of defining sensible values was performed due to a lack of a good reference dataset. We have converged to the following definitions of valid and invalid pixel-level mowing events:
For a valid mowing event on the pixel level, we require a minimum count of connected pixel-level mowing events of 3 and their relative coverage exceeding 0.2 for small parcels or a minimum count of connected pixel-level mowing events of 8 for large parcels. Below you can see the 2D distribution of the defined metrics. The distribution is split into a valid and an invalid region via a red line which represents the conditional border described above.
Unfortunately, no reference datasets for evaluating the pixel-based approach were available to us. However, we did have a general-purpose reference dataset used for the evaluation of the object-based mowing detection. Since the latter contained parcels that were also partially mowed, we have included the results from the pixel-level approach to see if any improvement can be made.
The results below show two confusion matrices (CM), where the one on the left (right) represents results without (with) taking pixel-level mowing events into account. As explained throughout this blog post, the object-based results might contain false negative examples (FN, bottom left cell) with only partially mowed parcels where the algorithm failed to detect a mowing event.
By using the pixel-based approach we are now able to pick up such cases and tag them as true positives (TP, bottom right cell), which leads to a decrease of FN and an increase of TP. Since there is no such thing as a free lunch, this also decreases the count of true negatives (TN, top left cell) and increases the count of false positives (FP, top right cell). This only means that the tuning of your algorithm should be governed by how false positive/negative cases are treated from the business rules’ point of view.
Conclusion
We have shown how to successfully detect mowing events with a pixel-based approach for a parcel that would have otherwise been falsely accused of having no agricultural activity at all.
Such an approach brings value through tackling the last 10 % of cases that don’t respect the assumptions necessary for an object-based approach. However, at the expense of additional computational overhead.
Check the Area Monitoring documentation for more information.
Further reading
This post is one of the series of blogs related to our work in Area Monitoring. We have decided to openly share our knowledge on this subject as we believe that discussion and comparison of approaches are required among all the groups involved in it. We would welcome any kind of feedback, ideas and lessons learned. For those willing to do it publicly, we are happy to host them at this place.
The content:
- High-Level Concept
- Data Handling
- Outlier detection
- Identifying built-up areas
- Similarity Score
- Bare Soil Marker
- Mowing Marker
- Pixel-level Mowing Marker (this post)
- Crop Type Marker
- Homogeneity Marker
- Parcel Boundary Detection
- Land Cover Classification (still to come)
- Minimum Agriculture Activity (still to come)
- Combining the Markers into Decisions
- The Challenge of Small Parcels
- The value of super resolution — real world use case
- Traffic Light System
- Expert Judgement Application
- Agricultural Activity Throughout the Year
Our research in this field is kindly supported, in grants and knowhow, by our cooperation in Horizon 2020 (Perceptive Sentinel, NIVA, Dione) and ESA projects (Sen4CAP).
