MAFAT Radar Challenge — There and Back Again
This is the 1st article in our MAFAT competition series, where we’ll give an in-depth look at the different aspects of the challenge and our approach to it.
Data Science competitions are a great way to participate in the wider domain of Data Science and work on novel problems and datasets. Various companies, organizations and governments run competitions on a wide range of problems, and in 2019 the Israeli Ministry of Defense “Directorate of Defense Research & Development” (DDR&D) got in on the action with the launch of the MAFAT challenge series. The second competition kicked off on July 15th, 2020, and ran until October 15th, with more than 1000 participants and 4300 submissions over the course of 3 months.
As recent graduates from the YDATA data science program we wanted to try our hand at working with a completely unknown dataset in a totally novel domain for us, and hopefully gain new knowledge and sharpen our skills. We landed in a respectable 24th place, and in this series of posts we’ll give an in-depth look at our process. Although we didn’t win, we hope that the knowledge and experience we gained during the challenge can be beneficial to others.
Summary
The target of the challenge was — to quote the official website:
The participants’ goal is to classify segments of radar tracks of humans or animals using the I/Q signal matrix as an input. The task at hand is a binary classification task; the tracked objects are either humans or animals. The data is real-world data, gathered from diverse geographical locations, different times, sensors, and qualities (high- and low-signal to noise ratio — SNR).
The competition was divided into two parts, with different conditions:
- The public phase with unlimited submissions, evaluated on a subset of the Public Test Set
- The Private phase where teams were limited to 2 submissions and where models were evaluated on completely new and unseen data.
The ultimate goal was to produce a model that performs as well as possible on the Private Test Set — the performance on the Public test was not relevant to the final leaderboard.
The official challenge website has a lot of detailed information about the background and specifics of the challenge.
Competition Data
The competition goal was to classify radar segment data as humans or animals, using ROC AUC as the metric.
The dataset consisted of:
- 6600 training segments
- 106 test segment for the Public phase; 248 segments for the Private phase
- Auxiliary datasets:
- 50k Synthetic Low SNR segments
- 31k background-noise segments
- 49k controlled human segments
A segment entry consisted of an I/Q matrix that represented a fixed time-window of a doppler radar reading, a vector describing the center of mass during the same time-frame, and metadata about the radar reading.
A more in-depth look at the dataset is forthcoming in a separate post.
Data Processing & Augmentations
Our focus was on generating more training examples using various augmentation techniques, specifically:
- Flipping the IQ matrices horizontally and vertically.
- Stitching together adjacent radar segments and creating new readings by shifting along the X-axis, or “long-time” axis.
We hit significant memory and compute limitations when trying to utilize these augmentations to the fullest, which slowed down the speed of experimentation.
This necessitated the writing of a streaming dataset which was implemented using the IterableDataset class in Pytorch.
To read hezi hershkovitz’s article explanation of how we implemented the augmentation: https://medium.com/definitely-not-sota-but-we-do-our-best/augmenting-radar-data-with-shifts-flips-81beb857e705
To read Adam Cohn’s article on creating a more efficient and faster data pipeline: https://medium.com/definitely-not-sota-but-we-do-our-best/the-mafat-dataset-a-closer-look-e567773071bc
Data Transformations & Representations
We experimented with using different visual representations of the radar data as inputs to the models, specifically spectrograms and scalograms.
Although theoretically promising, using scalograms as input to Neural Networks did not yield any improvements in model performance.
To read Hernan Rosenblum’s article on how to use scalograms and spectograms when working with signal data: https://medium.com/definitely-not-sota-but-we-do-our-best/spectrograms-and-scalograms-visualizing-signal-data-555d2fc2cf64
Model Architectures
We tried using several different Neural Network architectures, from basic Alexnet CNN-style algorithms to Temporal Convolutional Networks.
Surprisingly, the Alexnet architecture and a TCN model achieved very similar results using Spectrograms with the doppler burst vector overlaid.
For our final submission in the Private phase we used Bagging to train a simple Logistic Regression model on the predictions of our best-performing CNN and TCN model.
To read my post regarding different architectures and ensembling methods: https://medium.com/definitely-not-sota-but-we-do-our-best/radar-data-architectures-and-ensembling-e1dbe1179147
Final Results
Our models suffered from overfitting, as the training and validation AUC was 95–99%. This is likely because we used segments from the same track in both the training data and the validation data.
However, we were very pleased that our models maintained a very similar level of performance on both the public and the private test sets.
The best result we achieved on the Public test was with an Alexnet architecture that scored a 80.3% ROC AUC, which meant we ended that phase in 78th place out of 205 teams with submissions.
We scored 79.35% ROC AUC on the Private test phase and ended the competition in 24th place out of 47 teams with submissions.
Conclusion & Acknowledgements
We’d like to extend a huge thank you to Dmitri K from Yandex who was an invaluable mentor and resource during these three months, and to Y-Data and Kostya Kilimnik for encouraging us to participate, providing encouragement during the process, and above all for putting Dimitri at our disposal 🙂
