Data Sciencing Signals — From a Challenge on Radar Tracks to How We Do It at Augury
As a data scientist, I enjoy participating in data science challenges. It’s basically a playground where I can play with my toys without worrying about infrastructure, data engineering, deployment or tests. What makes it even better is that sometimes it’s also beneficial to my day to day job.
A few weeks before I started working as a data scientist at Augury, while working as a data scientist at a gaming company, I participated in a challenge where we had to classify radar signal tracks.
At Augury, we classify acceleration, magnetics and temperature signal tracks. Although the signals are different from the signals in the challenge, and even though we didn’t finish in the top three of this challenge (no prize for us…), I did receive some helpful ideas that can improve Augury’s AI products.
In this post I’ll share my solution to the challenge and how part of it can be implemented at Augury. I’ll also explain how my experience at Augury has given me ideas on how we could have handled this challenge better.
But before all of that, let’s understand the challenge…
MAFAT Radar Challenge
The challenge was set by MAFAT (Israel Ministry of Defence — Directorate of Defence Research & Development), and the goal was to classify whether a radar signal segment represents a human or animal. The radar signals were recorded by doppler-pulse radars and were given as
IQ matrices (In-phase and Quadrature — a convention to express the sine and cosine of alternating signals) of complex values, where the I is the real part and the Q is the imaginary part. Each segment was represented with an IQ matrix with the size of 32x128.
The x-axis represents the pulse transmission time,“slow time”. Meaning each sample contains 32 pulses.
The y-axis represents the reception time of signals of one pulse, “fast time”/“velocity”, divided into 128 equal sized bins.
For example, for Pulse Repetition Interval (PRI) of 128 ms, each Y-axis is a bin of 1 ms. For a pulse sent in t(n) and a signal received in t(n+m), where 0<m<=128, the signal is set in the “m” bin of pulse n.
In addition, each IQ matrix had its “doppler burst” — a time vector with the same length of the X-axis — defining which velocity cell in the column contains the tracked object’s center-of-mass.
There was also some metadata containing the estimated Signal-to-Noise Ratio (SNR), geolocation type, specific radar ID, and for the training dataset there was also the label (human\animal).
To sum up, we had the IQ matrices, their doppler burst and some metadata.That’s it.
So how did we plan to tackle this challenge?
Our Plans
I’m not going to elaborate on the “regular data science stuff”, such as splitting the dataset to train and validation with no leakage, data balancing and basic augmentation techniques. Let’s talk about the interesting stuff:
We planned to tackle this challenge using various angles, and eventually compare them all.
Tabular models on calculated features
In this part, we calculated features based mainly on the doppler burst and its “halo” — the values that wrap the values of the doppler burst in the IQ matrices. Statistical features, such as the mean\median\std of the doppler bursts’ values, and the same for the bursts “halo”.
We then had a variety of features for each segment, and we used ML models aimed at tabular data.
No features were domain-related, due to a lack of domain knowledge. This, I believe, was one of our biggest drawbacks and I will elaborate on that later on.
Image models on IQ matrices as images
We had an idea to refer to the IQ matrices as images and use common practices for image classification (Custom made CNN, pre-trained ResNet, etc.)
Image models on pre-processed IQ matrices as images
Here, our idea was the same as the previous, with a preliminary step of pre-processing the IQ matrices using the Fourier transform. As this preprocessing made the IQ matrices look more informative in our eyes, we believed it would be more informative to the models, so we had the highest expectations from this solution path.
Using the above three approaches yielded our solution. After hyperparameters tuning, comparing models and ensemble techniques, lets deep dive into the best solution we achieved.
Our Winning Solution
The NN architecture that referred to the IQ matrices as images, worked much better than the other solutions, but only if we pre-processed the matrix columns using the Fourier transform. Meaning, from our plans above, applying image models on IQ matrices didn’t work very well, while preprocessing the IQ matrices and then applying image models worked great. As for the tabular models technique, it also produced some good results.
Eventually we converged to an architecture of 4 different NN models, where the input is the IQ matrices after processing them with the Fourier transform along the y axes, and one non-NN model on the calculated features, all merged into a stacking ensemble.
- Pre-trained ResNet34, where we unfreeze the top 200 layers [input = preprocessed IQ matrices]
- “Criss-cross” 1d conv net, where we had 1d horizontal and vertical convolution layers with 2 fully connected layers at the end [input = preprocessed IQ matrices]
- Fully 2D convolution network of 5 layers, with 2 fully connected at the end [input = preprocessed IQ matrices]
- 1D conv + LSTN along the peaks of the y-axis (the “doppler burst”) [input = preprocessed IQ matrices]
- Ensemble of Random Forest, XGBoost and MLP [input = statistical features]
Each deep NN based model from above had a 64 neuron fully connected layer as the second last. This layer was used for the final ensemble.
The output of the small ensemble of Random Forest + XGB + MLP went through embedding, so that we would also have a 64-length vector as output to use for the final ensemble.
These 64 vectors were the input of the second step, where we used a Transformer encoder, followed by a fully connected layer.
After tuning this architecture, we achieved an ~0.84 roc-auc score on the first phase test set, which placed us 50th out of ~230, and an ~0.8 roc-auc score on the second phase test set, which placed us 25th out of ~50.
What does this all have to do with Augury, you ask? Well.. great question
Augury’s Data Science Structure in a Nutshell
At Augury we listen to machines. We do so by recording 4 seconds (=session) every hour of the machines’ vibrations, magnetic field and temperature.
Each acceleration and magnetic session is processed using DSP tools, and features are calculated on the processed signals (spectrums). These feature values for each session are used as input to the ML\DL models that we use at Augury.
You might have noticed some similarities in the architecture ideas.
Similarities, Differences and Bottom Line
Comparing Augury data to the challenge data, every Augury’s machine data is conceptually equivalent to 3 IQ matrices.
One for acceleration, one for magnetic and one for temperature, where each machine’s recorded 4 second session is equivalent to an IQ matrix column.
In the challenge, one of the things we did was to process the raw signals using the Fourier transform — same as Augury.
But here is a crucial step we were missing in the challenge — calculating domain specific features from the processed data.
One of Augury’s greatest advantages is the combination of strong domain knowledge with custom-built algorithms. We have vibration specialists working together with data scientists to achieve excellent performance. The group that I participated in the radar challenge with, had no domain knowledge in the scope of radar signals. However, I believe that if we’d had a dedicated effort to investigate and calculate at least some basic radar-specific features, or even spectral features that could add information to every type of signal, we would’ve improved our score. It’s not surprising that an algo-team from a company with a product related to radar signals was in one of the first places.
On the other hand, in the challenge we gained a lot of value by applying CNNs and RNNs straight on the raw data and spectrums, without a manual feature extraction process.
As of now, at Augury applying NNs on the raw data and spectrums performed worse than using the domain based calculated features and these features trends. Nevertheless, we have a few new ideas that are currently in the research phase, so that may change soon. Stay tuned!
