Traffic forecasting, an amazing experience.
I have worked recently in a competition organized by BCG Gamma for traffic forecasting in Paris. The goal was to predict the traffic for the next 5 days in 3 avenues to optimize delivery for a virtual company. We worked as teams of 5 students for 1 week to create the model, predict future traffic and simulate a business interview.
Our models :
Unfortunately, me and my team had very little time for this competition, so we used some unrecommended practices in Machine learning :
1. We tested few models: since we didn’t have enough time, we simply ran into the most hyped models like:
- Facebook’s “Prophet” model.
- Some statistical models like: ARIMA, Exponential Smoothing …
- TFT: Temporal Fusion Transformers.
- N-BEATS: neural basis expansion analysis for time series.
2. We only used 1 avenue for model selection: the models that we used take too much time for training (except “Prophet” and statistical models), so we only used 1 avenue as a dataset for training and testing (back-testing: train up to day T and predict future 5 days, restart the training on day T+5)
3. We searched for bigger datasets: At the beginning, we only had 1 year time series for the 3 avenues, so we went to search for more data (archived data for the past 8 years), the problem was that in a limited time, it would have been better to improve the quality of our inputs by adding other useful data like weather, coronavirus cases… Instead of adding less important data (especially because of the pandemic behavior changed)
The results:
As we have trained on 1 avenue only, we found that N-beats performed better than others, we were also surprised that TFT didn’t give good performance at all. So, we trained our final model on 8 years for the 3 avenues (took 3 hours in a google colab pro) and predicted the traffic at last moments of the deadline.
Six days latter we had a meeting with the BCG Gamma team to discover with all our surprise that we were poorly ranked with a high RMSE score. After some investigations, we found that indeed our score for the avenue 1 (the one we used for model selection) was very good, but for avenue 2 our RMSE was twice the average.
Never use only 1 time series for model selection!
What we could have done:
The best team have used Catboost, which a simple machine learning model in comparison to our N-beats deep learning model. After the deadline, we had much more time to do some experimentations, so we tried LightGBM model which is a machine learning model that is extremely fast, it even performed better than Catboost. Once again, keep calm and keep it simple!
To Remember:
It is very important to start by using light and reliable models for your problem. Starting by classical machine learning models instead of the stronger and more powerful deep learning ones.
Never try to use one class/set/cluster (like 1 avenue) of data to do your model selection. Good practice is to use different models for each cluster of data or at least use diverse samples from the different clusters if you use 1 model.
