How to train your LSTM for time series forecasting

(and why some of the methods out there are fundamentally flawed)

My master’s dissertation in data science was about airline fleet planning. This topic combines two areas I’m passionate about: Business Optimisation and Aviation. Business optimisation fascinates me because it transfers very complex real-world problems with intricate relationships and dependencies to this mathematical world, which is probably as complex as the first one but structured. And then it reduces it into a single objective function before solving it numerically for the optimal decisional variables that satisfy all constraints. This is magical to me.

IBM Decision Optimization (Tutorial: Beyond Linear Programming: https://ibmdecisionoptimization.github.io/tutorials/html/Beyond_Linear_Programming.html)

I won’t describe my dissertation in detail; that would be another post. However, I’ll say this: I gathered a vast dataset of all domestic flights in the US from 1990. Then, after cleaning and manipulating it, I engineered some features that were essential for the analysis I was going to conduct. In the end, multiple datasets were aggregated, each for a specific analysis.

Now, the question becomes how to forecast. I chose two algorithms: ARIMA (or more SARIMAX, really) and LSTM. The reason is that they’re both solid and can theoretically incorporate exogenous variables into their calculations (hence the X in the SARIMAX).