Forecasting USD-MNT Exchange Rate — Part 2: Machine Learning

When a good model doesn’t work

Robert Ritz
Dec 3, 2018 · 9 min read

The past few weeks have seen an amazing depreciation of the Tugrik. As Mongolia relies so heavily on imported products, this weighs heavily on everything from prices of potato chips to fuel to vehicle imports. Also, the International Monetary Fund released a report earlier this month reviewing the Mongolian macroeconomic situation. I’ll leave the macroeconomic commentary to others, but I hope this shows the importance and scope of this problem.

In part 1 of this series I showed one tool for forecasting time series data, Facebook’s Prophet tool, and applied it to the USD-MNT exchange rate and meat prices. The result was a model that didn’t give high confidence for forecasting exchange rates, but a seemingly well fit model for meat prices (which are highly seasonal).

In this article, I’ll show how to apply machine learning algorithms to forecast exchange rates 3, 6, and 12 months in the future. This article will be more technical than most, so feel free to skip to the bottom with the results if data munging and code isn’t for you. All of the code and data for this project are on Github here.

The Problem

Let’s start with a statement of the problem. It is always a good idea to start with a well-defined problem to work from. If you have all of the data and skill you could want, but not a defined problem, you won’t have a good result.

The Problem: Given available macroeconomic data, can we accurately forecast USD-MNT exchange rates 3, 6, and/or 12 months in the future?

The Data

Approaching this problem, this was my least certain piece. I’m no expert in macroeconomics, so I had to do some research (and ask Kevin, our economics instructor at LETU Mongolia, for some advice). From my research and discussion, I found that money supply and balance of payments were the places to start. I dug around 1212.mn to find that data and more and came up with the following data features.

  • USD — USD-MNT exchange rate
  • CPI — consumer price index. This is the change in CPI from the previous month in percent.
  • m2 money supply
  • m1 money supply
  • current: the current account from the balance of payments
  • capital: the capital account from the balance of payments
  • financial: the financial account from the balance of payments
  • errors: errors and omissions from the balance of payments
  • balance: the balance of exports and imports in Mongolia

Other features, such as interest rates, quasi-money, and government account balance were excluded as they hurt the accuracy of the model.

Setting It Up

I collected all of the data above and combined them into one dataframe. The code and details are located here. One challenge was the periodicity of the various features. Our exchange data is daily, some data is monthly, and others quarterly. For our daily exchange rates, I took the last value of each month. For the quarterly data, I copied the quarterly value to each month in that quarter. This gives us a dataframe of monthly data that is easier to work with.

First, we will import the libraries we will be using and also load our data into a Pandas dataframe.

The first and last 5 rows of our initial dataset. 130 rows with 9 columns. The last 4 months have no balance of payments numbers.

Now we can start to do some basic visualization and analysis of these features to get an idea of how they relate to our target variable, the USD exchange rate. One simple method is making a pairwise plot to show how every numeric variable relates to every other variable. As all of our data is numeric, this is a quick way to see everything at a glance. Along the diagonal of the plot is a histogram of the feature.

We can also show how each feature correlates to each other feature. A correlation plot looks simple, but it is worth some time taking a look at this.

The very light and very dark boxes show a strong positive or negative correlation between the features. A positive correlation would be that as the USD-MNT exchange rate increases (the currency depreciates) the other feature increases as well. A negative correlation is the opposite. As the exchange rate rises the other features decreases.

We are particularly interested in correlations to the USD feature. M2, m1, balance, and financial have strong correlations. Put simply, as the money supply and export balance increase, the Tugrik depreciates. As the financial balance increases, the Tugrik appreciates. It should be noted this is correlation only, and this does not prove a causal relationship.

Feature Engineering

Currently our dataframe isn’t exactly what we need. This is because our machine learning algorithms only learn row by row and aren’t aware of other rows when learning or making predictions. We can overcome this challenge by imputing previous time value, or lags, into our data.

After some trial and error, I determined that 4 lags (or 4 months) work best. The code below creates a function that will create 4 lags for each feature in the ‘features’ list. Our new dataframe now has 41 columns!

You can see that “lag1” has the previous months value, “lag2” and two months previous, and so on. This gives the algorithm some knowledge of the previous month.

Next we can add the year and month into the column. This gives the model a “sense” of time instead of these other more abstract numbers.

Finally, we can take the difference between the current and lag exchange rates. By itself the algorithm isn’t able to reach these sorts of conclusions, so we can calculate this and add it to our dataframe as well.

Create Target Values

Our goal is to predict USD-MNT exchange rates 3, 6, and 12 months ahead. To do this we can create 3 new columns with the future 3, 6, and 12 month exchange rates from our dataset.

Our final dataframe has the following columns:

The Models

To prepare our data for the machine learning algorithms, we first split our dataframe into our predictor variables (X) and our target variable (y). We then separate those into training and testing sets (80% for training, 20% for testing). The training set will be used to train the model, and the test set will be used to test the models accuracy on data it has not seen before.

We tested the following models from the scikit-learn Python package with our data: Linear Regression, Neural Network Regression, Lasso, ElasticNet, Random Forest, and Extremely Random Trees. As we are predicting a continuous variable, we are using regression. In addition, we are predicting multiple outputs, and not all algorithms support this.

I didn’t spend much time fine tuning the models’ hyperparameters as I wanted a good baseline to compare. The test set error of all of the models are shown below. Mean absolute error was chosen for its interpretability.

Mean Absolute Error for the models tested. This error calculation was chosen for its interpretability.

Our best model was the Extremely Random Trees model. We can view how the predictions in the test set compare to the actual exchange rate. The test set is the data that was withheld from the model (20% of the total) during training.

The horizontal axis is the actual values, and the vertical axis are the predictions made by the best model (Extremely Random Trees). This looks quite good, and with a mean absolute error of 28.12 MNT it seems like we have a reasonably good model. Let’s visualize our USD-MNT time series and plot the test set predictions to get another view.

Code to generate data and plot below.

The green, orange, and red dots show the forecast for the corresponding date. Green dots are 3-month forecasts, orange are 6-month forecasts, and red are 12-month forecasts. These forecasts were made using the test dataset, which is not used to train the model. What we can see is that our model is doing a pretty good job of forecasting based on unseen data.

Feature Importance

One of the things created by the extremely random trees model are features importance’s. These are weights generated by the model for each feature (column) of the data and then used to create forecasts. We can interpret these weights to better understand which features are more or less important in making a forecast. They should not be interpreted to show real-word importance of the features, however.

We can see that the current and previous exchange rates as well as the year are the most important features. In addition, we can see that the m2 money supply and it’s lag values are the next most important.

The Forecast

Now that we’ve established that the accuracy of the model on unseen data, let’s make a forecast using the most recent available data and see how it does. This is June 2018 as of the time of writing. This is because balance of payment data is released quarterly, and not monthly by the NSO. Here is our forecast 3, 6, and 12 months ahead from June 2018. There are two views given, one going back 24 months from November 2018 and the other going back 48 months.

Forecast with 24 month window
Forecast with 48 month window

Uh oh. Our model doesn’t seem to be doing very well. Or is it? It seemed that our model did quite well on unseen test data, but this is quite far off from the measured values. This could be for several reasons:

  • Something happened in the past several months to cause our predictions to go wrong. This could be changes in our data that we haven’t seen yet (like balance of payments).
  • Outside market volatility that hasn’t been seen in the past 10 years.
  • There is an issue with our model.

Judging from the IMF report mentioned earlier and personal observation, the first reason seems the most likely. This doesn’t spell death for our model, and instead I believe it shows the extraordinary situation the Tugrik is in currently.

Conclusion

From what we’ve seen above it seems reasonable to say that we forecast USD-MNT exchange rates with reasonable accuracy given complete up to date information at the monthly level. The data used to build these models was taken from public sources, and it is reasonable to assume economists at MongolBank or even commercial banks have access to more complete up to date data.

Our latest forecast was made using data from June 2018. We can see that in July and August the measured values were deviating from the forecast quite severely. One possible use case for a model such as this is to identify unusual depreciation as it is happening. This could be used as a signal to MongolBank or private businesses to prepare for a period of increased volatility or depreciation. A future area of research could be using forecast balance of payment data to make exchange rate forecasts.

As usual, you can find the data, code, and images for this article on Github here.

Liked this article? Give it a clap, comment, or send me an email at robertritz@outlook.com. Starting next week I will be on winter break, but will continue writing again starting January 2019.

Mongolian Data Stories

Using data analysis, visualization, and machine learning to tell Mongolia’s story. Interested in writing for Mongolian Data Stories? Send an email to Robert Ritz (editor of MDS) at robertritz@outlook.com.

Robert Ritz

Written by

Data Scientist and Director of LETU Mongolia. Keen observer of Mongolia.

Mongolian Data Stories

Using data analysis, visualization, and machine learning to tell Mongolia’s story. Interested in writing for Mongolian Data Stories? Send an email to Robert Ritz (editor of MDS) at robertritz@outlook.com.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade