Fetch Forex API with Python and Pandas

Having written more complicated implementations of real-time forex data, it occurred to me that I have missed a simple implementation of REST API for python users. So here we are trying to learn to consume current and historical forex data using python and pandas. If you looking for quick implementation see our article on our Python SDK.

Before we begin a caveat, you need to have an elementary understanding of Python, I mean basic like the one that comes from watching “learn python in under 10 minutes” videos. That said, this tutorial should be helpful to people with experience in other programming languages and people who would like to use our Forex REST API.

So let’s begin

You will need to signup to our API, just click Join API for Free. Once you have the key keep it safe. This and Python environment is all you need to follow this tutorial. For simplicity purpose, I would recommend users with no python set up to head to Google Colab and launching a Jupyter notebook. For those who have Python installed, these programs can be run locally.

You will need to install a few libraries but for Colab you don’t need to install just import. If you are a beginner to Python or want to test a small bit of code quickly Collaboratory from google is very good.

Calling real-time forex rates using live endpoint

import requests
from pprint import PrettyPrinter
pp = PrettyPrinter()

You can copy-paste the above code in a cell on Google Colab Notebook and then hit shift+enter or click the play button.

We can see the cell has run and we have a new cell to write our next bit of code. Before we run the code below we need to understand what it means. The url is the endpoint we are calling, the currency is a comma-separated string and api_key is the key you got by signing up. To call the forex REST API we will need the requests library which we imported in the previous cell, requests library has a get function that takes in a URL and a JSON parameter that in this case is the “querystring”. The response is what we get back from the API we then print the response using PrettyPrinter so it looks nice.

url = "https://marketdata.tradermade.com/api/v1/live"
currency = "USDJPY,GBPUSD,UK100"
api_key = "api_key"
querystring = {"currency":currency,"api_key":api_key}
response = requests.get(url, params=querystring)
pp.pprint(response.json())

As you can see we have got live rates for USDJPY, GBPUSD and FTSE100 in not more than 6 lines of codes. However, to do some data analysis or see the rates in a more presentable way, we will need to put them in a table or as we say DataFrame in pandas.

So here we will like to introduce pandas which is an extensive library created for data analysis. pandas make the job of iterating, tabulating, modifying and calculating data fairly easy but for this tutorial, I’ll keep the scope limited to getting, saving and tabulating data from the API. So let’s continue

import pandas as pd
df = pd.DataFrame(response.json()["quotes"])
df

Pandas can also get the data from the API but here to understand things better we have used requests. It’s ideal to use requests if the data you are requesting is not an exact data frame. We pass in quotes we received in our response.json into a pandas function that converts data into a data frame and voila! we now have our data tabulated in 1 line of code.

The pandas’ data frame above is more readable but we still not there. We will do some adjustments to make it more readable. We can see we have NaN in the instrument column and ask, bid, mid are not in order and we don’t have a timestamp, so let’s fix that quickly.

# if all instruments are currencies just write a # in front of line 1 and take # off line 2 in the below code. If all instruments are CFDs just write # in line 1 and 2.
Import numpy as np
df["instrument"] = np.where(df["base_currency"].isnull(),df["instrument"],df["base_currency"]+df["quote_currency"])
#df["instrument"] = df["base_currency"]+df["quote_currency"]
df["timestamp"] = response.json()["timestamp"]
df["timestamp"] = pd.to_datetime(df["timestamp"], unit="s")
df = df[["instrument","bid","mid","ask","timestamp"]]
df

We will import numpy which is another library that goes hand and glove with pandas then will change the instrument column using the “np.where” function which takes in (condition, is true, if false), and then will create a new column called timestamp and writes the timestamp we received from API response.json and convert it to a readable timestamp.

You can now see why Pandas is popular and Python productive. And all this code executes really fast. You can also do this:

You can save your work with a single command as shown below and share your work via a CSV file.

df.to_csv('live_rates.csv')

For an in-depth overview of our endpoints check out our rest API data documentation page.

Calling historical forex data using historical endpoints

Daily Historical

Daily Historical data endpoint is very similar to the live endpoint the only difference is instead of timestamp the response JSON has a date and instead of the bid, mid, ask in quotes, we have open, high, low, close. We have already defined currency, and api_key so all we need to provide is a date we require data for.

url = "https://marketdata.tradermade.com/api/v1/historical"
date = "2021-03-15"
querystring = {"currency":currency,"date":date, "api_key":api_key}
response = requests.get(url, params=querystring)
  
df = pd.DataFrame.from_dict(response.json()["quotes"], orient='columns', dtype=None, columns=None)
df["instrument"] = np.where(df["base_currency"].isnull(),df["instrument"],df["base_currency"]+df["quote_currency"])
df["date"] = response.json()["date"]
df["date"] = pd.to_datetime(df["date"])
df = df[["instrument","bid","mid","ask","date"]]
df

You can see it is easy to make tables out of data and it’s not far fetched to think that with a bit of pandas magic you can slice the data as you want. You can also loop through the requests and ask for data for different dates the possibility are endless. For the other two data historical endpoints: minute_historical and hour historical you can only call one rate per currency at a time so there is a need to loop through the data.

Minute and hour Historical

The calls for both endpoints are almost identical so will just use one example for minute historical as shown below:

fx = ["EURUSD", "USDJPY"]
dates = ["2021-03-15-13:00"]
array = []
url = "https://marketdata.tradermade.com/api/v1/minute_historical"
for i in fx:
    for date in dates: 
          querystring = {"currency":i,"date_time":date, "api_key":api_key}
          response = requests.get(url, params=querystring)
          array.append(response.json())
df = pd.DataFrame(array)
df

By now we can see it’s not very difficult to ask for different dates and different currencies and with a bit of work, it is possible to make a system that checks rates and highlights certain alerts based on some conditions we preset. However, this outside the scope of this article and we advice exploring pandas which is a very handy library for data analysis.

Calling time-series forex data using timeseries endpoint

If you requesting historical endpoints the request are generally of a price at a point in time. We will now look at the time series that is used for trend analysis and charts and getting data in chunks. However, we don’t need the requests library as we can do all our stuff using just pandas. Time to speed up how we get data.

url = "https://marketdata.tradermade.com/api/v1/timeseries?"
currency="USDJPY"
start_date="2021-03-01"
end_date="2021-03-22"
format="split"
interval="daily"
df=pd.read_json('https://marketdata.tradermade.com/api/v1/timeseries?currency='+currency+'&api_key='+api_key+'&start_date='+start_date+'&end_date='+end_date+'&format='+format+'&interval='+interval)
df = pd.DataFrame(df.quotes['data'], columns=df.quotes['columns'])
df.tail()

The pandas can request data from an API using the function such as read_json or read_csv. Our forex API timeseries endpoint takes in a parameter called format when that format is set to “split” it’s really easy to convert it into a pandas data frame. We then read the columns that passed in from the API and then print the last five lines of data we have received using command df.tail() if you want to see the first five data points simply do df.head().

Data is already formatted in a way that’s easy to play we can also make a similar request for hourly and granular data all we need to do is just add minor changes like start date and end_date changed to “YYYY-mm-dd-HH:MM” and interval changed to hourly. You can see below:

We can do a lot of stuff with the above data frame like make an extra column for range, % change, rolling correlation and volatility. However that outside the scope of the article. You can check out our data visualisation article that shows basic pandas commands for correlation and volatility.

Calling tick historical forex rates using tick sample endpoint

Finally, we will look at tick historical data that is very useful if you doing any serious data analysis including trading algorithms and machine learning. However, tick data can be used for many purposes. We will be able to get our data in a pandas data frame in one line of code from our rest API. Before we run our code just one reminder, the tick data is available to free users for only the past four days, not including today, you can’t request more than thirty minutes in one single request on a free plan and it will cost 20 API requests for each call. So I would advise saving the file locally using df.to_csv(“example.csv”) and then reading it in by df.read_csv(“example.csv”). Also, one reminder for Google Colab users because there is a gap in our start date and end date format for tick data endpoint, we will need to fill that gap with “%20”. This is something I noticed in the Google Colaboratory notebook you don’t need to do it in Jupyter notebook or Python script.

# start date is 2021-03-22 14:00 but %20 is added to the gap as url dont like empty spaces (this is added automatically but Google Colab doesnt seem to like this)
currency="EURUSD"
start_date="2021-03-22%2014:00"
end_date="2021-03-22%2014:30"
format="csv"
url = "https://marketdata.tradermade.com/api/v1/tick_historical_sample/"+currency+"/"+start_date+"/"+end_date+"?api_key="+api_key+"&format="+format
df =pd.read_csv(url)
df.tail()

As you can see you have just pulled almost 2000 tick historical data points for EURUSD. You can get the Jupyter Notebook file from our Tradermade Github page. Hope this article helps both beginners and experienced programmers in getting data from the Forex REST API using Python and pandas. Please leave your comments and suggestions.