I have previously created a model to predict the Forex market:
Pragmatic Deep Learning Model for Forex Forecasting
Using LSTM and TensorFlow on the GBPUSD Time Series for multi-step prediction
Now we want to use this model for trading under a commercial trading platform and see if it is going to generate a profit.
The techniques used in this story are focusing on the model in my previous story, but they can be tweaked to fit another model. The intention here is to make the model usable by other systems, e.g. a trading platform.
You can find the source code for this story in directories “LSTM-FX-CTrader-Client” and “LSTM-FX-Prediction-Server” here:
This is the companion code to Pragmatic Deep Learning Model for Forex Forecasting. So, if you want to understand the…
In the previous story, we have trained and tested a model and saved the resulting model as a directory and the scaler used for the data as a file.
The model and the scaler are the only items that we need, in addition to understanding the input and the output parameters.
Trading in this story refers to Algorithmic Trading, also known as Quantitative Trading. Algo trading is when a trading strategy expressed in code, assesses whether a trade could be profitable and executes this trade automatically with minimal human intervention.
The automated trading strategy is referred to as a Trading Bot. Today, there are plenty of commercial algo trading platforms where you can host your own bot, here are two examples:
cTrader: A manual and an algo trading platform. It accepts bots written in C#.
Meta Trader 4: A popular platform for Forex manual and algo trading. Bots can be written using a proprietary language, similar to C, called MetaQuotes Language 4 (MQL4).
I am going to use cTrader in this story to host our bot.
When we want to expose a software system A to be used by another software system B, we use the term “we are exposing an Application Programming Interface (API) from A”. We call the consumer, in our case the trading platform, as client and we call the producer, in our case the packaged model, as server.
Tech talk: We want to create a prediction server and expose an API to allow 3rd party trading clients to consume it.
The best way to make our model consumable by the vast majority of trading platforms is to use the REST protocol and wrap our model within a RESTful API. In less technical terms, this is making our model accessible via a URL where you pass a URL that contains the input and get back “text” that contains the output.
The RESTful APIs use HTTP to communicate over the network between systems, this means that if our model is in Python and hosted on Linux and it needs to be consumed from a system written in C# and hosted on Windows, then that is possible as long as the two languages can use the HTTP protocol, and any semi-decent programming language supports HTTP.
To expose our model via RESTful API, we need to host it (wrap it) with a web server. Since we are using Python for the model, one popular non-production web server for Python is Flask.
You can use Windows or Linux, Flask works on both, but I would recommend using the same OS used for training, but not necessarily the same instance. Setting up Flask from your console:
conda install flask
pip install flask
The full source code setting up the system is on my GitHub page for setting up the server.
What is interesting from the source code is the following:
@app.route('/predict/<string:ticker>/<int:batch_size>/<int:window_size>/<int:ma_periods>/<float:abs_pips>/<int:pred_size>/<string:instance>/<string:series>', methods=['GET'])def predict(ticker, batch_size, window_size, ma_periods, abs_pips, pred_size, instance, series):
This allows Flask to accept a URL like this:
The server might host multiple models, so to differentiate them, I made them identifiable via ticker, batch size, window_size and moving average periods, so a model might be called: gbpusd-32-256–14.
ticker: The official name of the instrument, GBPUSD in our case.
batch_size and window_size: parameters required by the model.
ma_periods: simple moving average smoothing periods.
abs_pips: the limit when your prediction will stop and the prediction results will return. So, if it is 0.0008 and your current price for GBPUSD is 1.3000 then the model will stop predicting if it reaches 1.3008 or 1.2992 .
prediction_size: how much forward units max you want to predict and before you hit the abs_pips. Take the previous example, if your prediction_size is 30 units (minutes in our case) and you hit 1.2992 after 5 units, prediction will not continue and will return “Sell”, but if 30 units are reached without hitting the limits, the system will return “No Action”.
instance: a reserved integer for future use. For now, supply any integer.
The last section, series, of the URL is composed of a comma-separated prices:
length = window_size + moving_average_periods = 256 + 14 = 270
Your web server can run from the console:
To test if your server setup is working, try the example URL from the source code.
We have a prediction server. To use this prediction server, a client needs to supply a URL in the previous format and then gets a prediction.
I used cTrader, a commercial manual and algo trading platform, as a client. The full source code and setting up the system is on my GitHub page for setting up the client. The language used for the bot is C#.
This is the bit where our code requests a prediction from our server:
The code above prepares a URL made of the inputs that our server requires. Then it passes it to the server and gets back a “1”, “-1” and “0” which respectively means Buy, Sell and I don’t know.
The code above is executed on every unit pass, hence the name OnBar. In our example, it is using 1 minute as a unit.
I ran this bot in backtesting between 24/08/2020 and 30/08/2020 with £1000 capital, having the client server setup above and it made a small profit of £49.
However, I tried other periods on backtesting and it was gaining or losing few pounds per week. I call this promising result and I believe they can improve by enhancing the model, but I would not use this ml model in production with real money in its current condition.
These stories are meant as research on the capabilities of deep learning and are not meant to provide any financial or trading advice. Do not use this research and/or code with real money.
Extensions and Limitations
REST vs gRPC
REST is the greatest common denominator for trading platforms and for modern systems in general. However, it is not the fastest. A faster and popular API type is gRPC and just to be clear, faster in this sense means a fraction of a second.
Meta Trader 4 does not support gRPC, without hacking. cTrader is using .NET 4.0 classical, which came long before gRPC, so it is hard to use this protocol with this version of .NET.
However, if fractions of a second matter, consider gRPC (or maybe a faster protocol).
JSON is a data structure that is often used with REST. I have not used JSON on purpose to comply with the vast majority of clients.
I was able to represent my input and output in a simple way, but once your model input and output become more complex, you might want to consider JSON.
Having the model hosted in a web server makes it easy to request multiple predictions at the same time, e.g. within 0.2 seconds gap. This might be the same client requesting multiple predictions or multiple clients requesting multiple predictions at the same time.
This might lead to a run-time error if your model does not support multithreading (we say it is not thread-safe). The safest way is to access your model once at a time, which means when it is not busy predicting, but if you need it to predict at a random time, you will probably need to write extra code to make your model thread-safe. I have not tested this for thread-safety.
If you are hosting your server remotely, you might consider setting your web server to HTTPS, that is installing an SSL certificate, if you want to have a secure connection between your client and your server. This will mean that if someone can intercept the communication between the client and the server, they will not be able to decipher what is being predicted.
Authorisation and Authentication
If you are hosting your server remotely, it is accessible to the public. Although, it is hard to know what are the expected parameters for it to be useful. You don’t want to leave it to chance, protect it via a RESTful API security protocol.
There are multiple security protocols, the most popular one is OAuth2. Discussing these protocols are outside the scope of this story.
Hosting your Model
There are professional ways to host your model: The Cloud. You can rent a cloud server virtual machine or go for a serverless option by using an ML hosting platform such as Azure ML.
The server option is less expensive, but you will have to manage the operating system yourself and spend time in the nitty gritty details, while the serverless option is more expensive but requires less maintenance overhead.
I hope this clarified the idea of an end-to-end process and using an ML system built with Python from another system built by another programming language and not necessarily located on the same network.
Although the model performance was mediocre, it is not yet optimised and there is a huge room for options that can be considered to enhance its predictions, I shared few of them in my previous article.
My intention was to share a working ML prediction framework that is usable and extendible. I hope this framework would be of use to the readers.
My background is 20 years in software engineering with specialisation in finance. I work as a software architect in the City of London and my favourite languages are C# and Python. I have a love relationship with practical mathematics and an affair with machine learning.