Building my own Cloud-based Robo-advisor

11 min readJan 6, 2020

A long, long time ago, I worked in quant asset management. It’s the business of rule-based investing or algorithmic trading where you base investment decisions on data and models rather than human judgement and fundamental assessment of markets or companies. Nowadays one would say it’s data-driven investing.

Get rich or lie on the beach tryin’

My actual job was to do data analytics and to design profitable strategies that would typically earn some risk premium and to combine several of them in a portfolio. As the decision making is purely algorithmic, the implementation can be fully automated.

When you do this all day long, you’re eventually attracted by the somewhat romantic idea of letting the money work for you, get wealthy easily by running an automated algorithmic trading strategy on your own account and live a glamorous life happily ever after. We often used the term “beach trader” as a symbol for it — you know, the trader that lies on the beach while the algorithm earns his living and eventually checks his P/L with the smartphone between two sips from a cocktail.

As a side note: As appealing as the idea may be, chances are that one won’t consistently beat the market (well, unless you’re Jim Simons). Instead, the risk premium you’re earning eventually turns out to be more risk than premium and the losses you take wipe out your profits. You then tell yourself “oh screw it, I didn’t want to lie on the beach all day long anyway!” Well, at least I did.

Reviving an old idea

Back then, algorithmic trading relied on local infrastructure. My broker of choice, Interactive Brokers (IB), came up early with APIs to their backend so that one could build trading strategies including automated trade execution through the API. I never wanted to have a server at home running 24/7 (too noisy and ugly) or touch one of the low-level programming languages supported by the API at the time (too cumbersome), so I never automated the whole process.

But now that there’s cloud computing and Python support, I thought I could give it another try. More for the sake of building a cloud architecture and creating my own robo-advisor that follows a disciplined investment approach rather than chasing big trading profits. Being a fan of serverless, I immediately started sketching an architecture on the Google Cloud Platform (GCP) in my head, with Cloud Scheduler triggering Cloud Functions that would calculate trading signals and place orders in the market and so on.

Unfortunately, IB’s API solution comes from a time when “the cloud” was just some fluffy thing in the sky. Unlike modern APIs such as GCP’s client libraries that talk directly to Google’s services, IB’s solution requires a piece of gateway software meant to run on premise, connecting the API client to IB’s API servers and handling the authentication, routing orders etc. The worst thing about it is that it’s not designed as headless software, i.e. the application requires some sort of display and you have to log in with your username and password through a UI. Ugh.

Given this legacy, a completely serverless solution solely with Cloud Functions was off the table, because the gateway requires a compute instance running somewhere anyway. So, should I set up a Windows instance on GCP? Nah… this lacks the fun component and moreover, cloud computing can do more than just replicate what you would build on premise.

Put components into containers…

Since I had already envisioned some kind of microservices architecture with Cloud Functions, I found it quite straightforward to put the single parts into Docker containers and use Kubernetes to orchestrate them. Of course, I could also run the containers on a GCE instance, but I like it a bit more managed. The way you get things like pod lifecycle management, DNS, rolling updates, health checks etc. with Kubernetes makes operations quite a bit easier.

So, the first challenge was to containerise that IB gateway relic. Fortunately, I was not the first one to try this. “The internet” has come up with solutions how to run the gateway headlessly. Tools such as IBController or (preferably) IBC take care of starting the gateway, logging you in, handling dialogue windows etc. You just need to make sure you install and run a window manager like Xvfb. If you’re interested in the details, head over to GitHub.

Once you have your gateway up and running in a container, you can start making use of the IB API and writing your trading application. IB-insync is a really great, object oriented wrapper for IB’s Python API that makes the use of the API way easier (hats off to you, Ewald de Wit).

… mesh them…

Here’s what the application looks like — the main components are:

IB gateways are the connector to IB’s servers. It consists of a Deployment and a corresponding Service. There is one for the live account (with real money) and for the “paper” account (simulated trading). The deployments must not have more than one replica each as the gateway software only allows one session per user ID. Once deployed, any microservice within the cluster can connect to the gateway to get market and account data, place orders etc. With IB-insync and Kubernetes’ DNS, connecting to the gateway service is as easy as ib.connect(host='ib-gw-live', port=4003). The IB gateway software has a tendency to hang, cause a burst in memory usage, lose connection to IB’s servers or log you out. Therefore, it’s crucial to include a health check to have the gateway container restarted in case it’s not responding for whatever reason. I added a small web server container to the gateway pods that queries the current time from the gateway to see if it’s alive. It is sufficient to define a livenessProbe.httpGet in the gateway container spec, and Kubernetes takes care of repeatedly performing the health check. See the full configuration and code on GitHub.
Strategies are the microservices that calculate signals or weights of a particular investment strategy upon HTTP request. I used gunicorn and Falcon as web server framework because Flask and IB-insync don’t go well together. Every strategy has its own Deployment and Service, so a request.get('http://my-strategy:8080') would produce a response with a JSON object containing the signals of my-strategy, such as buy X and sell Y or invest 70% in X and 30% in Y. You can find an example of a (pseudo) strategy on GitHub.
Allocators are the orchestrators of the investment process. They make a request to the strategy services to obtain trading signals and convert them into orders like buy 12 shares of X and 345 shares of Y. Allocators are CronJob objects, i.e. they are started according to a schedule, e.g. hourly during trading hours or daily after the market close or on the first day of the month. Check out GitHub for an example of how an allocator runs the above pseudo strategy once a day.

Let’s do a quick example, following the diagram above: Say I want to hold stocks that had a positive news flow or a strong performance the day before (no-one would want to do that, it’s just for the example’s sake).

The allocator-1 CronJob is started every business day after the market close. It sends a request to the strategy-1 Service and one to the strategy-2 Service.
Both strategy Deployments will then independently fetch the necessary data from public sources for the news flow and from ib-gw-live for the stock returns…
… calculate the signals, and send them back to allocator-1 in the HTTP response.
allocator-1 then consolidates both strategy signals and requests account value, current holdings, market prices, and FX rates from the gateway to come up with buy and sell orders.
Finally, it places the orders in the market through the gateway.

The illustration also shows a second allocator used to test a new investment idea with simulated trading.

… and ship them to the cloud

Let’s deploy the application on GCP. First, we need a small cluster on GKE. By small I mean g1-small. You’ll typically need one node per gateway, so if you want to have both the live and the paper one deployed, create a cluster with two g1-small nodes. You can also set up a single-node n1-standard-1 cluster and place both gateways on the same node, however, with g1-small nodes you get smaller (i.e. cheaper) increments should you ever need to scale the cluster. Of course, the optimal cluster configuration ultimately depends on how many concurrent strategies you want to run or whether you use resource hungry machine learning models for example.

When you create a standard cluster, the nodes by default get external IP addresses and hence, the cluster is exposed to the internet even if you don’t define an Ingress. I found that I sleep better when my trading infrastructure is less exposed to the world. Therefore, I decided to create a private cluster, i.e. one with no external IP addresses and therefore no public endpoint. Instead, the nodes only have internal IP addresses and are reachable solely from within the VPC. This isolates the cluster from the public internet.

However, this also has two major consequences: First, you can’t kubectl the cluster from your local machine or Cloud Shell; and second, if your cluster nodes don’t have an external IP, the application can’t communicate with the outer world, in this case with IB’s servers or a public data source that your strategy may require. Hence, we need two more things: a bastion host and Cloud NAT. The bastion host allows us to kubectl from there in case we need to administer our cluster resources, given it is in the same network as the cluster and given the appropriate firewall rules. Cloud NAT enables the cluster to reach the internet while protecting it from unwarranted access.

Of course, I also used other GCP products and tools to garnish the architecture:

Deployment Manager to create the VPC network and subnetwork, the firewall rules, network tags, Cloud NAT, bastion host, and finally the GKE cluster. You could argue that using Deployment Manager for such a small setup that doesn’t even need to be repeatable is a bit over engineered. But I really learned to appreciate it: Spinning up and tearing down everything with one command instead of having to create or delete single resources is quite handy. Also, it’s a great way to document your infrastructure and its configuration.
Logging and Monitoring to catch application errors, trigger notifications and monitor the cluster in general.
Firestore as the database to store application settings (e.g. the cash quote) and parameters as well as to log application activity (e.g. strategy signals or orders). I used Firestore for this instead of deploying a database to the cluster because it doesn’t require additional cluster resources (nodes) and is easy to use.
Cloud Build and Container Registry to build and store the Docker images.
Source Repositories for the source code.

Trust is good, control is better

Alright, the application is up and running, but it’s somehow hidden away in its private cluster. It would be nice to see what it is doing, considering that I’ve given it power of attorney to invest my money. Of course, I can check the logs in Stackdriver and Firestore to get an idea of what’s going on. Also, I can always log into IB’s account management website. However, as IB allows only one session per user ID, I’d have to delete the gateway deployment for a moment or take the whole application offline. Not really practical.

So, let’s add a dashboard that shows account and portfolio summaries, trading activity etc. I also want to have some sort of kill switch that lets me close all open positions and stop trading activity in case something goes terribly wrong. I could easily deploy a web app to the cluster, but then it would have to be a public cluster rather than a private one.

Therefore, let’s use App Engine instead. We have the choice between the flexible environment that has at least one instance running 24/7 and the standard environment that scales down to zero if there’s no traffic. Given that the dashboard app will have very little traffic because I’m the only user and I’ll only check it out every once in a while between two sips from a cocktail, the standard one is way cheaper. However, in contrast to the flexible environment, the standard one has no instances in your own VPC network, a prerequisite since the private cluster only allows VPC-internal traffic.

Enter Serverless VPC access (currently in Beta). It allows you to create a connector your App Engine (standard environment) or Cloud Function can use to communicate with the resources in your VPC. It will cost you two f1-micro instances (may be subject to change though) which is still cheaper than the n1-standard-1 an App Engine flex instance would cost. If you choose to use Firebase and Cloud Functions instead of App Engine, using Serverless VPC access is also the way to go. By adding an internal LoadBalancer to the gateway service(s) in the cluster, we get an internal IP address the web app can use to talk to the gateway through the Serverless VPC access connector.

The last thing to do is to add an authentication layer, preventing the web app from being accessible by the whole world. The easiest way is to set up Identity-Aware Proxy so I can simply use my Google account credentials to log into my web app.

To summarise, here’s the complete GCP architecture for the algo-trading solution:

Not quite a free lunch

Finally, some cost considerations. Investing is about money, after all.

Running a cluster with two g1-small nodes 24/7 costs at least about USD 28 per month (including sustained use discounts). Depending on which GCP-region you chose to deploy your application, it can be higher, e.g. about USD 37 in case of Zurich region. The privateness feature makes the setup considerably more expensive as you also need Cloud NAT and Serverless VPC access which together cost around USD 40 in a cheap region (the pricing of Serverless VPC access may be subject to changes once the product is out of beta stage).

So, all in all, we’re talking roughly about costs of USD 70 per month. If you run a 1 million dollar portfolio, that’s about 8.4 basis points off your annual return, so pretty affordable. If instead it’s 10 grand, it would cost you 8.4 per cent annually, quite a hefty fee from your virtual investment manager. If you choose to curb security for lower costs and do without the private cluster option, the total price will come down to about 30 USD per month — still somewhat expensive compared to a lean, completely serverless solution that scales resources and costs down to zero when they’re not needed. I haven’t given up hope yet that one day, IB will come up with a more modern API solution that makes the gateway software obsolete…