Why your great trading idea is secondary at first
A guide to get your priorities straight on individual AlgoTrading
Over the years, I have had countless conversations about passive income and the personal path that each one has tried to take to this big goal. It’s been exciting!
I’ve heard all sorts of interesting ideas — discussed many well known and sometimes even novel trading strategy for a selection of instruments on meaningful timescales. And I’ve implemented quite a few of them with fully automated trading algorithms.
When trying to put ideas in code, the biggest insight for me was that the main problem is not the idea. The two major challenges were to oversee the complexity of a trading system to get its implementation right, and second to maximise congruency when you translate theoretical entries & exits into real trades.
None of our ideas is worth a Cent if our trading reality is far off the road in a bumpy desert, whilst in theory we’re supposed to be cruising smoothly on a highway.
To get us started, lets go back to this significant moment when people get really excited about AlgoTrading
Listen, I have a great idea !
From one of my recent conversations I took this great one away…
the “Twitter idea”
It’s super easy buddy! I’ll utilize the Twitter API and run the incoming stream through sentiment analysis which gives me how the community thinks and from there I distill my entry signal on a single stock or maybe even the whole market and then I’ll rock’n’roll !
Okay, here’s another one I invented myself years ago, to get a little more fancy… the “Outburst-Correction idea”
I’ll take a very liquid future (ES, NQ, YM or maybe RTY), calculate local High’s and Low’s on intraday to identify the “trend” and then I position with the trend with a moderate loss- and tight profit-stop. There’s potentially no problem to risk 5x as much as I could earn since I expect to be mostly right anyway!
One more, now we go really nuts, this started as a trade engine execution test and produced astonishing insights… the “Modulo-Long idea”
Futures are just great and “Long” as a direction seems to be much more likely. On a simple Modulo-5 of the current price I go Long with ES, take 2 points (100$) as profit and leave. Yeah this could easily happen 30x a day and I don’t worry about risking more than I could earn either. Long happens anyway!
Let’s not rashly judge. Each of these ideas is a great opportunity to dive deeply into the details and to start finding the reasons why things work or fail.
To get us going, I’ll get to the point right away…
An End-2-End AlgoTrading environment
So we have an interesting idea, great! And lets say we could express that idea in a coded implementation — which by the way is hard enough and covered in a later chapter.
Well, that’s just this green bubble on the next chart, and its not more or less than this single piece in a more complex End-2-End AlgoTrading system which you need (to some degree) to get your idea to work
On a very high level…
- You’ll need some sort of bucket, which I call TradeTactic from now on
- It’s used to pair the implemented idea (I will call Project) with its Parameters to tell it how to run
- in addition it stores a Behavior Configuration to tell your Broker “What” (the Instrument), “Where” (the Account) and “How” (the Number of Shares, Acceptable Slippage, expected Order Types) it is expected to execute
- from a flow perspective, the TradeTactic is producing Theo Trades that are translated by a TradeEngine service into Order Requests
- these get sent to your Broker that is bridging to your Stock Exchange
- On the way back, the Exchange provides Market Data, Orders & States, Executions and Positions which the TradeEngine has to manage and translate into Real Trades that are mapped to your Theo Trades
- Out of this mapping your TradeEngine can calculate the diff, which is its actual “manufacturing order”
- Ideally, this flow becomes an endless loop that’s finally producing positive & profitable outcomes.
If you’re still not having enough, lets look at the most relevant aspects in more detail…
Understanding the mechanics :
One… The Instrument
It would be inappropriate to give you a concrete suggestion when choosing the trading instrument. The instrument has to fit your idea and to a good extent also to you and to your conviction. Are you interested in single stocks or in entire markets? Is gambling of agricultural commodities ok for you or are you more interested in crypto currencies? Do you want a leverage effect known in many Futures or even risk everything with Options? This is a purely personal decision that everyone has to make for themselves.
I currently only trade index futures as they combine 5 key features that influence congruency between theoretical and real trades:
- Liquidity is very high — to maximise congruency you need as many ticks as possible daily which also drives the volume. As an example, ES has an average of ~43K ticks/day during the past 10 years. It gives you a 24 hours average of ~12 ticks/minute, or an opportunity to enter/leave the market every 5 seconds. Of course ES is much more liquid during normal trading hours, where you have multiple opportunities to trade every second, so its more likely you get the price you want.
- Leverage effect — for every point one of the mentioned index futures moves, you’ll move your unrealised Performance/Loss (PL) with the instruments multiplier. YM is on 5 (means one point on YM moves your PL at the amount of 5$), NQ uses 20, ES and RTY come with 50. However, having a higher multiplier doesn’t necessarily mean these instruments are more risky since they usually move in submultiples of 1. Since YM moves in 1:1 and NQ in 1:4 the PL amount in both is the same 5$ for the smallest possible tick. What’s important here is that the leverage effect allows you to win (but also lose) hundreds of dollars every day. We’ll see why this is important for AlgoTrading.
- 24x5 Trading — I mentioned the average number of ticks on a 24 hours scale, which seemed a little bit weird at first. Why wouldn’t we just use the regular trading hours or even better the “Liquid Trading hours” (an info that some Brokers provide)? My advice here is to keep things as simple as possible in the first place — You don’t want to manage trading hours of the actual stock exchange behind your broker and the time difference to the host that’s running your trading system including daylight saving time. Don’t under-estimate the level of complexity in this area! Of course all this is possible and you may have scenarios where you really need to, however my personal experience taught me to eliminate time offset factors completely and work with instruments that are traded 24x5 and rest at the weekend.
- Daily movements (and also corrections are high) — in both directions, which make futures very good candidates for LONG and SHORT trading. Every movement creates the chance for a profit, well theoretically.
- Commission are reasonable— depending what Broker you use, the Commissions for a single contract are usually acceptable, relative to the chance for profit the futures give you. On the pricing model I currently use, the futures are at roughly 2.05$ per share and order, which makes 4.10$ per share & transaction. On the assumption you could constantly make just one point on YM on every transaction, you’d already be slightly positive… well actually flat, which only is supposed to tell us one thing: commissions on futures are potentially ok for day-trading scenarios.
I’m sure you have noticed from the previous approach that my focus is rather short-term, which explains why I only differentiate two major aspects when I look at time timescale: Are we looking at intraday or at longterm ? Timescale is usually specified as a parameter that gets fed into your Trade project — and I used to call it Time Unit (TU)
On intraday, the TU describes how many minutes of data flow into one Candle.
Let’s look at a few scenarios on ES on different TUs to see its relevancy…
Same day, same data, same trading idea, same parameters besides time unit: three completely different behaviours with different results, no surprise. In general, no matter which TU you pick on intraday, its much harder to maintain congruency on the intraday scope, simply because the profit margin on a day-trade is usually much smaller compared to a longterm trade, relative to commission and slippage.
On longterm, I’m using the number of days to define the width of a single Candle, which allows to control the trading behaviour on pretty much any desired timescale. I usually stay at a single day to make sure I trail stops daily.
Three… Order Types
Another critical factor for congruency (and maybe the Yin whilst Slippage is the Yang) is the right selection of the Order Type. Here’s a full overview of whats possible:
An order is an instruction to buy or sell on a trading venue such as a stock market, bond market, commodity market…en.wikipedia.org
The price doesn’t matter, you can’t even specify a price limit. You want in or out and the stock exchange trading algorithm will most likely find a partner on the order book which also could be relatively far away from the current price, so it could slip. Its like going to a blind date, don’t be surprised if things go a little odd ;) However there are two scenarios where MKT makes a lot of sense:
- Positioning for a longterm, e.g. a trend-following trade: You expect to hold it for weeks or months and you really cant miss it. Even though your entry might slip a bit, that’s irrelevant and you’d pick MKT to be sure you get in
- Emergency exit: for some reason your regular stop-loss order wasn’t filled and the trade you already wanted to leave continues to go really bad for you. You feel like sitting in a crashing plane. Fortunately you still have your MKT parachute and you pull it to leave no matter what price. Forget about congruency in this scenario. Sure, slippage will be a disaster anyway
STP and LMT orders
Your decision for one of these depends on the scenario. The STP order type is useful if you need to trigger specific events to enter or leave the market at a specific price. Its default behaviour is that it turns it into a MKT order at the moment when the price (auxPrice) is reached, which gets you into the downsides mentioned above. LMT orders on the other side execute exactly at the price you wanted or better, with the risk though that they may not get executed when the exchange cant find a matching trading partner for you on the order book.
STP and LMT orders behave differently based on the scenario:
You can also pair STP and LMT to combine some of their advantages. Another thing to keep in mind is: by default orders for some instruments are entered with a restriction referred to as Regular Trading Hours (RTH) only, which means they execute only during the normal trading hours. For a 24x5 AlgoTrading project you have to specify on your order to execute also outside of RTH.
Back to our hunt for congruency, you see that there’s actually no panacea that helps to select the order type. I keep saying that, again it’s all up to your trading idea and the timescale you’re trading in. You can certainly minimize slippage with LMT or STP-LMT orders, however you pay with the risk to not getting your Trade (which might quite often be a happy circumstance anyway ;). Here’s a typical configuration I normally use on day-trading projects, this one works with a StopLoss and a ProfitTaker stop to manage the exit:
We’ve touched this phenomenon a few times already, its our Yang — essentially you expected a specific price but you’ve got something else, too bad! The difference between your desire and the actual reality we call Slippage. On a future we’re mostly talking about a minimal possible tick or two (on a regular trading day) and if you’re trading more longterm it doesn’t really hurt and you can possibly ignore it. However on intraday this is what’s killing most of the ideas that look so great in theory. As with the Order types, you have to decide for yourself (and it has to fit to your idea) whether slippage is relevant, what an “acceptable” slippage could be and whether the “acceptable” is what the market is willing to give you.
Keep in mind that “acceptable” again is just relative, its something you can assume or assert and you really need to do so to run relatively reasonable backtests of your idea, however never forget that the market is dictating the reality and the market as always is right and mostly unpredictable.
If Slippage is relevant for you (and on all intraday scenarios it will be), make sure you think about these three aspects:
- the “fool test” — include the “acceptable” slippage (and maybe even a buffer) into your parameter optimization and then backtests. This will tell you how far you can go until slippage kills your idea.
- the “adventurer test” — run your trading idea on a paper trading account and evaluate the actual real slippage you get against your theoretical assumptions.
- the “badass test” — if this still runs great, you’re possibly ready to run your great idea on your Live Account and again you evaluate congruency of your slippage assumptions
Fife… Your Implementation (the Project)
So we’ve looked at a number of critical factors that influence how theo trades get ideally congruent with real trades. However, where do theo trades come from?
Lets look at the “Modulo-Long idea” once more — we’ve learned that the trading system ideally runs in an endless loop that’s essentially processing market data from your Broker to feed a number of TradeTactics that run our diverse TradeProjects. So the project is a piece of code (could be a Java POJO or whatever language you prefer) that’s following a clear interface — since you certainly wanna be able to run different ideas in parallel, and in fact you have to for risk diversification reasons. This interface could look somewhat like this:
A project has to be generic enough to initialize with a parameter set (and TU is certainly a parameter you’ll always have). It will consume data via addDataPoint and finally provides its Results (including the theo trades and their states) and if you like a chart for visualization. This pattern allows you to run your trade project in a backtesting and live-trading scenario without any change.
To give a rough idea of how the “Modulo-Long idea” works, check out this pseudo code:
Lets assume you feed this project with
PARAM_MODULO=5 and you’d run this against YM, it would create a theo trade whenever
currentPrice%5 == 0, so it would position LONG at price e.g. 25855, 25900, 25905. Even though this might not be the best idea to generate profits, what we have done here is still pretty smart:
We’ve have implemented an idea into a project. Going back to the first chapters, you’d now have the green bubble as one of the ingredients of your trading system. Its a small piece but a very important one, since your project will be driving a big portion of the success of your entire idea.
- You have an idea, great! If you now understand that this is important but definitely not the first thing to hunt for I’ve reached my goal
- You need an implementation of your idea, hands on the keyboard buddy!
- Select the instrument, make sure it fits to your personality and your idea
- Select the timescale and other parameters you project need
- Finally tie all this together and decide on order types, account and the number of shares
Ready for Rock’n’Roll ? Well whats still missing is just this mysterious trading system I was talking about. We’ll get there in one of my next articles.
That’s a really interesting topic, isn’t it? And I’m happy to share the knowledge I have gathered over many years and hope you can grow with it.
I think I’ll make this article the starting point into a broader series focussed on AlgoTrading. I feel several of the aspects mentioned above have to be separated and need to get into the appropriate level of detail on dedicated articles. Here are the things I’ll start to look at…
- The grey eminence in your AlgoTrading system — a technical inside/out perspective of your Trading systems TradeEngine service
- The bitter pill called Backtesting — a critical reflection of my journey to find the holy grail in AlgoTrading
- Community AlgoTrading development — a strategy to lift geek garage projects into serious businesses
All these articles are still WiP & you can influence how fast they come with your feedback here & today.