Making Xplan Small Again
“Less is more” has been heavily embraced by the world of design and architecture for over half a century. The world of software – particularly enterprise software – has, to put it mildly, been slower to catch on.
Truth is ever to be found in simplicity, and not in the multiplicity and confusion of things.
Sir Isaac Newton
Xplan is a software platform for tens of thousands of financial services professionals globally including financial advisers and private wealth professionals. It’s a functionally rich, broad and comprehensive product suite that is flexible and scalable to suit a spectrum of business models both as a specialised or integrated end-to-end solution. Xplan originated around 18 years ago.
And in that time, according to line-counter, the main repository has grown to a formidable 2.7 million lines of Python code. That makes Xplan a serious contender for being the largest Python codebase on the planet. While there has been continual development and enhancements, much of the original architecture remains, and in current terms it would be classified as a “monolithic architecture”. Any opportunity to make a large monolith smaller should, in my opinion, be seized.
As a software engineer at Iress in the Xplan Portfolio team (affectionately known as “The A-Team”), I work heavily with the part of Xplan that’s responsible for communicating with Iress’ Portfolio System.). For both performance and historical reasons, Xplan talks to the Portolio System using a collection of custom in-house transport protocols and messaging formats, rather than using web-based APIs over standard HTTP.
The utility code that powers this communication has, for most of Xplan’s history, been embedded in the main codebase. This means that a whole lot of code that has nothing directly to do with financial advice has been coupled together with actual business-centric code. It also means that if anything outside the main Xplan app wanted to talk to the Portfolio System, it would have to reinvent the wheel and implement all that utility code again. That latter point was particularly concerning because here at Iress, we have begun moving down a path of building microservices; but we can’t build microservices that need to call the Portfolio System if we don’t have a reusable library to facilitate such calls.
And so I, bravely, foolishly, naïvely, took it upon myself to split this functionality out into a stand-alone library.
I began the endeavour in early 2018. Because I still had the usual features to deliver and bugs to fix in the meantime, all work on it had to be confined to my downtime, which – for me, as for most Iressians – is never all that plentiful. I actually got a bare-bones working prototype built fairly quickly, within a month or so: I gave it the super-original name ips-client-python. It wasn’t mighty pretty, it didn’t support everything that was needed for an MVP, and it didn’t integrate with Xplan. But it was enough for a stand-alone Python script to talk to IPS, something that had never before been possible.
It took me another good six months or so to get ips-client-python feature-complete, and test all the configurations that it needed to support. (Warning: gory tech details below — if they start to make your brain hurt, feel free to skip down to “Phew!”).
In particular, it needed to be able to talk to our Portfolio System using any of four different transport protocols, only one (the easiest one) of which I had gotten working for the initial prototype:
- Proxied RPC: a standard TCP socket connection (specified by server name and TCP port number) to a daemon called
ipsproxy, and from there an MS-RPC connection to IPS itself (the easiest one, works on Windows and *nix) - MS-RPC: an RPC connection (specified by server name, and by named pipe or RPC port number) directly to the Portfolio System (only works on Windows, as it relies on the custom
_msrpcPython library that Iress built, and that only compiles with MS Visual C++) - CyORH: an ORH (Order Routing Hub, an alternative to TCP/UDP developed internally by Iress) connection (specified by ORH address, which is somewhat similar to but not the same as an IP address) directly to Portfolio System (only works on Windows, as it relies on the custom
cyorhPython library that Iress built, and that only compiles with MS Visual C++) - orhbroker: a ZeroMQ connection (specified by server name and TCP port number) to a daemon called
orhbroker, and from there an ORH connection to IPS itself (works on Windows and *nix)
For the messaging format, Xplan talks to Portfolio System using SDO (Service Data Objects), which means (in this context) a custom Iress implementation of the SDO official spec. ips-client-python needed to support two different SDO wire formats (in combination with any of the above transport protocols):
- XML: requests / responses sent as XML text, either using Python’s built-in XML libraries (Windows and *nix), or using the
_sdoparserIress custom XML library (Windows only) - Binary: requests / responses sent in a custom binary format, using one of two Iress custom libraries,
pysdo(Windows and *nix) orcysdo(Windows only)
Phew! That’s a whole lot of Iress in-house jargon, and my apologies that the gory details above won’t mean much to anyone who’s not a dev at Iress. But I thought I should include it all, in the interests of completeness and transparency, and in order to provide a sense of what ips-client-python needed to support, and the challenges that were involved. Anyway, once I’d gotten it working with all those different protocols and formats, it was time to polish it until it gleamed.
If you’re going to undertake a task like this, you have to make it worthwhile. And a big part of that is making the final product much higher in quality than the original. So that’s what I did. ips-client-python is fully Python 2 / Python 3 compatible.It has 100% unit test coverage.. It has good documentation, particularly its comprehensive ReadMe. It has been thoughtfully refactored (to what I hope is a good level, while not going overboard). It verifies and builds in CI from GitHub to Artifactory via TeamCity (moving to BuildKite) & SonarQube in under a minute (Xplan itself typically takes about an hour to do the same, on a good day). It supports everything the old code did (because it has to), and it supports some cool new things too (just because it can).
The final task, a year or more since the endeavour began, was to make Xplan use ips-client-python, and to then remove from Xplan all the code that it no longer needed, which in the end was around 5000 lines of code. This was no small feat. In particular, apart from the not insignificant dev work involved there was a huge amount of testing required, both manual and automated. Testing of every major configuration. Testing in multiple OSes. Testing locally. Testing in UAT. Testing in CI. Test, test, test. Why so much testing? Because this was a big change and Xplan is an Iress flagship product, used by thousands of financial professionals every day, and it kind of needs to not fall flat on its face. So when a big change does hit the codebase, here at Iress, we take testing and quality seriously!
In order to allow for the right period of regression testing, we merged the ips-client-python change to Xplan in master, but then reverted it from the release branch that we (currently) create weekly, for a period of about a month. This allowed us to test the change in our UAT environments that run on the latest master while avoiding any issues in the actual releases that get deployed to our end users.
However, part and parcel of undertaking a change like this, is accepting that, no matter how much testing you do, bugs can still make it into production and if they do, they will fall squarely at your feet (as they did for me), and they will need to be fixed urgently (as I did). But never give up, never surrender!
After more than a year, my ips-client-python endeavour is finally complete. It has achieved its main objectives which were to allow software outside of Xplan to talk to Portfolio System and supporting our strategy, to take a step in simplifying Xplan. This has paved the way for future improvements, and it has improved the quality of our software. That’s good news for Iress, its clients and users.
