Engineering at TenX
It’s been a couple of months since I started my new role at TenX as VP of Engineering (and Product), and when Julian asked me to do an Engineering Q&A, I thought it’d be helpful to reflect on some of the challenges and decisions we made through the first half of 2018.
A bit about me — my name is Alvin and I’ve been at TenX since October 2017. My first couple of months was ramping up, getting things organised and basically settling in with the team. I was previously from PayPal and had actually left to be a Venture Partner (my role as Head of the PayPal Innovation Lab prevented me from pursuing this whilst employed there). I continue to dabble in venture funding, but my focus is on TenX, where I work with a team of around 17 engineering and another 5 product colleagues to deliver our vision of Making Crypto Spendable Anywhere.
At the time I joined, the team had about 7 people, and the key project was scaling in anticipation of a huge surge in demand due to a German TV spot. We had cards “live” at that time and a little under 100,000 users on the platform. The system was working fine, and the team figured that horizontal scaling of the javascript monolith would allow us a bit of time to get past the surge, but very likely not serve millions of users.
By January we had another 4 engineers, all dedicated to rewriting the monolith (an excellent MVP written by founders Toby and Michael, that actually scaled to 6-digit users with minimal refactoring) in Rust. But then we lost the cards, and so we decided to take advantage of the “crisis” to build out a truly scalable distributed system.
In hindsight, that was probably a little crazy: our choice of Rust was unproven, and we had little experience with distributed systems. Doing both for the first time introduced a fair bit of uncertainty (in terms of difficulty and timeline), though this was offset somewhat by the safety features Rust provides. If it wasn’t possible, it wouldn’t be possible — we were not too concerned that we’d end up building something that only half worked. Taking on this engineering risk was a luxury that the situation (both the crisis and the strong MVP) afforded us. We were no longer fighting fires and trying to squeeze that little bit more out of the monolith.
Fast forward a couple of months, where in between working on this we put together a cold storage system, made major changes on the Mobile app, added heaps of small features, became PCI-DSS compliant and began technical integrations with at least four different Issuing-partners…
…And now halfway through 2018, we’re finally looking forward to deploying the first of our new microservices in production!
We considered a few options for the first service: Without a lot of experience and a smaller team at the time, we tried first to replace a small peripheral part of the monolith (this was in December, when we still had cards live and were afraid of breaking the monolith). This didn’t work very well; it was not seen as particularly useful or important, and more urgent issues kept pulling engineers away from doing this. Besides, the function-being-replaced was working fine, and worse still, if we were going to do anything to it we really ought to improve it to do a whole lot of other things on the backlog! We abandoned this approach after a few weeks, though the work was not wasted.
Attempt #2 — triggered by losing our Wavecrest cards, we decided to go right at the heart of the monolith. The question we asked was: “What would create the most business value as a service?” Two candidates were immediately obvious: User balances, because it is the single most important thing we needed to scale (every user has a balance, even if they don’t have a card), and it is also the one service that would benefit most from a re-write in idiomatic Safe Rust.
The other was a set of functionality broadly grouped as “Payment Authorisation”. Split out as one or more services, this functionality would benefit from being re-written in a high-performance compiled language. It was also the perfect time to make such a transition, as it would be one of the most complex transitions we would have to manage if users had to be able to continue making multiple transactions per second.
We decided to do both (hindsight: focus on one). User balances, because we felt that we had get deep into the core functionality of the monolith, bring the whole team together to move forward (rather than continue with one team trying to improve the monolith and another trying to tear it apart) and also because engineering did not have full control over partner timelines. Payment authorisations, because we were confident that our business development and product teams would quickly secure our card product roadmap, and we wanted to have a new architecture in place by then.
Some of the lessons we learnt:
- Start with one, not two.
- Find the problem with the most value.
- Encountering challenges up front makes everything simple afterwards.
- It’s not just technical; that big, scary problem of extracting complex core code into a standalone service also represents the team’s dynamics. Get everyone on board and you’re building a winning distributed team.
I hope some of these lessons are useful.
So which service are we launching first? Will it be the boring and safe User-Balances, or are we finally ready to get cards back out with our brand new payments authorisation service? I’ll give you a hint: We’ve been assigned a BIN number!
There’s a lot more in Engineering that I hope you’ll find interesting. The systems described above was implemented by our talented engineering team (not me). In the coming weeks I’ll be introducing the team in future blog posts so that they get to tell you directly about the work they do, and the awesome personal and professional achievements they’ve accomplished within this short time.
Alvin Jiang
VP Engineering & Product
28.06.18
