My InvoiceXpress Postmortem

8 min readApr 8, 2015

I still remember the day I started on InvoiceXpress. I’m that kind of guy that’s never late, and on my first day because of train schedule miscalculation, I was at work one hour later. I was worried but no one really cared and it showed me the relaxed nature of this company. I always wanted to work on IX, it’s a sexy Portuguese startup that I had always followed.

I had never touched a Mac and I never had done any Ruby. And there I was with this new alien OS learning a new exciting language. I got some onboarding by the team and was being productive in a couple of weeks.

The first months are always quite motivating. I was learning a lot of stuff, trying to adapt to OSX, learning new ways to develop using ruby on rails, and learning a new product that is quite big and filled with features.

In a couple of months I was a one-man-army on the product. I would have some help from time to time and there was someone more experienced on the product that would help, but I was the guy.

A darker side of InvoiceXpress started to emerge. Through the years, the product had seen many developers with a mindset that would favor fast time to market against quality.

While I might understand the need of a startup to ship fast and start earning some money, the truth is that it was used has an excuse to ship poor quality code. And that was starting to show.

The project had no unit tests, it would take a couple of days to install on a new machine, and a vast quantity of duplicate code.

It was common to ship something on feature A that would break feature B (and C, and D…). At that time I remember doing one of my top mistakes ever. Because of a bug of mine and the lack of tests and quality that I was inheriting, a customer had a huge problem with their data. I’ve had enough and told management that I wanted to leave to greener pastures.

A new mindset

Management wouldn’t want me to leave, so they basically offered me technical leadership on the project. I would do it my way. At that time I also had a rookie coming to the team, Quim. It became my mission to make IX a great project that would make us proud. It was also my chance to start leading a team, something that I had never done before.

We started focusing on quality and learning new ways to develop. We started writing unit tests and added full integration tests with cucumber.

We stumbled on clean architecture and started using interactors and presenters. Also started a new test suite with only fast rails-free unit tests. We moved from SVN to github and bumped a version on rails.

Some time later a new team member: Paulo. Now we were a team of 3 developers and 1 power-designer.

Technical challenges

We faced several challenges. One day we got a call from our hosting provider, saying that they would disconnect our machine, put it on a plane, and move it to a data center in Spain. The downtime would be about 10h.

This was unacceptable to us and force us to move fast to something we were already planning for: EngineYard. But the product wasn’t ready for Engine Yard’s way of doing things. We relied on account images on the hard drive, our web workers communicated with delayed job workers via files on disk, and we had a pretty old stack and no way to properly configure a new server.

This moved us to start making a bootstrap script that would install/prepare all the app. At the time our rails didn’t even support bundler.

We also hoped that this move would fix some issues we had with fulltext search. We were experiencing downtime every time someone would make a text search. We relied on mysql’s fulltext and it was giving us a lot of problems.

The move went well, but the fulltext problems escalated. Apparently, the problems were somewhat mitigated because we were running on a top machine with SSD disks. On the cloud, we didn’t have so many resources and the app hanged all the time.

Technical debt is a bitch. And it was biting us hard.

The makeover

I still don’t know how we convinced management to do a major makeover on the app. I believe that the new awesome design that Mari presented was the selling point.

We started rebuilding the app, cleaning up the views and replacing them with clean views that used presenters. We started to move logic from the models and the controllers to interactors and writing tests for it. We started to use pull requests and all the developers had to review them. We removed mysql’s fulltext and started to incorporate elasticsearch.

This was a huge effort that took about a year.

It was too long. The work we had to do was way bigger than we expected and we still had to maintain the app in production. I’m very proud of what we accomplished, but if today, I would have done it differently. I would have divided the efforts in milestones and deployed several services.

The team was becoming quite productive and we were learning a lot. At this time a new rookie entered our team: Cláudia. I no longer considered myself as the leader. Paulo and Quim were already top performers, and I saw them as equals. I was delighted while I stepped back and watched them introducing Cláudia to the project.

However, it was hard to sell the quality/testing of the new version to management when we were taking so long. This brought some tension and stress to the team, and I believe that ultimately led to Quim’s exit to Talkdesk.

Makeover’s deploy

How could we easily deploy the new version? The version was so different, the makeover’s branch was 5k commits ahead of master. This was pretty critical.

We started testing having both apps running on the same database. The new version had some new features, but it didn’t mess around with the old one and we could have them both running side by side. To achieve this we had a precious help from Engine Yard’s staff.

This made the makeover’s deploy very easy. We put it live, and no one was affected because all accounts were running on the older version. We implemented a way to match each account to a version and started putting trial accounts on the new version. This was our main release.

After that we would migrate accounts and would be fixing bugs. We did have some problems, mainly because we left behind some features that were crucial to some accounts.

Technical issues reported, from the makeover’s deploy to this day

As expected support rouse. Our elite support team was overwhelmed with cases. We are fortunate enough to have an elite support team. I’m talking about people with no technical background that have learned SQL and do a lot of technical support.

All is well now. We kept doing fixes, writing regression tests and at this moment things are stable.

New tech stack

All my experience with InvoiceXpress made me consider ruby and ruby on rails as a development framework. We had so much bad code, so much slow code, so many problems, that I started asking myself if I could have better. I started learning scala and functional programming and ultimately fell in love with clojure and react. But that’s a story for another post.

It’s all about the culture

I believe that the main shift in mindset, started a culture of quality that today is already providing InvoiceXpress with great results. The last 3 months, boosted by Fiel’s focus on the top process, and Paulo’s focus on async communication, has made us even more productive and happy.

We Are SWAT and it’s great to work here.

We are still learning a lot and constantly changing our development process, but at this moment I think we are at a peak. We are now focusing on productivity and having the right things done.

We have Joana, our product owner. She’s in charge of the backlog grooming. Together we eradicated one line backlog tasks implemented new rules and processes. Joana will write the specs for a task: she will do the research, ask for input of the support crew, ask for wire-frames from the designer, etc. Everything that is need to build a task with a proper spec.

At some time she will also call the developers to the discussion, and we will make a more technical review of the task and add an estimate. When all is done, the task is ready to be on the backlog. This analysis is giving us a great boost.

Now we have a great culture, great team members and a management that fully supports us. The code’s quality is increasing and we have 1700+ tests. We can prepare a new machine in about 20 minutes with automated scripts. The company embraces remote and lifestyle quality and I love being here.

Even so, I’m leaving InvoiceXpress.

What’s next for InvoiceXpress?

I leave behind a great team that is on a fast track to success. I’m so proud and so eager to see what they will do without me, how will they adapt and flourish. I believe that sooner rather than later I will be learning a lot more from them.

Soon Sergei, our new team member from Russia will begin working on InvoiceXpress and will bring a lot of experience to the team. It will be very interesting to see how the team will adapt on having a non-Portuguese element.