Help the Helper: Chasing the cause of an unstable build in Travis

Do you think travis is getting slower at running your ember build? Maybe it’s your fault?

Today we experienced a total fail in our build process. We did the right thing, we all stopped what we were doing in order to fix it.

The problem was that the build got to a certain point in travis and then started getting slow. This had been happening for a week or so but we had assumed it was something going on with travis itself.

There is a bit of back story here, we had experienced similar problems to this in the past and thought it could have been an issue with PhantomJS. We thought using firefox could be a solution and for a while it was. But then we ended up with the same problem, still one “get out of jail” card left though, right? Chrome.

Again it worked but after a few weeks the problem was back. That was today. It was the tipping point, instead of just slowing down it slowed down, disconnected and then failed!

The huge amount of deprecation warnings we had yet to fix wasn’t helping with the visibility of what was happening and probably wasn’t helping the build time either. We tried profiling to tease out the source of the problem. It showed us that pretender was holding on to a lot of resources and that memory usage was increasing throughout the test suite but no more than that.

We may have done the right thing at the point of failure but we had not done the right thing from the outset. We should have been keeping our app as idiomatic as possible and where deviation was necessary understand the consequences.

We eventually put all the pieces of the puzzle together and found we had caused this problem by creating our own test helper that all tests imported instead of using the default moduleForAcceptance. We added some things that were important to us and we were calling Ember.run(this.application, ‘destroy’); but critically we were not calling the destroyApp helper. This is a helper created by ember-cli to allow add ons to add functionality that needs to occur when the app shuts down. When we had installed mirage and it had added the server.shutdown line from it’s blueprint into the destroyApp helper but it was not getting called. This was causing every test to use up resources rather than each test starting from a clean slate. The problem only presented when travis ran out of resources, the same thing was occurring locally but there were always enough resources available and therefore no problem.

You may have things like this in your own code base and i guess the important thing we learned today was to understand where parts of ember-cli are used and when we do something non idiomatic make sure we still route the execution flow correctly after we are done. In this case call the destroyApp helper after each test!