Continuous deployment is hard (and do not inherit from Struct.new)

We recently had an incident at work on our main application. This is the story of why it happened and what we learned from it. The initial part of this post applies mostly to Ruby applications but the rest of it is language agnostic, so you can keep on reading even if you don’t use Ruby.

What happened

After a deployment, all our background workers were gone. Jobs piled up, error notifications started to pop up and our support team started to receive calls and emails with complaints.

After checking that the instances running the workers were alive, we then looked at the logs. The processes that run the background workers were crashing during boot.

The problem

The processes running the workers were failing with the following error:

TypeError: superclass mismatch for class Country

The stack trace told us that the error was happening inside a gem that we have used for ages. We initially thought that it was somehow a mismatch with the Country class we have on the global scope (that's common on Rails projects), but after I looked at the source code of the gem the problem became clear.