Unicorn Not Trapping USR2

One of our main rails projects at Wolox is hosted in AWS on an Ubuntu 12.04.1 LTS using Unicorn and Nginx. A few days ago we started a migration from Capistrano2 to Capistrano3 and noticed that there was something wrong with Unicorn when we tried to reload it.

This is how we reload Unicorn:

restart|reload)
 sig USR2 && sleep 5 && oldsig QUIT && echo “Killing old master” `cat $OLD_PID` && exit 0
 echo >&2 “Couldn’t reload, starting ‘$CMD’ instead”
 run “$CMD”
 ;;

What we noticed was that when USR2 signal was sent, no new Unicorn Master process was created. The process was receiving the signal, but it was not responding to it.

See also: Spoon Testing with Robotium

After some hours of digging into this issue, we realized that Unicorn was not trapping USR2 signal. This signal is responsible for re-executing the running binary, creating a new master and the child processes. From Unicorn documentation:

> USR2 — reexecute the running binary. A separate QUIT should be sent to the original process once the child is verified to be up and running.

It took us a lot of time to come across a way to fix this. The process is receiving the signal but apparently some code is trapping USR2 signal before Unicorn does. We ended up adding the following code to our unicorn.rb file in the before_fork method:

before_fork do |server, worker|
 Signal.trap ‘USR2' do
 puts ‘Since a black hole is lurking and eating USR2s we will hit http_server#reexec ourselves’
 server.send(:reexec)
 end
 …
end

We found this fix here.

See also: Four Reasons Why Rails Rocks at Wolox!

We hope this problem is fixed anytime soon, since we do not know what implications trapping the signal in the before_fork method has.

Posted by Matias De Santi (matias.desanti@wolox.com.ar)

www.wolox.com.ar