How we moved from Ruby to GO, decreased our cost by 1400% and increased response time by 500%

As of today, it’s been more than 6 years since we have written the first lines of the Insider’s Mobile suite. Without any knowledge of building a web application, the first challenge was picking the right platform that would stick with me for years of sleepless nights of developing new features and debugging problems.

Criteria;

1. We were in need of developing with high velocity, so the learning curve of the language had to be low.
2. The community. The community is a crucial factor that’s being overlooked, there should be lots of content, docs, people that would help with your problems.
3. High availability of third party packages — a crucial item if you are building MVP that needs to go live, fast.
4. A modern language that would get us a better talent pool, high caliber engineers who stay at the edge of the technology.

With these criteria in mind, we decided to go with Ruby with Rails framework.

RoR was perfect for us to create new features since during the bootstrap phase, we needed to build almost a new feature / product per week.

We were using Capistrano to deploy and unicorn as the http server, devise to authenticate.

For years, it served it purpose. But then..

Problem

We had lots of publisher clients. This meant a flash news story would create an enormous traffic increase on our end. That was causing our system to — roughly — per minute request counts to escalate from 30k to 400k in 10 seconds.

This was an entirely different problem to solve.

• Ruby is not designed to handle applications that need high concurrency. Rails model dictates that every request will use work process whereas processes represented and handled by unicorn workers.
• It was taking around 3 minutes for a new worker instance to scale up by an auto scale group and by the time it completed the operation, the system load would turn back to normal. It was just a waste of resources and valuable monetary resources.
• Ruby is not designed to be an asynchronous language. Sure there were lots of background workers (rails 5’s native worker support was not even released back then) *RoR was a complete memory hog. We had to write custom bash scripts which check the workers and kills them (than used https://github.com/kzk/unicorn-worker-killer for literally the same purpose)

Short Term Bullets that failed

We applied lots of patches to handle this problem,

• Pooling requests coming from SDK to minimize the number of requests that hit the Load Balancer, which was creating a data delay.
• Scale up the auto scale group BEFORE the pushes delivered to the end user RIGHT after creating a campaign, which had lots of false positive means burning money and for a new worker node to become available was already taking too long for this approach.

At that point, we knew that we had to leave RoR behind. We had to find the right tool for this job, where we needed a new language that is async by design.

Solution

After weeks of research and quick benchmark tests we decided to re-write our APIs in GO.

This marked the start of a new era for both our APIs and our wallets.

After months, RoR codebase was filled with monkey patches. A rewrite would mean a clean slate, free of the patches that were making the codebase almost unreadable and unmaintainable.

The first iteration of API was not testable and didn’t include any unit tests , the chance of implementing a test driven development was great.

Speed; HOLLY COW. Even the most basic operation performance difference is insane which was exhilarating for us which will enable us to create state of the art applications.

A short example

Ruby

numbers = []
size = 100_000_000
Benchmark.bm do |bm|
  bm.report('filling array') do
    for i in 0...size
      numbers.push(i)
    end
    puts  " array size: #{numbers.size}"
  end

  bm.report('iterating array') do
    counter = 0
    numbers.each do |number|
      counter += number
    end
    puts  " sum of numbers in array: #{counter}"
  end
end

Go

size := 100000000
numbers := make([]int, size)

fillingArrayStarter := time.Now()
for i := 0; i < size; i++ {
  numbers[i] = i
}

fmt.Printf("array size: %d took: %s", len(numbers), time.Since(fillingArrayStarter))

iterationStarter := time.Now()
var counter int
for _, number := range numbers {
  counter += number
}

fmt.Printf("\nsum of numbers in array: %d took: %s", counter, time.Since(iterationStarter))

Results

• Initializing an array and filling it with 10 million incremental number

• Iterating the filled array and getting the sum of all the numbers

Rewrite Process

We chose our first api to re-write and started the process.

First impressions

1. RoR has a built-in ORM called ActiveRecord that comes with schema migrations and these migrations were absolutely amazing to dynamically add new columns, maintain multiple dev environments without even breaking a sweat. Ex;

db:migrate # runs (single) migrations that have not run yet.
db:create # creates the database
db:drop # deletes the database
db:schema #:load creates tables and columns within the (existing) database following schema.rb
db:setup # does db:create , db:schema :load, db:seed 
db:reset # does db:drop, db:setup

For go, we needed to maintain our own structures with custom scripts and custom structures, which was hard to adopt at the beginning.

GO is DEFINITELY not magical, which makes it;

• Easy to read
• Easy to maintain

If we elaborate;

In the screenshot above, a simple initialized array of integers has 10s of functions by default!

In GO world, the only magic is how fast it is. You even have to implement the unique function yourself.

Example function to get;

func uniq(intSlice []int) []int {
    keys := make(map[int]bool)
    list := []int{} 
    for _, entry := range intSlice {
        if _, value := keys[entry]; !value {
            keys[entry] = true
            list = append(list, entry)
        }
    }    
    return list
}

Bright side of this lack of magic; you will never ever be surprised by your code. Everything is in your hand, and your hands alone.

Down side, the road of transition from Ruby to Go means lots of writing utility functions.

Deployment is unbelievably easy on go. You just run go build and that's it. You will have an executable file which contains all of your source is and packages as compiled. And again, it has a cross compiler by default as well.

GOOS=linux GOARCH=amd64 go build
GOOS=windows GOARCH=amd64 go build

As we stated in the problematic part of rails, for a new docker app to be come available, unicorn workers needs to be initialized, which was taking around 3 minutes. In the Go World, the application was able to become ready to handle traffic in a couple of seconds which was crucial in the need of scale up.

Endgame

The results were better than we anticipated.

• Before GO, we had (on average) 14 c5.xlarge (8 CPU / 16 GB Mem) instances and on each major push campaign (where the request count was increasing by %300 in seconds) we had huge outages and latency issues on all of our API’s.
• After GO, 14 c5.xlarge decreased to 2 c5.large. The number of needed scale up instance count wasn’t any need to occur since the average execution time were also decreased dramatically. The system was written with unit tests which enabled us to adopt CI and decreased our bug ratio per sprint by around 5% percent per quarter.

Check out our career page to use latest technologies with us!