Go Heatmap Yourself

Recently we introduced a heatmapping feature on Strava, our first feature built using the Go programming language. This is part of work to move Strava towards a service-oriented architecture with well-defined interfaces that can be written in any language.

The service has two main components: the front end host and heatmap generating workers. Our main code base interacts with the front end: requesting to build, view and update maps. The front end then delegates jobs to the workers using a Gearman queue. The workers aggregate the activity data, generate the tiles, and upload everything to Amazon S3. Metadata is saved to DynamoDB and tiles are served to the user via CloudFront.

Pretty simple architecture, so why use Go? Well, we wanted something with a few key features: a built-in webserver to accept API commands, easy tie-in with existing C code, and speed to eliminate any overhead from an inherently slow aggregation process. A prototype in Ruby was painful, so we looked elsewhere.

Go fit the bill with its net/http library, cgo and compiled binaries, but it wasn’t all roses and lollipops along the way. First, Go lacks mature, fully supported client libraries. There is code on Github for just about anything, but very little of it seems “production ready.” As a result, we spent a lot of time building and testing our own libraries.

Another issue we ran into was a performance hit from cgo, fortunately we found a workaround. While very easy to use, there is a slow context switch when switching between Go code and C code. For example, during the aggregation step the Go code feeds the lat/lng points to the C code using something like:

for _, latlng := range latlngs { C.hm_addLatLng(latlng[0], latlng[1]) }

This worked, but when we modified the code to something like:

for i := 0; i < len(latlngs); i+=10 { C.hm_addLatLngBulk(latlngs[i][0], latlngs[i][1], latlngs[i+1][0], latlngs[i+1][1], ..., latlngs[i+9][0], latlngs[i+9][1], ) }

we got a 30% speed increase over this data aggregation step. That means that at least 30% of the runtime was spent making the function call, not doing any work. Something to keep in mind for the future.

Personally, I’m a fan of Go and was happy to have used it on this project. I enjoy the C style and Go makes most of the mess of C easier with a built in string types, slices for growable arrays and garbage collection, all while compiling into a single binary. We still have more to learn about Go, and time will tell if it was indeed the right choice for this project.

And as always, if you’re interesting in working on cool athlete-related projects like this, check out the open engineering positions at Strava.

Originally published at labs.strava.com by Paul Mach.