Go types, an error story
Bugs are often infuriating, but often the most infuriating bugs turn out to be the most interesting. I’m going to discuss in brief detail, a bug which caused a lot of head scratching. But we learnt a lot in the process, about Go’s type system and how some of the popular libraries in Go deal with some of the ambiguities.
One of the features we work on is a web-based SQL IDE. It’s pretty cool actually. It hooks into that users Redshift data warehouse and allows them to securely write queries and view the data in a nice format. Also outputting the data into various locations, and in various formats.
It’s a brand new feature, so we’ve been doing lots of improvement work and debugging. When we were testing with some real data, we noticed that numeric columns were appearing as base64 encoded. Weird! Really weird.
We did some digging and noticed that the base64 encoded data seemed to be coming from the Go service which ran the queries. We placed some logs here and there, and noticed that the query was returning `[]uint8{51, 48, 48}` instead of the number `300`. Then, when we attempted to marshal the data into JSON to send back to the browser. That byte array was being converted into a base64 string.
I’ve been using Go for a while, but was pretty baffled by these two strange behaviours. I wasn’t entirely surprised that the query was returning something strange. Because the queries could be anything on lots of different tables, we couldn’t pre-define the data types we expected back and cast the response back to a struct. So I had to use a lot of pointers and interfaces to be able to parse the response.
I did some digging, spoke to a few people in the Go slack channel. It turns out that the byte array, returned by the `lib/pg` Postgresql driver is because the author rightly didn’t want to assume what type of integer you were expecting, as converting between the types could cause inaccuracies and imprecisions. The driver returns an array of characters, for each digit in the numeric value returned from Postgres. In my case `51, 48, 48` -> `300`. So all I had to do was check whether or not the value was a []byte and simply wrap it as a string `string(rawVal)`. That’s one mystery solved! But why was it returning a base64 string?
After doing yet more digging with my team, the `json.Marshal` function doesn’t know what to do with a byte array, so it assumes it’s binary data. The Marshal function doesn’t have an exact match against an array of bytes, in terms of type conversion, so it converts to a base64 in an attempt to preserve the data instead.
So there you have it, this bizarre behaviour actually makes a lot of sense. It’s also because we were attempting to deal with loose types, by using lots of pointers, interfaces and reflection, which you should always use with caution in Go. It is after-all a strongly types language, so any attempts to ‘get around’ this, can lead to trouble without caution!
View more articles like this in the Peak Content Hub.
