Hacking at Oscar
Developing a Thrift wrapping solution for Go during Oscar’s hackathon
Twice a year, Oscar engineers, product managers, and designers put all of their core projects on hold for our hackathon. In true hackathon spirit, we compile and upvote each other’s ideas in a Google Doc, assemble in teams, and race to build things in 24 hours or less.
Over the years, Oscar hackathons have produced an exciting array of inventions that endure to this day: a famed lamp at our HQ that is triggered by claims data and lights up in pink or blue when a new Oscar member has a boy or girl, a Google Chrome plugin that allows us to anonymize user data, and a web page that provides company-wide visibility into our website deployments.
My hackathon project focused on relieving a less visible engineering pain point: engineering efficiency. I sorted through my backlog of complaints, and landed on a quirk caused by using Golang (Go) with Apache Thrift. We were having to write a lot of repetitive code to translate Thrift-generated Go code into a more natural language mapping, which increased the likelihood of introducing bugs to our repo.
The challenge of using Apache Thrift with Go
Three years ago, Oscar decided to move to a more service-oriented architecture in order to work in a more language-agnostic manner. As we built increasingly complex and interwoven systems, multiple apps and consumers had to make requests of the same services. We wanted to build our infrastructure in such a way that allowed these services to operate in isolation from other applications and services that consumed them. As a result, we decided to use Thrift as an independent data serialization format. Thrift let us:
- Maintain our service definitions as an independent resource.
- Generate server skeletons in our language of choice, to build on top of.
- Generate language-specific libraries that clients of the service could use as well.
Thrift was appealing because of its simplicity, but that’s also what can make working with it frustrating. The client and server implementations that Thrift generates are bare bones, and any enhancements have to be figured out separately.
A short while after adopting Thrift, Oscar started using Go. I hadn’t used Go before Oscar, but I quickly came to appreciate its strengths. It’s strictly typed with limited features, and easy to comprehend (walk through https://tour.golang.org/list, and you can understand any Go code). But as with any coding language, there are trade-offs. Go is a typed language without a solution to generics, which is like being back in the Java stone age.
The clash of these two issues led to a lot of repetitive code. Repetitive code leads to bugs in production when we update one version but forget the other exists. For example, we generally support the same retry mechanism on any client call made to a service. To address this, we built as much as we could into a common client base and ended up with the following wrapped client pattern:
// MultiplicationServiceRPCClient implements MultiplicationService with RPC-specific logic.type MultiplicationServiceRPCClient rpc.Client// NewMultiplicationServiceRPCClient returns a new MultiplicationServiceRPCClient.func NewMultiplicationServiceRPCClient(
transportFactory rpc.TransportFactory,
options ...
rpc.ClientOption
) *MultiplicationServiceRPCClient {
client := rpc.NewClient(transportFactory, options...)
return (*MultiplicationServiceRPCClient)(client)
}
The only difference between clients was stylistic: “MultiplicationService” could have been any word or even the same word. Each decorated client only supports the methods for one Thrift service, so this definition was created each time.
The methods themselves were also almost exactly the same as shown by the decoration on a Multiply method below:
func (c *MultiplicationServiceRPCClient) Multiply(n1 *services.Int, n2 *services.Int) (resp *services.Int, err error) {
err = c.Retrier.Do(func() error {
client, transport, innerErr := c.getThriftClient()
if innerErr != nil {
return innerErr
} defer transport.Close()
resp, err = client.Multiply(n1, n2)
return err }) return}
The only difference in each implementation was what the line “resp, err = client.Multiply(n1, n2)” became once it was translated into Thrift.
This inspired us to build a code-generating solution in Go — one that generated wrappers around clients and servers based entirely on the Thrift definition. In doing so, I learned about the language’s rich code generation support.
The Hackathon
Day 1, 1 p.m.: Kickoff
I found a native Thrift package for Go that already parsed the Thrift definition into a Go struct. I roped in Matt Dee (every project is better as a team, even if you do have to spread the glory). We determined that we could use the Go parser in conjunction with Go templates (which implement data-driven templates for generating textual output) to create a workable solution. Neither one of us were familiar with the template library, so this seemed like a pretty convenient opportunity to add it to our tool belts.
We pulled in the parser library and got to work. Shortly after, we learned that we didn’t have the Go translation for Thrift definitions — we’d have to figure out ourselves how to generate Go code out of Thrift. Still, it was a hackathon, and this sort of algorithmic problem is fun. It’s what engineers wish there were more of when asking interviewees boring questions about the longest palindrome.
We did have a couple points working in our favor for this project:
- First, we didn’t need a perfect translation. Because we performed code generation into the repo, implementation errors would just fail to compile. We were blocked on code generation until the issues were resolved, but services didn’t go down like they would if we were developing a run-time solution. Hooray for iterative development!
- Second, we had a good repository of Thrift services built up over years of development — the beauty of microservices! If we could handle those, we felt that newer services and definition updates were unlikely to cause the apocalypse.
Our first goal was to take one Thrift service definition and generate a wrapper for its client.
Day 1, 2 p.m.: The weeds
After lunch, the development and collaboration process began in earnest. Soon, we formed a shared bank of numerous commits with helpful messages ranging from “uhm” to “abcde” to “please work”.
Then we dug into the Go templates. The library had a rich set of examples, but we still hit learning curve problems. Slight derivations from the examples led to confusing results. They did allow us to define clean templates without feeling like we had to hack in what we wanted. As we got more comfortable, the template logic became cleaner and we eventually ended up with gen-client. The verdict: it was definitely a library worth adding to our tool belts. Once we got the hang of the templates, we were pretty taken with their power.
Day 1, 6 p.m.: Hope
Eventually, we did find our way to a working solution. We then doubled down on optimism and generated all of our services. It didn’t end well. We ran into a couple problems:
- First, the Thrift generation for Go follows initialism style, but our Thrift definitions did not. So we lifted a key bank of initialisms directly from Thrift’s repo.
- Second, we faced a magic word problem. Even though Args and Result aren’t Go keywords, as far as Thrift is concerned they are. The solution? Slap an underscore after them.
Day 2, 10 a.m.: Success
We knew we weren’t handling optional Thrift fields completely. In some cases, they would generate bad code. We also hadn’t gotten around to handling enums in service definitions at all. But time was running out and our bank of Thrift definitions was generated, so we moved on with our parse solution. On the plus side, we had more than a prototype — we had a working solution.
Day 2, 4 p.m.: Judgement time
As time dwindled away and the judging hour approached, we decided to stop building. We cleaned up our code up as best we could and prepared ourselves to present the code-generating solution. We didn’t win, or come runner up, but today, more than half a year later, it’s still the solution all our Go code uses to wrap Thrift. The only update? More generated functionality. You can find a full example on GitHub here.
