Building Scalable Web Services in Golang
Latest Update!
This article has received a lot of love over the past many years, so I’ve decided to collect all my knowledge on the topic and distill it in a detailed, interactive, and immersive learning experience on Educative -
This course will take you from beginner to pro, giving you all the knowledge and confidence you need to excel in your career. All you need is a basic understanding of Go; this course takes care of the rest!
If you have any doubts or concerns, as always, please feel free to reach out to me.
(original article below)
For the past few months, I have been neck-deep in writing and rewriting services in Golang. Be it migrating a service from NodeJs, or merely restructuring a hastily put together Go package, it’s been a lot of fun learning and developing a modus operandi that makes the dev process faster and more intuitive each time I set out. Having enjoyed this learning curve thoroughly, here are some of my learnings and experiences for others to ponder and debate.
Write idiomatic code
This one goes without saying. One of the very intrinsic aspects of the language is its insistence on doing things the “right” way. From the way golang projects are structured to the strictness in linting and documentation, every single aspect of Golang, done right, is delightfully intuitive and ergonomic.
Writing good Golang code isn’t just about expressing your thought or idea in an algorithm. If you took a working C/C++/Java function and translated that to Golang, it’s highly unlikely that would yield remotely decent Golang code. To truly be able to make the best out of what the language has to offer, one has to first understand the nuances of the language.
Check out this handy article for some quick pointers. If you have time, I’d strongly recommend going through this exhaustive documentation provided by the official website.
You can easily find plugins/extensions/linters that help improve the quality of Golang code you write, and I personally love using VSCode with its Golang extension. It’s fast, beautiful, and feature-rich enough for most use cases.
Write self-sufficient, agnostic packages
Whenever I approach the task of setting up the barebones of a web server, I ensure separation of concerns to the highest degree possible. I always have multiple small packages that are created for a purpose, and they contain all logic related to that purpose encapsulated within themselves.
Here’s a great tutorial to help you setup a basic golang web server well, and write unit tests for the same.
A package, for me, typically includes -
- Data types relevant only to that package, if any. This includes structs I would like to create to handle mutex locks on particular local resources.
2. Interface(s) to define behaviour, and functions for logic execution, most likely called on the data structs mentioned. Feel free to follow design patterns such as the Builder Pattern to help you lay out the basics in terms of data types and interfaces.
3. Functions that implement the core logic (obviously). Expose selectively the functionality that will allow others to use this package’s features.
One of the functions is often the package’s init() function, that does basic setup necessary for the package to perform its chores. Setup default values, refresh tokens, launch background go-routines, etc. Works great with the Singleton Pattern for doing one time tasks.
Let’s take the given project structure for example. It belongs to a REST service I’m currently writing. The main.go file (not shown in image) is the start of execution and imports the server from service package.
As you can see, each concern is separated into a package — rethink for connecting/reading/writing from a rethink database, auth for performing all authentication and jwt token based logic, a config package for reading/manipulating your configurable files, and so on.
Each of these packages can now be tested independently. Note the service_test.go file in the service package. You can have a test file for each package to ensure thorough unit testing.
Bottom line, always try to ensure that nobody in your service is doing something they’re not meant to do, or that they’re having to repeatedly do across multiple places.
Use an HTTP Client, or write your own middleware
I cannot stress how important and useful this point is. A lot of standard headaches that accompany serving content as a backend service can be handled using a simple middleware. You would otherwise need to write a lot of annoying code at multiple places, for eg. authentication, checking/adding headers, gzip-ing your responses, etc. These are all typical of web services, and it’s highly likely that they’ll be a pain in your neck if you don’t plan well.
Mat Ryer beautifully explains this in his articles, so be sure to go through them for a wonderfully interesting read.
At its core, the idea is really simple. You want to have a function which accepts a handler function as a parameter and returns a handler function as a parameter.
Why, you ask? Because here, in this isolated module, you can modify your request and response objects for all your vanilla chores such as authentication, headers, compression, and leave your actual handler function to focus purely on implementation logic.
This proves to be really powerful when there are a bunch of alterations you need to do for every request/response pair your process and serve. While I would recommend everyone to give writing their own middleware a shot, if you want to cut to the chase, there’s plenty of great options already out there. Check out Chi — a lightweight, robust, and handy router, that has plenty of features and middleware.
Custom http.Client turned interceptor
More often than not, your production service will need to make network calls. Network calls to other APIs, fetch resources, download streams of data, etc. Now to do that, you’d naturally turn to Golang’s http package, do a swift and easy http.Post(...) and be done with it. I’m here to tell you — DON’T.
Try not to use the default http.Client object to make your network calls. Instead, write a separate function, that defines a custom http.Client object. Most importantly, I’d like to request you to set a specific network timeout, above all else. Believe it or not, the default client does not specify a timeout. Say goodbye to your response time if that weather API you’re hitting sees downtime.
Don’t believe me? Read this well-written article that echoes my feelings.
Now, what’s this about an interceptor? Well, done right, this function will be your singular pipe, that every network call you ever make, passes through. This could involve custom configurations, modifications of request/response, redirection, authentication, sound familiar yet? This function will be your uber powerful pit stop where you could do amazing stuff, aside from just making the network call! Statistics, analytics, customization, redirection logic, you name it. The limits are enforced by your creativity.
Here’s the stripped down version of the function I often use myself. This will simply use a custom http.Client to make a network call you. You could measure response time, log metrics of error cases, modify cookies, whatever you’d like to do. Go ahead, have fun!
Learn all this and more!
I’ve poured over 2 years of hard work and a healthy amount of my sanity into my course, and I’m certain that you’ll find it immensely helpful. So Go check it out !
From one Gopher to all of you out there, keep building amazing stuff, stay awesome!
