Avoid N+1 Problem In Graphql Federation With GO (Part 3)

6 min readJul 18, 2021

In Previous tutorial we have manage to get the gateway and federation up and running, but it is still not optimized case it will case N+1 problem and slow down your query overtime, the the amount of data gets larger.

In this last part, I will show you a real case of N+1 problem. So let’s populate more users data and more posts and we observe how many queries to database is executed when we get all the users included the post each user has created.

So now , I have created more users, I have 5 users in my database now. let’s create post for each user.

Now, we will perform query to select all users with all their posts, and observe how many queries are executed.

In the user service, we only executed one sql query.

But in the post service we executed 5 queries. so you see, N + 1 in this case , 1 stands for the 1 query perform in the user service, and because user service return 5 rows, means 5 users, then the post service executed the “get post” for each user. So if your amount of users increase to one thousand, then the N will be one thousand, you will perform one thousand queries to database just to response to a single request. How inefficient is that, it consume alot of resource ofcourse.

Now, we are going to solve this problem with dataloaders. You can explore more about dataloaders in the docs: https://gqlgen.com/reference/dataloaders/

Let’s get started

go get github.com/vektah/dataloaden 
mkdir dataloader

So in our current root project directory in the post service, we are going to create folder named “dataloader”. Under the folder let’s create a file named dataloder.go. Think of it as our controller that will manage our dataloaders.

Above is the minimum set up for our dataloaders. Put above code to the dataloder.go file that we have created previously, so basically for the dataloader to be able to be access by the route, then we will need to implement it as a middleware that later will append the dataloader context to the main context of the GO Service, And later when the dataloader is called inside the Application , the resolver will recognize the dataloader and will be able to find where it is through the dataloader context appended to the main application context.

The CtxLoaders function is used to extract the dataloader context from the main application context which contain a struct named loaders that accommodate all the dataloaders function that can be called or used by the resolver.

Since we are going to use middleware then we need to define a router to handle the graphql routing instead of just using a simple “net/http” package to handle our route.

For the router I’m going to use “github.com/go-chi/chi”

go get github.com/go-chi/chi

We have told the router to use the dataloader middleware. Everything is set and now let’s code the dataloders.

let’s change directory to our dataloader and generate the file that will help us utilizing the dataloaders.

cd dataloadergo run github.com/vektah/dataloaden PostsLoader int []*myapp/graph/model.Post

after running both commands above you will find a new generated file under your dataloader directory.

I will explain what the second command is about

“go run github.com/vektah/dataloaden”, in this part, it’s just running the code inside the dataloaden library.
“PostsLoader”, is the name of the struct that will be used as our entrypoint to access everything inside the generated files.
“int”, is the datatype of the callback’s parameter that we are going to pass to the dataloader.
“[]*myapp/graph/model.Post”, is the path to the model that we want to use as the disired return value from the callback that we pass to the dataloader. myapp is our root directory name that we specify in the go.mod file when we run go mod init command. In this case , we want the callback to return []*model.Post type.

Now , let’s implement the callback function that will be used by the dataloader to load our data efficiently. Create new File named postsLoader.go under service directory.

Here is the callback function you need to implement.

So, basically the dataloader will wait and accumulate all the userIDs to an array, and pass that array of user IDs to the callback function. In the callback, we will utilize this IDs to query in all the posts data which corresponding to the userIDs in the array. That way, we perform fewer query to get the data instead of just doing the query n times. After getting the posts data from database, we mapped it to each user and return it as the value of the callback.

Maybe you are wondering exactly how many time, the dataloader will do the query to the database?, I am going to explain it from our case.

Before that let’s go to our dataloader.go file under the package dataloader

Add properties to the loader struct that was previously empty.

The “*PostLoader”, is the struct that is defined inside the generated file. So here , we are telling the application that the loaders struct has a property named “Posts” with the type of “*PostLoader”. Now, we need to supply the required field of the “PostLoader” struct.

Your dataloader middleware now should look like above.

wait, means you are telling the dataloader to wait before executing the callback.
maxBatch, tells the dataloader , exactly the limit of the length of userIDs array that will be pass to the callback later.
fetch, receiver the callback function that will be executed to fetch the posts data.

So, the amount of times dataloader will execute the query is.