High-performance API gateway

6 min readMay 9, 2022

At Jeevansathi, we have more than 1 million monthly active users and we strive to provide the best user experience to them. To handle such traffic, we had transitioned from a monolithic to microservices-based architecture.

The Problem we faced
Over the time as our microservices scaled, a need was felt to handle authentication of external requests at multiple microservices. This meant we had to develop and maintain the same code for authenticating and authorizing users at multiple places.

To solve this problem , API gateway proved to be a powerful pattern for managing external requests in a microservice-based architecture.

In this blog post, I will explain how I implemented an API gateway using spring-cloud-gateway and also the benefits it brings as compared to a normal web-mvc based routing service.

What are the various API gateway implementations in the market?

Spring-cloud-gateway, Netflix Zuul, Kong, Apigee, etc are some of the implementations of API gateway which can be used.

Spring-cloud-gateway¹ is a reactive API gateway based on spring-webflux and project reactor. Since it is non-blocking in nature, a limited number of threads can process the same number of requests as compared to a blocking gateway like Zuul1. This enables the use of fewer resources and helps in scaling the application.

Spring ecosystem supports spring-cloud-gateway and not Zuul2. Since our microservices architecture is built around the spring framework, using the spring-cloud-gateway was a better alternative.

How I implemented API gateway using spring-cloud-gateway?

Our requirement involved calling an authentication service from the API gateway and then forwarding or blocking the request based on its response. All external requests are received at the API gateway which passes only the authenticated and authorized request to the respective downstream microservice.

Spring-cloud-starter-gateway is the dependency that needed to be added in our build.gradle file :

//https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-gatewayimplementation group: 'org.springframework.cloud', name: 'spring-cloud-starter-gateway', version: '3.1.0'

Configuring routes :

For configuring downstream services, I needed to add only these properties in application.properties file.

service.port=8090 //port for apigatewayspring.cloud.gateway.routes[0].id=service1
spring.cloud.gateway.routes[0].uri=http://127.0.0.1:8080/
spring.cloud.gateway.routes[0].predicates=Path=/service1Id/**

Here id is any unique identifier of route and uri is where the incoming request needs to be routed. If the incoming request matches the predicate that is defined for that route, it is routed downstream else an error is returned.

For example, if we call http://127.0.0.1:8090/service1Id/demoApi where API gateway is running, it will route the request to http://127.0.0.1:8080/service1Id/demoApi where our microservice is hosted.

Configuring timeouts:

One thing I learned at Jeevansathi is that having proper timeouts is very crucial when working on a system that receives high traffic, otherwise, it can quickly snowball into multiple issues.

I configured global timeouts using the following property:

//global timeout for all routes
spring.cloud.gateway.httpclient.response-timeout= 2s

However, there was a situation where I needed different timeout settings for a specific route. Luckily, spring-cloud-gateway provides support for handling this use case as well by using the following property:

//timeout for one specific route in ms
spring.cloud.gateway.routes[0].metadata.response-timeout=3000

Configuring filters :

There are mainly three types of filters from which we can choose based on our requirements.

GlobalFilter² — Applies to all the routes but not the controllers hosted in the gateway. Spring provides many inbuilt filters for normal use cases and if required we can implement custom filters as well.

For example, we can apply an inbuilt filter ‘AddRequestHeader’ that adds the header ‘request-header’ in all requests with the value ‘request-header-value’ in the following way:

//Apply filter for all routes using springs inbuilt filter //"AddRequestHeader"spring.cloud.gateway.default-filters[0]=AddRequestHeader=request-header, request-header-value

2. GatewayFilter³ — For configuring specific routes, we can apply a route-specific filter in the following way:

//Apply filter for a specific route using spring inbuilt filter //"AddRequestHeader"spring.cloud.gateway.routes[0].filters[0]=AddRequestHeader=first-request-header, first-request-header-value

3. WebFilter⁴ — Applies to all routes as well as the controllers defined in the gateway by us. We need to implement the WebFilter interface in our custom filter and then write our custom logic by overriding the filter() method in the following way :

@Component
@Slf4j
public class CustomFilter implements WebFilter {@Override
    public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
        log.trace("inside custom filter");
        ServerHttpRequest request = exchange.getRequest();
        // logic for request validationreturn chain.filter(exchange);
    }
}

For implementing the API call to the Authentication service, I used WebFilter as it could do authentication for requests with controllers defined in the API gateway as well as routes.

The usage of reactive HTTP clients like WebClient helped in handling the overall request reactively, even though our downstream microservices were blocking in nature.

Since this reactive approach involves a limited number of threads to handle all requests, blocking any thread will cause the entire application to come down. To avoid this, we used a tool called Blockhound⁵ which detects any blocking call and throws an exception.

Load Test

For load testing, we mocked a fixed delay API in the downstream microservice. The comparison was done between a web-mvc based application and a spring-cloud-gateway-based application that uses spring-webflux. We first called the fixed delay API from API gateway and then through web-mvc based application for load testing.

Key points :

The response times of both are almost the same at lower concurrencies(~100). Since the fixed delay API is a blocking API, the response times were not expected to be different.
At higher concurrencies, the API gateway performed 2x better in terms of throughput and 99 percentile.
This can be attributed to the reason that tomcat is limited by the max number of threads that can be opened but the API gateway runs on a completely different event-loop threading model, so it can handle significantly more requests.

Since our downstream APIs are blocking in nature, we did not expect the response times to be different. What we expected was that reactive gateway will be able to have better resource utilization as compared to non-reactive because of the different threading models used, and this was confirmed in the load test results.

For reading more on how reactive programming works, refer here.

Challenges

Shifting to writing code in reactive pattern using operators provided by spring requires a change in how we think about request flow.
One challenge we faced was that logging based on MDC cannot be done in a reactive application. This is because MDC is based on Threadlocal and in a reactive world, a request is not necessarily tied to a particular thread.
To overcome this, we made use of the reactor context⁶ which is propagated throughout the reactive pipeline and wrote custom methods for logging.

Conclusion

To conclude, an API gateway is an important component in the Microservices architecture pattern. It accelerates the development workflow of backend teams as they can now focus on coding business logic instead of focusing on authentication, monitoring, etc.

In this blog post, we focused mainly on the routing aspect of API gateway but there are many other functionalities like rate limiting, service discovery which we will explore in the coming blogs. Till then, stay tuned.

High-performance API gateway

What are the various API gateway implementations in the market?

How I implemented API gateway using spring-cloud-gateway?

Challenges

Conclusion

References

Written by Jeevansathi Engineering