API Gateway Design Pattern Implement Rate Limiting and Circuit Breaker on Microservices
Overview
The microservices architecture makes us have many service and route addresses for handling routing endpoints and tracking every request. This complexity might be overwhelming for backend services, ideally, they should only handle business logic.
What is API Gateway
API Gateway is an application between client and service that centralizes endpoints and routes requests to microservices. API Gateway is implemented to handle requests from clients to backend services and encapsulate the complexity, sometimes it is used to handle authentication and authorization and make the service focus on business logic.
Implementing API Gateway has several advantages, such as handling service discovery. When adopting a microservices architecture, the service must be scaleable, and it would be difficult for the client to address the endpoint.
It can centralize the technology that the client uses to communicate, although you use several protocols like REST, gRPC, or GraphQL for backend services, the client can use one protocol through API Gateway.
Also, with API Gateway, we don’t need to expose the backend service to communicate directly with the client, this can reduce the risk of attacks from intruders and improve the security of your services.
Implementation
Prerequisiter
This documentation would use Spring Cloud Gateway as an API gateway. You can visit and read the documentation if you are not familiar with it yet.
github repository: https://github.com/wahyubagus-ars/spring-api-gateway
Pull the Docker image of the two service springs for testing.
docker image pull wahyubagus/user-service:0.0.1 &&
docker image pull wahyubagus/product-service:0.0.1
# run the images with these following command
docker run -p 8081:8081 wahyubagus/user-service:0.0.1 &&
docker run -p 8082:8082 wahyubagus/product-service:0.0.1cURL the user and product service to make sure the container is running well.
#user-service curl request
curl 'http://localhost:8081/api/user-service/get-user' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1'
#user-service curl response
{"status":"SUCCESS","data":{"id":1,"name":"Bruce Wayne","email":"brucewayne.corps@mail.com","age":30}}
#product-service curl request
curl 'http://localhost:8082/api/product-service/get-products' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/119.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1'
#product-service curl response
{"status":"SUCCESS","data":[{"id":1,"name":"Macbook Air M1","price":18000000},{"id":2,"name":"iPhone 14 pro max","price":20000000},{"id":3,"name":"iPad pro 2021","price":10000000}]}Generate a new spring project with api-gateway dependency in pom.xml
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
<version>2.2.10.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
<version>1.0.0.M1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis-reactive</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>Routing and Filters
Define the configuration application in application.yml, and run the API Gateway app on port 8085.
Route predicates would match any incoming request, the Spring Cloud Gateway has built-in function predicates criteria to match. And you can combine all of those.
After we match the predicates previously, this would execute the gateway filter, which can modify the incoming request and outgoing response.
server:
port: 8085
spring:
cloud:
gateway:
routes:
- id: user-service
uri: http://localhost:8081/
predicates:
- Path=/api/user/**
- Method=GET,POST
filters:
- RewritePath=/api/user(?<segment>/?.*), /api/user-service/$\{segment}
- id: product-service
uri: http://localhost:8082/
predicates:
- Path=/api/product/**
- Method=GET,POST
filters:
- RewritePath=/api/product(?<segment>/?.*), /api/product-service/$\{segment}Above, our path predicates are /api/user/**, and we define the method as GET and POST. If the request matches both criteria, it will be redirected to uri.
After matching the predicates and before this is redirected to Uri, we can modify the request through the filters. We can rewrite the path that previously was /api/user/** into /api/product-service/**. The parameter (segment) would be wrapped in regex and replaced with the parameter or continue path.
call cURL to test API Gateway, the result must’ve been the same as cURL previously.
# call user-service through api gateway
curl 'http://localhost:8085/api/user/user/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/119.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: id,en-US;q=0.7,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache'
# call product-service through api gateway
curl 'http://localhost:8085/api/product/product' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/119.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: id,en-US;q=0.7,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache'Circuit Breaker
The idea of a circuit breaker is to prevent our service from calling the remote service if the call is most likely to fail, this can help avoid overusing resources on both sides. The worst case it can cause is that our service or application is out of memory, shutting down the program.
The circuit breaker detects it’s going to fail by tracking previous requests. If most of the requests fail, it’s likely to fail.
Close state is when the request for remote service is successfully made. The circuit breaker state would be “close” and pass the request normally.
Open state is when the remote service returns an error greater than the maximal counter error that we configured. In this state, the circuit breaker will prevent calling the remote service and return an error instead. And open state will be held for the time duration we configured.
Half-Open state happens after the time duration in open state and lets pass a few requests to the remote service to check the current status of the remote service. If the error still exceeds the counter error, the state will go back to “Open”, and if it’s below the counter, it will run normally.
For the implementation, we can adjust the application.yml to implement a circuit breaker.
And define fallbackuri for the redirect API if the circuit breaker is in an open state because it failed to call service through the circuit breaker.
server:
port: 8085
spring:
cloud:
gateway:
routes:
- id: user-service
uri: http://localhost:8081/
predicates:
- Path=/api/user/**
filters:
- RewritePath=/api/user(?<segment>/?.*), /api/user-service/$\{segment}
- name: CircuitBreaker
args:
name: circuitBreakerInstance
fallbackUri: forward:/fallback/user
- id: product-service
uri: http://localhost:8082/
predicates:
- Path=/api/product/**
filters:
- RewritePath=/api/product(?<segment>/?.*), /api/product-service/$\{segment}
- name: CircuitBreaker
args:
name: circuitBreakerInstance
fallbackUri: forward:/fallback/productImplement the fallbackuri API with the FallbackController
package com.apigateway.controller;
import org.apache.commons.lang.StringUtils;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import java.util.HashMap;
import java.util.Map;
@RestController
public class FallbackController {
@GetMapping(value = "/fallback/{segment}")
public ResponseEntity<Object> fallback(@PathVariable String segment) {
Map<String, String> mapResponse = new HashMap<>();
mapResponse.put("status", "SERVICE "+ StringUtils.upperCase(segment) +" IS UNAVAILABLE");
return new ResponseEntity<>(mapResponse, HttpStatus.INTERNAL_SERVER_ERROR);
}
}For testing, you can stop one of the containers that is running (user-service or product-service) and try to call it with cURL. The response would be like what we define in FallbackController.
{"status":"SERVICE USER IS UNAVAILABLE"}Rate Limiting
If anyone can access our API as much as they want, potentially it can cause the performance to become slow, and here API Rate Limiting would handle it for reliability and less latency.
API Rate Limiting is a set number of maximum requests that clients can request from our API or service. If we set the maximum request the service can handle to 10 per second, if requests are exceeded, the remaining request would receive an error response, ideally, it would return 429 — Too Many Request.
You need to install and run Redis on your local machine and add Redis configuration for connecting in application.yml
spring:
redis:
host: localhost
port: 6379And for configuration, API rate limiting can be defined in the application.yml like circuit breaker
- name: RequestRateLimiter
args:
redis-rate-limiter.replenishRate: 2
redis-rate-limiter.burstCapacity: 4
redis-rate-limiter.requestedTokens: 1
key-resolver: "#{@KeyResolverConfiguration}"redis-rate-limiter.replenishRate: 2 — This property represents the rate at which tokens are replenished in the rate limiter. In other words, it indicates how many tokens are added to the bucket per second. In this case, 2 tokens are added per second.
redis-rate-limiter.burstCapacity: 4 — This property defines the maximum number of tokens that the rate limiter can hold at any given time. It represents the maximum number of requests that can be allowed in a short burst.
redis-rate-limiter.requestedTokens: 1 — This property is specifying the number of tokens that a client is trying to acquire or consume with each request. In this example, the client is trying to acquire 1 token.
key-resolver: “#{@KeyResolverConfiguration}” — This configuration is related to how the rate limiter identifies requests and assigns them to a specific key for rate limiting. The key resolver is responsible for extracting a key from the incoming request, and this key is then used to track the rate limit for that specific client or request.
Create a KeyResolver Bean implementation, it’s used to define the key that would be the quota for API rate limiting.
@Configuration
@Component("KeyResolverConfiguration")
public class KeyResolverConfiguration implements KeyResolver {
@Override
public Mono<String> resolve(ServerWebExchange exchange) {
return Mono.just(Objects.requireNonNull(exchange.getRequest().getRemoteAddress()).getAddress().getHostAddress());
}
}In this example, we set the key based from the client’s host address.
And we can create a bash script file to test the API Rate Limiting by looping cURL seven times. And then execute it.
API_ENDPOINT="http://localhost:8085/api/user/user"
for ((i=1; i<=7; i++)); do
echo "Request $i:"
response=$(curl -s -w "Status Code: %{http_code}\nResponse Body: %{response_body}\n" -X GET $API_ENDPOINT)
echo "$response"
echo "--------------------------"
doneRequest 1:
curl: unknown --write-out variable: 'response_body'
{"status":"SUCCESS","data":{"id":1,"name":"Bruce Wayne","email":"brucewayne.corps@mail.com","age":30}}Status Code: 200
Response Body:
--------------------------
Request 2:
curl: unknown --write-out variable: 'response_body'
{"status":"SUCCESS","data":{"id":1,"name":"Bruce Wayne","email":"brucewayne.corps@mail.com","age":30}}Status Code: 200
Response Body:
--------------------------
Request 3:
curl: unknown --write-out variable: 'response_body'
{"status":"SUCCESS","data":{"id":1,"name":"Bruce Wayne","email":"brucewayne.corps@mail.com","age":30}}Status Code: 200
Response Body:
--------------------------
Request 4:
curl: unknown --write-out variable: 'response_body'
{"status":"SUCCESS","data":{"id":1,"name":"Bruce Wayne","email":"brucewayne.corps@mail.com","age":30}}Status Code: 200
Response Body:
--------------------------
Request 5:
curl: unknown --write-out variable: 'response_body'
Status Code: 429
Response Body:
--------------------------
Request 6:
curl: unknown --write-out variable: 'response_body'
Status Code: 429
Response Body:
--------------------------
Request 7:
curl: unknown --write-out variable: 'response_body'
Status Code: 429
Response Body:
--------------------------We called the API seven times, and the results got four responses, with success was returned as 200 — OK, because we define maximal token capacity as four, and the remaining three requests would be returned as 429 — Too Many Request.
Conclusion
The API Gateway is useful if we want to separate some stuff from the service, it can be centralized here instead of in each service, and the service is only responsible for their job. And make it easier for clients to communicate with scalable services.
Thanks!!
