System Design: API Gateway + Backend for Frontend(BFF) + Cache Stampede + Circuit Breaker

API Gateway:

Why Companies are adopting API Gateway in their Architecture?

As most of systems are build with Microservices Architecture i.e decoupled services with specific objectives. Frontend systems needs to interact with different Microservices to get desired data, but with this interactions we may problems. Some are listed below:

1. If we have n services then to make them accessible, all services needs to be exposed publicly(Public IP). This can add security concerns for the services.
2. Let’s take a scenario when UI needs to talk to multiple services to get content to be rendered. To get this done, we need call multiple services in parallel or serial order as needed. These multiple calls will add Network Latencies as for each call full network path needs to be traversed.
3. As most of systems needs to implement Authentication + Authorization for security. With Microservices architecture we can have have this logic on each service or we can create one service that can be called from each service. This leads to additional overhead on each service.

Above Challenges can be eliminated or reduced with API Gateway Integration. But, Before moving on how API Gateway will resolve above issues, lets first see what actually is API Gateway:

API Gateway can be considered as separate service which can act as middleware before the call is made to backend services(contains application).

Lets see how API Gateway will resolve above problems:

1. We can create separate secured network with Gateway Service and Other Services in it. From UI, we will call Gateway Service only (expose it Publicly) and other services will be called from Gateways( that can be kept Private). With this we can eliminate the Security Problems with API Gateway as Gateway have only access to other services in secured network.
2. For fetching data from multiple services, we can made a call to API Gateway that internally makes multiple calls to services and return aggregated data. As multiple calls will be originated from Gateway Service to Other services which are on same network that in turn significantly improves Network Latency and also helps in Scaling.
3. As calls will backend services will made via Gateway Service so keeping Authentication + Authorization logic on it handles user security problem globally for whole system.

Few Other Advantages:

1. Service Discovery: If we need to add new services in our system then it will made available to API Gateway automatically using Swagger or some other mode. Frontend only need to call API Gateway that will internally route to correct services and no additional logic or connection is needed.
2. Caching at API Gateway: This can helps in improving performance as well need to eliminate more business logic overhead at services end.
3. Circuit Breaking Capabilities at API Gateway: This helps us in eliminating unnecessary calls when services is down due to some problems. This helps in reducing unnecessary resources consumption by fallback mechanisms.
4. Logging: We can log data at Gateway Service as all the requests and responses will go through it only.This data can be used in debugging as well in monitoring.
5. Load Balancing: Having load balancer at gateway level helps in having a separate logic/balancer at each services level.

Few Famous API Gateway Available are: Kong(Azure), AWS API Gateway, Ambassadar, Ocelet.

Backend For Frontend:

BFF is almost similar to API Gateway pattern. In BFF we have multiple gateway services that will server different type of devices (IOS, Andriod and Web) against having single API Gateway.

Advantage of this pattern is we can have decoupled gateway service fro different that handles request and response gracefully. Also BFF pattern helps in keeping track of requests and corresponding request in more simpler way.

Also with BFF pattern, if due to some issues one gateway device service goes down then other devices can be served without any interruption. Underlying Microservices(Backend) will be called in the same way with BFF as with one API Gateway.

Circuit Breaker:

Take a scenario of hotel booking website, to get latest information of all hotels (+availability) , their corresponding rooms and other meta info, we need to exchange/access data(information) from hotel system. For exchanging information we need to make external service calls(HTTP Calls) between applications(or systems). We can consider external services calls as third party calls as it is outside our system or internal calls if we are hitting some internal service.

If status of destination service is healthy and also its availability is 99.99% then we can assume that all the calls made by us will be succeeded every time. This can be seen as Happy/Best Case.

Due to some unexpected scenario, status of application becomes unhealthy(down) but still our system have no idea about its health. So we keep hitting their services and in response we are getting error response i.e may be 5xx with some timeout or something else. As we are bombarding the external system, what actually we are doing wrong on our side is unnecessary exhausting our system resources. So there should be some mechanism to handle this failure. Here Circuit Breaker comes as Saviour.

Circuit Breaker can be considered as request interceptor means all the external request will go through it only. In Circuit Breaker configuration we need to define some threshold for maximum number of calls we can make in case of failure.

Now when this threshold is reached then circuit will be open i.e no more calls will be done to server for configured time. When this circuit is open for each request we made, we get some fallback(default) response. Now when the configured time is completed Circuit Breaker will partially close the circuit means it enables few request with external system.

When with this partially closed circuit if calls start succeeding then Circuit Breaker will close the circuit and will start serving requests at full intensity. If with partially request failed then it will again open the circuit and start serving fallback response.

Note: Circuit Breaker will not open the circuit for each failure call as that may be due to some intermittent issue. For example we can define configuration like it should open if 9 out 10 fails.

To conclude above discussion we can Circuit Breaker have three states:

1. Circuit Closed ( Everything is working fine)
2. Circuit Open (Destination service is failing and we are sending fall back response)
3. Half/Partially Open (Making few calls to check whether service is up or not)

One of most famous Circuit Breaker used by most of bug organisations is Hystrix (available for almost all the languages).

Cache Stampede:

Consider a case when we are using some Caching Mechanism in our application. So expected flow is first time request will come to application server , fetch the data from source(external source or database) , cache it and return response. From next time when same request come , it will be served from cache(if TTL not expired).

Now consider a website like Cricinfo(which actually using cache stampede). Around 1 lakh identical requests(finding score for same match at particular instant) came (parallel or at particular instant. Till this time no cached data is present so there will be cache misses. All these request served by application server by making db calls.

Above scenario that is very common in real time applications mainly, and we are wasting unnecessary resources/time for serving the same identical calls This scenario/case is called Cache Stampede Effect.

To avoid this we need a separate service that handles this stampede effect by combining or by using some locking mechanism identical request for the given timeframe.

I will come back with coded solution for the above problem in next part.

Part 1: https://medium.com/@rishabh171192/system-designing-f69d77c0d0a1

In case of any query and suggestions please connect with me at https://www.linkedin.com/in/rishabh-jain-a5978583/ or drop a mail @rishabh171192@gmail.com