Scalable Web Architectures Concepts & Design
For every web application, one of the fundamental factors that decides its success is its ability to seamlessly and efficiently accommodate growth, or “scale”, accordingly with clients' requests. A scalable web application is a website that is able to handle an increase in users and load, whether in terms of a gradual or abrupt surge, without disrupting end-users' activities.
In today’s world, customers have limited patience and empathy for a poorly-designed application. There is no such concept of waiting around for a web page to load, an image to upload, or a form to process. If your application isn’t designed properly, and able to handle requests in a timely manner, it will inevitably be left in the dust.
In this article, I will walk you through some of the architectural designs that web applications can be scaled. It’s not ideal to make the blog too lengthy by going into detail every corner of each architecture, I would like to, instead, serve as a mitigator, trying as much as I can to concisely and illustratively explain the core concepts, and direct you to some good resources for further reading & researching.
The content of this blog will be organized as follows:
- Monolithic Architecture
- Scale with Microservices Architecture
- Take away
- What’s next?
- References
What exactly does it mean to build and operate a scalable web site or application? At a primitive level it’s just connecting users with remote resources via the Internet — the part that makes it scalable is that the resources, or access to those resources, are distributed across multiple servers, which makes designing large-scale web systems error-prone and problematic. So why is there a need for robust and scalable web architectures? Before their inceptions, how were web applications structured?
I. Monolithic Architecture
The word ‘monolith’ means ‘one massive stone’, so when we talk about something monolithic we communicate an idea of a large unified block. In software architecture, a monolithic application is a self-contained, single-tiered traditional software application that contains the entire application code in a single codebase. Here’s an illustration of this architecture:
As you can see from Figure 2, all layers and components of the app are tightly connected. Applications followed this architecture are simple to develop, test, deploy, and manage since everything resides in one repository; there is no complexity of bundling different components into a single coherent workspace.
Monolithic applications can be successful if required resources are scarce and the number of requests needed to handle is modest, but increasingly people are feeling frustrated with them, especially when its tightly bundled attribute poses several major drawbacks when the applications need to be rescaled to extend computing capability:
- Continuous Integration/Continuous Deployment (CI/CD) is troublesome when a minor code change in a layer needs a re-deployment of the entire application.
- As components are grouped into one codebase, monolithic applications have a single point of failure. In case any of the layers has a bug, it has the potential to take down the entire application.
- Flexibility and scalability are also challenges in monolith apps as changing one layer often requires modifications and testing in all the others. As the code size increases, things might get a bit tricky to manage.
- Building complex applications with a monolithic architecture restricts technology leverage since implementing different components with different technologies may expose cross-component integration to compatibility issues.
These frustrations have led to the born of much more-scalable & robust architectures, which we will touch in depth in the following sections.
II. Scale with Microservices Architecture
In contrast to monolithic architecture, microservices architecture is an application structure that divides services into separate modules which are loosely coupled together, communicating with each other through light-weight mechanisms, often an HTTP resource API, WebSockets, or AMQP. Loose coupling means that each service implements a specific end-to-end domain or business capability within a certain context boundary, and each must be developed autonomously and be deployed independently. Each service should own its related domain data model and domain logic (sovereignty and decentralized data management) independently of the larger application.
Even though there isn’t a formal definition of microservices architectural style, there are some common characteristics that all applications adhering to this architecture should exhibit to make them scalable. These characteristics are smart endpoints & dumb pipes, decentralized governance, infrastructure automation, and failure isolation. Let’s discuss about each characteristic in the following sections.
1. Integration pattern: smart endpoints and dumb pipes
As a microservices application split up its structure into loosely coupled multiple components, natural questions to ask seem to be: In which protocols are pairwise components communication conducted? Given the protocols, in what patterns should the data be routed throughout the application?
a. Communication Protocols
When building communication structures between different processes, there’re many products and approaches that stress putting significant effort into the communication mechanism itself. A good example of this is the Enterprise Service Bus (ESB), where ESP products often include sophisticated facilities for message routing, choreography, transformation, and applying business rules. (ESB is also the fundamental difference between Microservices and Service-oriented Architecture (SOA). Further comparison between the two can be found at [2]).
Microservices, instead, favors an alternative approach: “smart endpoints and dumb pipes”. Microservices applications aim to be as decoupled and as cohesive as possible — they own their own domain logic an act more as filters in the classical Unix sense- receiving a request, applying logic as appropriate and producing a response, which is choreographed using mostly two commonly form of communications: Request-Response and Observer.
In the Request-Response communication pattern, microservices utilize the principles that construct the World Wide Web: one service invokes another service by making an explicit request, usually to store or retrieve data. The service then expects a response, either a resource or an acknowledgment. The most basic way to implement this pattern is using HTTP, ideally following REST principles. A standard HTTP based communication pipeline between two services typically looks like this:
Figure 3 shows a simple abstraction of an HTTP communication protocol between services. A load balancer can be used as a middleware tier to route requests to one of the instances of the backing service.
While REST design pattern is quite prevalent and well understood, it has the limitation of being synchronous and, thus, blocking. Asynchrony can be achieved using the second protocol paradigm, the observer model. In microservices, it would be more exact and specific to say that it should be a pub/sub pattern instead of the observer model (the difference between the two can be analyzed at [3]). A pub/sub pattern is an event-based, implicit invocation where producing service (publisher) publishes an event and one or more consumer services (subscribers) watching for that event respond to it by running logic asynchronously, outside the awareness of the publishers of the event. Subscribers can acknowledge the occurrence of the event by subscribing to a message queue/ broker service (hence the dumb pipe in the quote) using RabbitMQ, ZeroMQ, Kafka, or Redis Pub/Sub. A closer look at communication protocols can be found at [4].
Now, let’s talk about patterns in which data is routed/ collected throughout the application for seamlessly services integration.
b. Aggregator pattern
In the microservices world, Aggregator refers to a basic web page that invokes various services to get the required information or achieve the required functionality. This pattern is quite beneficial in scenarios that demand output by combining data from multiple services.
For instance, let’s say we have a ticking application, which is used to view details and book tickets for various activities like sports, movies, and other events. Assume that each activity is managed by a service within an architecture like in this figure:
The three services that manage the activities are exposed over REST endpoints, each implemented on AWS Lambda. Here, the aggregator service invokes the other three services for their data and links the data to its REST endpoint, which is exposed to the client. A more throughout analysis and code can be found at [5].
c. API Gateway
In a microservice architecture, the client usually needs to consume functionality from more than one services. If that consumption is performed directly, the clients need to handle multiple calls to service endpoints, which poses many problems that can severely affect service latency such as too many network round trips, cross-cutting concerns (authorization, SSL, …), tightly coupling between client applications and internal microservices design [6].
An API Gateway is a service that provides a single-entry point for certain groups of services. It sits between client apps and the services, acting as a reverse proxy, routing requests from clients to services. To address the above issues, it provides cross-cutting features such as SSL termination, authentication, load balancing, caching, …. It can also be used as a reverse proxy or gateway routing, and requests aggregation (kind of like batching multiple requests together).
Not only can we scale the system larger by API Gateway by serving more requests safely and timely, but we can also scale the API Gateway service itself by segregating it into multiple API Gateways based on business boundaries and client apps. For example, there can be three API Gateways, each deal with Mobile API for mobile clients, Browser API for JS clients in browsers, Public API for third-party developers, respectively (Introduction about Gateway segregation can be found at [6] about Azure API Gateway):
d. Chained or Chain of Responsibility (CoR)
Chained of Responsibility is a behavioral design pattern that produces a single combined output consolidated from multiple chained outputs from passing requests along a chain of stand-alone objects called handlers, or processing elements. Upon receiving a request, each handler decides either to process the request or to pass it to the next handler in the chain. The request travels along the train until all handlers have had a chance to process it. A handler also can decide not to process any further and stop passing the request down the chain.
By encapsulating the service inside a “pipeline” abstraction, CoR simplifies object interconnections. Instead of senders and receivers maintaining references to all candidate receivers, each sender keeps a single reference to the head of the chain, while each receiver keeps a single reference to its immediate successor in the chain.
Since all these services use synchronous HTTP request/response messaging, not until the request passes through all the services and respective responses are generated, does the client get the output. Therefore, it’s advisable not to make a long chain in this pattern.
e. Branch pattern
Branch design pattern is a pattern in which you can simultaneously process the requests and responses from two or more independent services. It extends the Aggregator and Chain Design pattern in which it uses aggregator to produce responses from multiple chains or single chain, based on demand.
In the above figure, service A, either a web page or a composite microservice, can invoke two different chains concurrently in which case this will resemble the Aggregator design pattern. Alternatively, Service A can invoke only one chain based upon the request received from the client.
f. Saga pattern
A saga is a sequence of local transactions. Each service in a saga performs its own transaction and publishes an event. The other services listen to that event and perform accordingly. If one transaction fails, the saga issues compensating transactions to roll back the impact made by preceding transactions.
There are two types of saga:
- Orchestration: There’s an orchestrator that dictates all the transactions and directs the participant services to execute local transactions based on events that are generated during/after previous transaction executions:
- Choreography: There’s no central coordinator in this type. Each participating service performs its transactions and publishes events. The other services act upon those events and perform transactions accordingly. They may not publish other events based on the situation:
2. Decentralized governance
One of the consequences of centralized governance is the tendency to standardize on single technology platforms. This problem is constricting, since not every problem is a nail and not every solution is a hammer, and it’s much more preferable to use the right tool for the right job.
Splitting the monolith’s components out into independently operated services, we have a choice when building each of them. Services can employ different languages (C++, Go, Node.js, …) and different data storages (SQL, NoSQL). The ability to use multiple languages during development means that each component can be built using the language and framework suited for that specific component. Additionally, teams can use new languages and frameworks as they are released to build out new components and services.
Decentralized Data Management: Database patterns
Adhering to the core principles of microservices, let’s quickly consider the below points when thinking about our database structure:
- Services must be loosely coupled. They can be developed, deployed, and scaled independently.
- Business transactions may enforce invariants that span multiple services
- Business transactions may query data that is owned by multiple services
- Databases are sometimes replicated and sharded in order to scale
- Different services have different data storage requirements
With the above principles in mind, let’s describe some of the database design patterns that microservices applications should follow.
a. Database per Service
While monolithic applications prefer a single logical database for persistent data, microservices decentralizes data storage decisions by letting each service manage its own database, either different instances of the same database technology (brownfield applications), or entirely different database systems (Polyglot Persistance — greenfield applications). Each database is designed privately for one service, and can only be accessed by other services through exposed APIs.
b. Shared Database per Service
Above, we have mentioned how having one database per service is ideal for microservices. However, if the application is a brown-field monolith and later being converted into microservices, denormalization is not that easy. Hence, an intermediatory solution between monolith and microservices is to let different services (Service 1 and Service 2 in Figure 13) share a single database. Each service freely accesses data owned by other services using local ACID transactions, which is more familiar with developers who have built the monolith application and still preserve the sense of distributed service model.
As expected, even though a single database is simpler to operate and more development-familiar, it may pose multiple problems such as schemas coupling, runtime coupling, or inadequate storage scaling for multiple services’ data.
c. Command Query Responsibility Segregation (CQRS)
In a microservice architecture, a service may query data collected from multiple other services. With normal databases, we use one system to perform modification to data and to query the data. With CQRS, part of the system deals with commands, which capture requests to modify state, while another part of the system deals with queries. For example, with event sourcing database implementation, developers might choose to handle and process commands as events, perhaps with storing the list of commands in a data store. Meanwhile, the query model could query the event store and create projections from stored events to assemble the state of domain objects.
This form of separation allows for different types of scaling. The command and query parts of our system could live in different services, or on different hardware, and could make use of radically different types of data store. For instance, you could even support different types of read format by having multiple implementations of the query model, perhaps supporting a graph-based representation of your data or a key/value-based form of your data. A closer look at this pattern can be found here.
3. Infrastructure automation
Faster release cycles are one of the major advantages of microservices architecture. But without a good CI/CD process, you won’t achieve the agility that microservices promise (When we talk about CI/CD, we are really talking about several related concepts: Continuous Integration, Continuous Delivery, and Continuous Deployment). A robust CI/CD process should be designed in ways that allow teams to build and deploy their own services independently, without affecting or disrupting other teams. Also, the CI/CD process should deploy the service new version through dev/test/QA/staging environments first for validation, before taking into effect at production.
Back in the days of summer 2019, one of my tasks at Got It was to implement a CI/CD pipeline management tool that accelerates infrastructure updates and product iterations cycles with AWS CodePipeline. The task’s objective was simple: build the backend for the internal portal that allows engineers to configure pipelines’ triggers with Boto3, deploys the pipelines, and returns any error(s) if have, to increase team velocity and work throughput in the staging environment. The tool plays an important part in Got It’s product ecosystem, helping it achieve the agility that the microservice architecture promises.
In the past, pipelines for monolithic applications tended to share the same characteristics of the application they were building. Since two different projects may have different pipelines, a company might have to deal with 1–5 pipelines in the case of monolith applications (assuming 1–5 projects). For microservices architecture, the number quickly jumps to 25 if each monolith is divided into 5 microservices. Based on this calculation, if the number of pipelines continues to rise linearly, it’s no surprise that a big organization with 50+ projects, each of which partitioned into 10 microservices, to maintain 500+ pipelines at a time. This explains why CI/CD pipeline design should be handle with great consideration for maintenance and addition of service to old pipelines as well as the creation of new ones.
CodeFresh has proposed a simple but powerful design to address the above problem. The basic idea here is that all services of a single application have almost the same life-cycle. They are compiled, packaged, and deployed in a similar manner. Therefore, instead of having multiple pipelines for each service, we have a single pipeline shared by all multiple services.
Now, pipeline construction very simple as it only consists of one shared pipelines for multiple git triggers, each trigger comes from git source of a particular microservices. When a new microservice is added in an application, the pipeline is already there and only a new trigger for that microservice is added. Therefore, as the number of microservices is growing, the only thing that is growing is the list of triggers. All pipelines are exactly the same.
4. Failure isolation
One of the benefits of microservices architecture is the failure isolation of components or features. In other words, if one service crashes, the other components can continue to operate until that service recovers. Taken an example of an E-commerce application. Ideally, whenever customers are trying to purchases items or viewing a certain item, our application attempts to create more sales by giving out some item recommendations on what customers should buy next, based on their chosen item(s). In this case, failure isolation means if somehow the recommendation service goes down, customers are still able to make their purchases with checkout service. Customers might not be presented with recommendations and some potential sales could be lost, but not all of them. The authentication service going down can be more devastating, but instead of a complete service disruption, users could still fill their carts, and when they are ready for check-out, can be presented with a screen warning them that if they are not able to complete their purchase at that moment, they should try again later. Also, as we can quickly roll back when there’s a failure, the disruption is short and the business impact can be minimized.
Failure can be isolated, and henceforth is mitigated. In order to do that, we need efficient strategies/ patterns to follow in the face of services partial failures.
a. Circuit Breaker Pattern
Here’s an interesting example I found in Sam Newman’s Building Microservices that related to this pattern:
In your own home, circuit breakers exist to protect your electrical devices from spikes in the power. If a spike occurs, the circuit breaker gets blown, protecting your expensive home appliances. You can also manually disable a circuit breaker to cut the power to part of your home, allowing you to work safely on the electrics. [8]
Indeed, in the microservices architecture, we will have one or more services calling other services for data/any other transactions. There’s a chance that the downstream service may be down due to heavy load or other dependent resources, or it may respond very slowly, before eventually returning an error. Things may be worse if the request is retried several times, costing lots of time before we finally get an error.
So, to avoid such problems, we implement a circuit breaker. One typical implementation is using the proxy as a circuit barrier. After a certain number of requests to the downstream resource have failed, the circuit breaker is blown, tripping from issuing the requests to that service for a particular period of time:
When the circuit breaker is down, there are some options: one is to queue up the requests and retry them later on, which can be appropriate in case the request is a part of an asynchronous job. If the request is a part of a synchronous call chain, it’s probably better to fail fast and either to propagate an error up to the chain or to choose a subtle degrading of functionality. Some other strategies can be found at [7].
III. Take away
Six principles of scalable web architecture design include availability, reliability, consistency, scalability, manageability, and cost.
In software architecture, a monolithic application is a self-contained, single-tiered traditional software application that contains the entire application code in a single codebase. While monolithic applications can be successful for small application size, it raises scalability problems such as redeployment of entire CI/ CD pipeline given code changes, single point of failure, tangled modifications & testing, and restricted technologies leverage.
Microservices architecture provides long-term agility. They enable better maintainability in complex, large, and highly-scalable systems by letting you create applications based on many independently deployable services that each have granular and autonomous lifecycles. As services are run independently of one another, we also have autonomy of development, deployment, and scale, for each service as each can scale out independently. That way, only services that need more processing power or network bandwidth will be scaled out on demand, saving the cost of scaling out other areas of the application that don’t need to be expanded.
IV. What’s next?
For the next article, let’s take a look at another two architectures to scale your web application: Service-oriented Architecture (SOA), and Serverless Architecture (FaaS) with AWS Lambda.
Stay tuned for the next article!
V. References
[1] KeyCDN. Comparing URI vs URL. 4 Oct. 2018, https://www.keycdn.com/support/comparing-uri-vs-url
[2] Edureka!. Microservices vs SOA. Youtube, 12 Mar. 2018, https://www.youtube.com/watch?v=EpyPFnjue38
[3] Ahmed Shamim Hassan. Observer vs Pub-Sub pattern. Hackernoon, 28 Oct. 2017, https://hackernoon.com/observer-vs-pub-sub-pattern-50d3b27f838c
[4] Nathan Peck, Microservice Principles: Smart Endpoints and Dumb Pipes. Medium, 1 Sep. 2017, https://medium.com/@nathankpeck/microservice-principles-smart-endpoints-and-dumb-pipes-5691d410700f
[5] Satrajit Basu, Microservices Aggregator Design Pattern Using AWS Lambda. Dzone, 7 Sep. 2018, https://dzone.com/articles/microservices-aggregator-design-pattern-using-aws
[6] Microsoft, The API gateway pattern versus the Direct client-to-microservice communication. Microsoft Docs, 7 Jan. 2019, https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect-microservice-container-applications/direct-client-to-microservice-communication-versus-the-api-gateway-pattern
[7] Microsoft, Strategies to handle partial failure. Microsoft Docs, 16 Oct. 2018, https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/partial-failure-strategies
[8] Newman, Sam. Building Microservice. O’Reilly, 2015.
