Heavy-HTTP

Breaking the barriers of payload limits with HTTP

(source: https://www.toastmasters.org/magazine/magazine-issues/2021/june/breaking-language-barriers)

The Rest and HTTP request-response pattern has almost conquered Client-Server communication. Even though the combination of Rest and HTTP provides an illusion of a universal methodology for communication there are many practical limitations in this approach. By introducing Heavy HTTP, we solve one of the key practical limitations of this communication pattern and extend its capabilities.

The size of the payload of requests/responses is one of the most significant practical limitations in this pattern. With the latest trends, modern web applications more often exchange large amounts of data back and forth with HTTP and Rest. But with a single HTTP request/response, the process is highly inefficient. Not just that most of the server implementations can’t handle large requests/responses. As an example, these are the payload size limits of the Request/Response handling interfaces of the world’s largest cloud service providers.

This payload size limitation literally ties our hands because,

1. You never know when the payload will go beyond the threshold
   In client applications, it’s hard to predict the client inputs. They may send a little amount of information or a very big chunk of data. The same would happen on the server end as well. With complex relationship models, even with pagination, the payload size can vary in a vast range. That means either some of the client inputs or server responses will be ignored (crashed!) because no one can predict the size of the data.
2. Implementation of alternative methods requires a lot of effort
   Providing an alternative communication would require additional boilerplate code and logic on both client and server ends. And the problem becomes much worse when it is required to measure the payload size depending on the payload type.

When you deal with large payload sizes the next inherent issue is connection time limitations. When the request is heavy it most likely takes a longer time to process and that could lead to connection timeouts. Wouldn’t it be cool if it’s possible to handle heavy requests asynchronously while handling other requests synchronously?

Well Heavy HTTP is there to save you from all the trouble. Heavy HTTP is a utility framework that automatically handles the payload limitations in client-server Rest HTTP communication with very minimal configurations. It utilizes signed URLs as an alternative communication pattern to overcome the payload size limitation and provides transporters to control the request data on the server side. Most importantly, it doesn’t interrupt the existing communication patterns. So only by performing very few modifications, any existing application can be retrofitted to work with Heavy HTTP.

Heavy HTTP consists of three main components

1. Heavy HTTP Client Connector
2. Heavy HTTP Server Connector
3. Heavy HTTP Transporter

Heavy HTTP Client Connector

Heavy HTTP Client Connector is an extension of the default HTTP Client that is used in the application. If the application is a browser, Heavy HTTP Client Connector extends the XMLHttpRequest and Fetch API. And if the application is written in Node JS then Heavy HTTP Client Connector extends the Node HTTP Client. For all the HTTP communication the extended Heavy HTTP Client is exposed via the same interface. Because of this pattern, regardless of the HTTP client wrapper library (Axios, Node HTTP etc) that is used in the application, Heavy HTTP can perform its magic.

Looking under the hood

At the initialization of the request Heavy HTTP Client Connector performs the following operations

1. Identify the type of the payload and estimate the size of the payload.
2. If the size of the payload is beyond the configured threshold shift to the Heavy HTTP Transporter to continue the communication. If not proceed with the existing communication pattern.
3. Provide the seamless experience of HTTP client to the HTTP Client wrapper library.

When receiving the response Heavy HTTP Client Connector performs the following operations

1. Check whether the response is a Heavy Response or not.
2. If it’s a Heavy Response then shift to the Heavy Http Transporter to fetch the data. Otherwise, proceed with the existing communication pattern.
3. Provide the seamless experience of HTTP client to the HTTP Client wrapper library.

Heavy HTTP Server Connector

This is a runtime-specific library that is responsible for the server end of communication with the Heavy HTTP Client that makes the request. Since the library is runtime-specific it needs to be attached to the runtime as an HTTP interceptor. Since the library is attached as an interceptor whatever the APIs provided by the runtime can be used without an issue with the library. Similar to the Heavy HTTP Client, Heavy HTTP Server also can perform its magic regardless of which API is used for the communication.

Looking under the hood

When receiving the request from HTTP Client, Heavy HTTP Server Connector performs the following operations

1. Identify whether the request is a Heavy Request or not (Based on request headers).
2. If it’s a Heavy Request shift to the Heavy Http Transporter to fetch the request data. Otherwise, proceed with the existing communication pattern.
3. Provide the seamless experience of the HTTP request to the runtime APIs.

When sending the response to HTTP Client, Heavy HTTP Server Connector performs the following operations

1. Identify the type of the payload and estimate the size of the payload.
2. If the size of the payload is beyond the configured threshold shift to the Heavy HTTP Transporter to continue the communication. If not proceed with the existing communication pattern.
3. Provide the seamless experience of HTTP response to the HTTP Client. (If the response is a heavy response, then the HTTP Client must be a Heavy HTTP Client to understand the protocol).

Heavy HTTP Transporter

Any temporary/permanent storage mechanism that provides signed URLs for upload and download purposes can be a transporter. The transporter is attached to the Heavy HTTP Server so that it is fully decoupled from the Heavy HTTP client. The Transporter gives you the control of handling the request and response as you like. If you want to process the request in a lazy manner (given that request is heavy) that also can be achieved as well.

Heavy HTTP Communication Protocol

Have an interest in the concept?

Well if you are interested we suggest you to go through our full GitHub documentation Heavy-HTTP.

Then if you are willing to help with coding, ideologies, testing or documentation, share your interest with us by dropping a message to heavyhttp@gmail.com.

Happy Coding! ✌️