Compression of data over HTTP on Google Cloud

Background

Cloud users sometimes wonder if there is a more efficient way to compress text and JSON data over the web than gzip. First, let’s see how to do it with gzip.

If an App Engine HTTP client sends an Accept-Encoding header with value gzip, then it will compress the contents returned(documentation). Currently, gzip is the only option directly supported with App Engine. This is similar to NGINX out of the box. HTTP/2 replaces gzip with HPACK because a security vulnerability was found with compressed HTTP streaming using gzip (doc). However, that is only for HTTP headers. QUIC leverages a similar method to HTTP/2 for headers.

To verify that your server is returning gzip content, use a Curl command like

curl -H "Accept-Encoding: gzip" -I http://localhost:8080/test.json

assuming that your server is running on your on machine at localhost. You should see output with a line like

Content-Encoding: gzip

This Gist gives instructions for setting up an Nginx server to compress JSON files. HTML files should be compressed by default.

Brotli

Brotli (br) can give substantially more compression than gzip but can be less efficient to compress (simple comparison). So, it may be more appropriate for files that are read often than dynamically created application data. However, there is also control over the level of compression with Brotli, which may be used to give an optimal balance for your application. Google has open sourced Brotli (github project), which lists more detailed benchmarks. Brotli is supported by the Cronet network client libraries, suitable for use in mobile applications. In Cronet, Brotli is not enabled by default but can be enabled by calling enableBrotli(true).

Brotli is also supported by the major browsers. The Mozilla documentation gives the proper HTTP header to use

Accept-Encoding: br

Although, Brotli is not directly supported by GCP, an open source Nginx plugin for Brotli (here) can be used with a Docker image (here). This should be adequate for Flex with a custom container and for Kubernetes. Simple instructions for using and verify this are provided in this Gist. After configuration of the Nginx server you should be able to verify that it is returning Brotli compressed data with a command like

curl -H "Accept-Encoding: br" -I http://localhost:8080/test.json
...
Content-Encoding: br