How to reduce web API (SSL) data transfer costs on Amazon Web Services

Here at GameAnalytics, we receive, store and process game events from 1.2 billion monthly players in nearly 90,000 games. These events all pass through a system we call the data collection API, which forwards the events to other internal systems so that we eventually end up with statistics and graphs on a dashboard, displaying user activity, game revenue, and more.

The data collection API is fairly simple in principle: games send events to us as JSON objects through HTTP POST requests, and we send a short response and take the event from there. Clients either use one of our SDKs or invoke our REST API directly.

We get approximately five billion requests per day, each typically containing two or three events for a total of a few kilobytes. The response is a simple HTTP 200 “OK” response with a small bit of JSON confirming that the events were received. The general traffic pattern is a high number of relatively short-lived connections: clients send just over two HTTP requests per connection on average.

So what would you guess is the greatest cost associated with running this system on AWS, with a fleet of EC2 instances behind a load balancer?

We wouldn’t have guessed that the greatest part of the cost is for data transfer . Data transfer from the Internet is free, while data transfer to the Internet is charged between 5 and 9 cents per gigabyte .

So we set out to do something about this and see if we could save some money here. We were a bit surprised that we couldn’t find anything written about what to do in this scenario — surely our use case is not entirely uncommon? — so hopefully this will be useful to someone in a similar situation.

1. Reduce HTTP headers

Before these changes, a response from this system would look like this, for a total of 333 bytes:

HTTP/1.1 200 OK Connection: Keep-Alive Content-Length: 15 Content-Type: application/json accept-encoding: gzip Access-Control-Allow-Origin: * X-GA-Service: collect Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Headers: Authorization, X-Requested-With, Content-Type, Content-Encoding {"status":"ok"}

(Remember that the line breaks are CRLF, and thus count as two bytes each.)

Since we would send this five billion times per day, every byte we could shave off would save five gigabytes of outgoing data, for a saving of 25 cents per day per byte removed.

Much of this could simply be removed:

• The Access-Control-Allow-Methods and Access-Control-Allow-Headers response headers are CORS headers, but they’re only required in responses to preflight requests using the OPTIONS method, so they are superfluous in responses to POST requests.
• The Access-Control-Allow-Origin is still required, but only when the request is a CORS request, which we can determine by checking for the Origin request header. For any request not sent by a web browser, we can just omit it.
• The Accept-Encoding header is actually a request header; including it in the response has no meaning.
• Finally, the X-GA-Service header was once used for debugging, but we don’t use it anymore, so it can be dropped as well.

So for the vast majority of requests, the response would look like this:

HTTP/1.1 200 OK Connection: Keep-Alive Content-Length: 15 Content-Type: application/json {"status":"ok"}

Sending 109 bytes instead of 333 means saving $56 per day, or a bit over $1,500 per month.

So it stands to reason that by reducing data sent to a third, costs for data transfer should drop by 66%, right? Well, costs dropped, but only by 12%. That was a bit underwhelming.

2. Also reduce TLS handshakes

Obviously, before we can send those 109 bytes of HTTP response, we need to establish a TLS session, by exchanging a number of messages collectively known as a “TLS handshake”. We made a request to our service while capturing network traffic with Wireshark, and discovered that it sends out 5433 bytes during this process, the largest part of which is made up of the certificate chain, taking up 4920 bytes.

So reducing the HTTP response, while important, doesn’t have as much impact as reducing TLS handshake transfer size. But how would we do that?

Read the full article here.

Originally published at https://gameanalytics.com on December 13, 2019.