Serverless Big Data Visualisation

A (mostly) serverless log processing, searching and visualisation solution.

Overview

• Approx. 10 million log entries per day delivered from Akamai
• Fully searchable
• Dashboard with logins
• Archiving of processed logs with expiration policies

Here we will detail a basic setup for a mostly serverless solution for Akamai log processing using AWS services. Akamai’s content delivery network is one of the world’s largest distributed computing platforms, responsible for serving between 15% and 30% of all web traffic.

Mostly serverless?

The brief stipulated that log delivery would be via FTP, meaning that at least one part of the solution had to utilise the more classical approach of EC2 instances.

This first part of this article will focus on a manual setup, using the AWS console, with a follow-up article detailing the creation of the same solution but using Terraform to create the infrastructure in code.

For the details on Elastic Search and Kinesis configuration head over to the in-depth post.