How to Ship Cloudflare logs to Sumo Logic with Lambda

With the advent of Cloudflare’s new Enterprise Log Share (ELS) REST API, it is now easy to download and inspect a zone’s access logs by a single RayID (Cloudflare’s fancy term for a request ID), a defined time period, or from the last RayID downloaded. In our case, we ship all of our other logs to Sumo Logic so everyone can analyze and parse them in any way imaginable!

Unfortunately, there seems to be no cut-and-dry way to integrate the two. Thankfully, AWS Lambda functions are now able to be run on a scheduled interval using CloudWatch Scheduled Events — sort of like you would a cron job on a server.

There are a couple of clear benefits of using a Lambda function instead of a job on a physical/virtual server you run yourself:

1. Low cost: Execution of a Lambda function costs next to nothing, and it may actually cost nothing if you remain within the free tier AWS hands out every month
2. Ease of development: In my case, I used the Serverless framework to allow me to quickly get up and running, invoke a function locally (which is key to developing and iterating quickly), and it makes deployments simple. I also recommend checking out apex, too.
3. Speed: Lambda functions are fast. This is important for things which require speed, such as parsing and ingesting logs for a high-traffic site like ours.
4. No servers to build/maintain: Write the code, test and deploy — that’s it! No Chef, no Docker, no nothing… just write your function, ship it and it is live.

How It Works

First, you’re going to want to create a new HTTP source to ingest the Cloudflare logs. Set the source category, host, etc. at SumoLogic to make the configuration of the Lambda function easier.

Next, create a new Serverless function:

$ mkdir -p ~/cloudflareToSumoLogic
$ cd ~/cloudflareToSumoLogic
$ serverless create \
    --name cloudflareToSumoLogic \
    --template aws-nodejs \
    --path ~/cloudflareToSumoLogic

Let’s walk through the serverless.yml configuration file:

functions:
  
  # This is the name of the Lambda function
  cloudflareToSumoLogic:
    # This is the name of the handler; this is the "main"
    # function called by Lambda
    handler: index.handler
    # This function was written to be run every one minute
    # Any adjustments will require code changes...
    events:
      - schedule: rate(1 minute)
    # Required environment variables:
    environment:
      # Grab this URL from the Sumo Logic source you just created
      SUMO_ENDPOINT: ...
      # Get these values from the Cloudflare admin UI
      CLOUDFLARE_ZONE_ID: ...
      CLOUDFLARE_AUTH_EMAIL: ...
      CLOUDFLARE_AUTH_KEY: ...

Finally, you’re going to need the function:

Now, try it out!

$ serverless invoke local -f cloudflareToSumoLogic
startTime: 2017-02-11T15:07:00.000Z
endTime: 2017-02-11T15:08:00.000Z
cloudflareOpts: { method: 'GET',
  hostname: 'api.cloudflare.com',
  path: '/client/v4/zones/.../logs/requests?start=1486825620&end=1486825680',
  headers:
   { 'X-Auth-Email': '...',
     'X-Auth-Key': '...' } }
res.statusCode:  200
res.headers:  { date: 'Sat, 11 Feb 2017 15:38:40 GMT',
  'content-type': 'application/json',
  'transfer-encoding': 'chunked',
  connection: 'close',
  'set-cookie': [ '__cfduid=...; expires=Sun, 11-Feb-18 15:38:39 GMT; path=/; domain=.cloudflare.com; HttpOnly' ],
  vary: 'Accept-Encoding',
  'x-cf-request-id': '00000000-0000-0000-0000-000000000000',
  'strict-transport-security': 'max-age=31536000',
  'served-in-seconds': '0.355',
  server: 'cloudflare-nginx',
  'cf-ray': '...-ORD' }
Log events: 659

Everything look good? It’s time to deploy:

$ serverless deploy
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading service .zip file to S3 (9.45 KB)...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.....................
Serverless: Stack update finished...
Serverless: Removing old service versions...
Service Information
service: cloudflareToSumoLogic
region: us-east-1
api keys:
  None
endpoints:
  None
functions:
  cloudflareToSumoLogic

Analyzing Ingested Logs in Sumo Logic

Now that your logs are being ingested every minute, you can run the following query and you should be able to start seeing results like the screenshot above:

_sourceCategory=...
| json auto
| toLong(num(timestamp)) as timestamp
| timestamp as _messageTime
| fields - timestamp
| fields - _raw

Replace _sourceCategory with the HTTP source category that you created earlier for your Cloudflare logs. This snippet parses the JSON lines automatically, translates the log timestamp timestamp to a time object, replaces the _messageTime with the log timestamp, then remove the raw JSON object and its ingest timestamp.

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMI family. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!