Envoy, Nginx, Apache HTTP Structured Logs with Google Cloud Logging

salmaan rashid
Jan 2, 2019 · 7 min read

Google Cloud Logging provides several plugins that allows you to easily emit structured logs for common applications.

For example, if you install the Stackdriver Logging agent, you can get logs using the following fluentd plugins

This sample demonstrates two of these plugins (apache and nginx) and how to configure them to emit not just structured JSON logs but as a specific HttpRequest protocol buffer. This article also describes how to configure Envoy proxy for similar httpRequest logging.

The script below sets up the full sample stack with Google Cloud L7 HTTPS loadbalancer and a managed instance group running the webserver. It does not demo Envoy with the L7 but you can set that up pretty easily with the template below.

In addition to the HttpRequest protocolbuffer, this example and configuration also includes the following in the emitted LogEntry

  • trace: This is the X-Cloud-Trace-Context header sent via the L7 Loadbalancer. see payload configurations
  • spanId: Inbound requests from Google L7 LB includes trace and span to cover.
  • httpRequest.latency: Latnecy value provided by nginx/apache and set as the latency` field
  • X-Forwarded-For Header is logged when available. That header is emitted by the L7 LB to indicate the actual originating IP address for the request. I've intentionally left the clientIp filed for the HttpRequest as the one provided though as a followup TODO: you are welcome to replace that value with the one extracted from this header.

Note, Stackdriver doens’t by default support all fluentd plugins but just the ones listed above. You are free to customer and define your own plugin described in the documentation in the following article auditd agent config for Stackdriver Logging

For Reference:

Usecases

First, i’ll just show what this will look like in nginx-access logs:

Filter by latency

Now that the logs are structured, we can apply a direct filter on those fields. For example, the following shows which requests took more than 2s to respond:

Trace Context logging

Each request in this specific example traverses Google L7 loadbalancer which also get logged. What that means is you can now trace a request in the LB logs down to the specific instnace in the managed group that handled the call. For example, if you start with a traceID from the nginx logs

  • nginx LogEntry: trace: "projects/mineral-minutia-820/traces/ba90fb6215439a958799f631ea63b24f"

You can find its corresponding entry in the LB LogEntry

  • LB LogEntry: trace: "projects/mineral-minutia-820/traces/ba90fb6215439a958799f631ea63b24f"

or run a query like this to cover both.

Combined LogViewer

Now that the frontend server is logging the inital httpRequest as well as the trace/spanId, a backend application that emits the traceId in a log line will result in log grouping. What that means is the inital request and application logs are grouped in a parent/child format as described in the following articles:

And just for reference (and becasue i authored it, :), here is the same impelmented within a webframework (Flask), directly

This article does not demonstrate this capability but if its useful, i can be convinced to provide an example.

Filter by SourceIP

If requests traverse the Google Loadbalancer, it injects the standard X-Forwarded-For header which includs the actual source IP address as well as the proxies this request traversed. In the case for Google, it will include the incident SSL Proxy.

For example

The origin ip is actually 73.162.112.208. Currently, the parsers just uses the provided value as the clientIP which is not the derived origin value (its still the loadbalancer). If you want to parse the actual origin IP, you will need to parse out the value from the header and add it to for the remoteIP.

For apache, its something like this:

parser_apache.rb:

apache.conf

You can find the full source here:

NGINX

If you prefer to use nginx, you can create the full LB->nginx managed instance group by running the following commands in sequnce:

Create

If can use the certs provided in this repo but if you prefer to setup your own…

then finish off the step:

You should end up with an L7 LB pointing to an instance group with four servers

NOTE: it may take upto 10 minutes for the L7 LB to provide the initial provisioning so feel free to get a coffee now

Test

Once the LB is seutp you can send traffic down

The responses above indicate how long the specific backend took. As you see, since we applied a variable delay into each instance as part of the startup script, each VM will take a minimum amount of time to respond

Delete

To delete the cluster, run the following in sequence:

Config

The following details the configurations used. Eventually, this should also get rolled into the standard google-fluentd library so wont' be a need to add the custom code below.

nginx

google-fluentd

Apache2

If you use Apache2, you can create the full stack by running the commands below in sequence.

Create

https://gist.github.com/salrashid123/5f1a2ddc74599611698778836dfb25d2

You can use the certs provided in this repo but if you prefer to setup your own…

then

Delete

Test

As with the example with nginx, you can send requests directly via curl in loop and gauge response times.

Config

The following details the configurations used for apache. Eventually, this should also get rolled into the standard google-fluentd library so wont' be a need to add the custom code below.

Apache2

/etc/apache2/apache2.conf

/etc/apache2/sites-enabled/000-default.conf

google-fluentd

/opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.5/lib/fluent/plugin/parser_apache2.rb

/etc/google-fluentd/config.d/apache.conf

Envoy

Envoy Access logs is fairly customizable and can write to any number of targets. For example, you can configure envoy to emit logs to remotely (see envoy_control#accesslog) or in this article, locally to a log file where Cloud Logging can do the rest of the legwork.

This article includes two separate configurations for envoy and google-fluentd: Default envoy settings and one optimized for GCP that includes the cloud-trace-context header.

Update: 1/6/19: Authored the fluentd plugin:

default

The steps to emit default logs is fairly easy: just set the path where the logs will write to

Custom header (cloud-trace-context)

You can customize envoy’s logs easily by adding in fields like custom headers into the log file. The following shows a snippet on now to setup the logs that extend the default configuration and just adds %REQ(X-Cloud-Trace-Context)%

Testing

To test this, you can incorporate the envoy configurations provided into a similar VM as described above. Remember to copy

If you are on a VM with google-fluentd and the configurations settings above, restart fluentd and run envoy:

Either send in a request via the LB or directly to test:

In cloud logging, you will see the standard httpRequest payload as well as envoy specific headers:

From there, you can further customize the headers you want to caputure by modifying the fluentd and envoy configurations.

Thats all folks

Google Cloud - Community

Google Cloud community articles and blogs

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

salmaan rashid

Written by

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store