Stackdriver Resiliency and Proxies

Introduction

Sometimes you can’t directly connect to GCP API, but need to use a proxy. This can be for a wide variety of reasons.

How do you know your application — in this case logging to Stackdriver — is resilient to your proxy? How can you test it?

This blog will walk you through, step-by-step, setting up a simple test using an nginx proxy, prove you are using the proxy, and demonstrate the Stackdriver Python client library to be resilient with proxies (and network) coming and going.

Requirements

• You’re using Linux
• You have nginx installed
• Nothing is running on port 443 (https) for your machine
• You have access to /etc/hosts and nginx configuration
• You have gcloud authenticated, a project, and access to Stackdriver

Setting up the Proxy

Edit nginx config

Add the below configuration /etc/nginx/nginx.conf. This will take all incoming connections to nginx (on the local machine) on port 443 and create a forwarding connection (raw TCP/IP) to googleapis.l.google.com port 443.

stream {
  server {
    listen 443;
    proxy_pass googleapis.l.google.com:443;
  }
}

Note that as it’s a raw forwarding connection it doesn’t interfere with the SSL end-to-end connection. This just forwards data, rather than decrypting and re-encrypting data.

Restart nginx

This assumes you’re using systemd (modern Linux).

service nginx restart

Configure DNS

Editing /etc/hosts allows you to re-configure DNS for just your local machine. This is useful for testing purposes only. Add the below mapping.

; temporary
127.0.0.1       logging.googleapis.com

This will cause all connections to logging.googleapis.com and logging.googleapis.com to be routed to 127.0.0.1.

The easiest (and oldest) way to test this — if you have it installed — is to use telnet:

telnet logging.googleapis.com 443

You should see “Trying 127.0.0.1” at the top and a connection should be made. This connections is re-directed to localhost, through the nginx proxy, and connects to googleapis.l.google.com port 443.

Why would this work?

This is a bit tricky. First you should use host to see the real DNS for logging.googleapis.com:

host logging.googleapis.com

As of writing, this is the result:

logging.googleapis.com is an alias for googleapis.l.google.com.
googleapis.l.google.com has address 74.125.140.95
googleapis.l.google.com has address 173.194.76.95
googleapis.l.google.com has address 64.233.167.95
googleapis.l.google.com has IPv6 address 2a00:1450:400c:c00::5f

So logging.googleapis.com is an alias for googleapis.l.google.com. So we can pass things through directly to googleapis.l.google.com.

The client thinks it’s connecting to logging.googleapis.com. It’s really going to localhost, and then transiting through the proxy to googleapis.l.google. The SSL encryption proceeds like normal, certificates are validated, and the proxy has no way to decrypt the data being passed through.

Testing it — Basic

TOKEN=$(gcloud auth print-access-token)
DATE=$(date)
PROJECT=$(gcloud config get-value project)
DATA="
{
  'entries': [
    {
      'logName': 'projects/${PROJECT}/logs/curl',
      'resource': {
         'type': 'global',
         'labels': { 'project_id': '${PROJECT}' } },
      'textPayload': 'API Write at ${DATE}'
    }
  ],
}
"
curl -v -X "POST" \
        -H "Content-Type: application/json" \
        --data "${DATA}" \
        -H "Authorization: Bearer ${TOKEN}" \
        -H "Content-Type: application/http" \
           "https://logging.googleapis.com/v2/entries:write"

This gets your current bearer token from gcloud, current default project, current date, creates a JSON message, and then uses the write method to write the log message.

Stackdriver Log Message

You can then go into the Stackdriver console and should see the message appear in the “Global” filter in your default project. It may take a few minutes to show up.

Note the receiveTimestamp versus the timestamp in the textPayload. This is the delay between when the message was received by Stackdriver (from the REST API) and when the log message was generated. Here it is the same time (within a second).

Testing it — Advanced

First we start with a small Python script — using the google-cloud-logging pip package — writing to Stackdriver. This requires you to be authenticated with gcloud or running on a GCE instance with scope to write to Stackdriver logging.

Running this script will produce log messages every second with the log message text containing time of the log message generation. This is different than what Stackdriver may receive the message. (Although normally the delay is short!)

I’m running this with Python 3.6.5 and 1.11.0.

Python Script:

import time
import logging
import datetime
# Imports the Google Cloud client library
import google.cloud.logging
# Instantiates a client
client = google.cloud.logging.Client()
client.setup_logging()
logger = logging.getLogger()
logger.setLevel(logging.INFO)
while True:
    time.sleep(1)
    n = str(datetime.datetime.now())
    logger.info('drip at {}'.format(n))

Once you start running this script, you can then go back to the Stackdriver logging and see the log messages appearing.

Log messages — normal working proxy:

Interrupt the Proxy

But now it gets interesting. Let’s stop and start the nginx proxy for 15 seconds and see what happens.

In a separate shell — let your python script keep running, keeping an eye on it — run the following:

sudo systemctl stop nginx

You can then see your program continuing like normal. (Note that earlier versions of google-cloud-logging may behave differently) After a minute or so re-enable the nginx proxy:

sudo systemctl start nginx

Now check the Stackdriver log messages.

Stackdriver Log Messages

You can see that the first several messages have a similar log received timestamp — the left column — as the log message itself. However, you can also see the gap between 13:01:21 and 13:02:34, where all of the following log messages have the same received timestamp. The log messages continue their normal time increment.

Conclusion

Now you have tested using nginx proxy while writing log messages with Stackdriver, and see how it has recovered from an interrupted connection.

Be sure to clean up your /etc/hosts! You don’t want to break your system.