How to log your application on Google Compute Engine

I have a custom daemon running on a Google Compute Engine instance, and I wanted to somehow get its log into a nicer interface. Some way to keep tabs on the log, without having to ssh into the instance all the time.

It took me a little bit of searching and piecing together to figure out how to do this, so I thought I’d share what was involved.

The first piece is Stackdriver Logging. It’s a service provided as part of Google Cloud Platform:

Stackdriver Logging

It provides a reasonably nice interface for your logged events. You can access it from your Google Cloud Console by going to Logging->Logs:

Great, so how do we get our application’s logs into here? First, you need to install the Logging Agent on your Compute Engine instance. It’s super easy, just following the install instructions here: https://cloud.google.com/logging/docs/agent/installation

FYI: fluentd is using about 300MB of RAM on my instance.

Once that’s installed, we need to add a custom fluentd configuration. With this config fluentd will watch our log file, and feed events to Stackdriver Logging for us. Here’s one of the configurations I ended up using:

<source>
type tail
format none
path /home/bitcoin/data/debug.log
pos_file /var/lib/google-fluentd/pos/bitcoind.pos
read_from_head true
tag bitcoind
</source>

This configuration captures the logs of bitcoind. It tells fluentd to read /home/bitcoin/data/debug.logjust like tail -f would. It should keep track of where it left off, so after reboot it’ll continue normally. The line read_from_head true tells fluentd, when first reading the file, to start from the beginning. This is useful if you already have stuff in the log and want fluentd to “catch up”. Because of the position tracking, this should only matter for the very first time fluentd reads the file. fluentd runs as root, so I didn’t need to do anything special with the permissions on debug.log.

Write this configuration file to /etc/google-fluentd/config.d/bitcoind.conf, or name the file however you wish. Then call sudo service google-fluentd reload. That should be it! fluentd will watch the log and dump entries into Stackdriver.

If you go back to Stackdriver Logging you should be able to pull up the logs for your instance and see all the logs for your app. Nice!

Note that, by default, it looks like fluentd will automatically capture your syslog after installation. So if you’ve set up your application to log to syslog already, you won’t have to do any of the configuration above. In my case, I had another daemon I set up using systemd, so I just set StandardOutput=journal in its .servicefile, which pipes its stdout to syslog. systemd prepends the service name to log entries, so it’s easy to filter in Stackdriver.

Happy logging!