How to log your application on Google Compute Engine

fpgaminer
fpgaminer
Jun 15, 2017 · 3 min read

I have a custom daemon running on a Google Compute Engine instance, and I wanted to somehow get its log into a nicer interface. Some way to keep tabs on the log, without having to ssh into the instance all the time.

It took me a little bit of searching and piecing together to figure out how to do this, so I thought I’d share what was involved.

The first piece is Stackdriver Logging. It’s a service provided as part of Google Cloud Platform:

Image for post
Image for post
Stackdriver Logging

It provides a reasonably nice interface for your logged events. You can access it from your Google Cloud Console by going to Logging->Logs:

Great, so how do we get our application’s logs into here? First, you need to install the Logging Agent on your Compute Engine instance. It’s super easy, just following the install instructions here: https://cloud.google.com/logging/docs/agent/installation

FYI: fluentd is using about 300MB of RAM on my instance.

Once that’s installed, we need to add a custom fluentd configuration. With this config fluentd will watch our log file, and feed events to Stackdriver Logging for us. Here’s one of the configurations I ended up using:

<source>
type tail
format none
path /home/bitcoin/data/debug.log
pos_file /var/lib/google-fluentd/pos/bitcoind.pos
read_from_head true
tag bitcoind
</source>

This configuration captures the logs of bitcoind. It tells fluentd to read /home/bitcoin/data/debug.logjust like tail -f would. It should keep track of where it left off, so after reboot it’ll continue normally. The line read_from_head true tells fluentd, when first reading the file, to start from the beginning. This is useful if you already have stuff in the log and want fluentd to “catch up”. Because of the position tracking, this should only matter for the very first time fluentd reads the file. fluentd runs as root, so I didn’t need to do anything special with the permissions on debug.log.

Write this configuration file to /etc/google-fluentd/config.d/bitcoind.conf, or name the file however you wish. Then call sudo service google-fluentd reload. That should be it! fluentd will watch the log and dump entries into Stackdriver.

If you go back to Stackdriver Logging you should be able to pull up the logs for your instance and see all the logs for your app. Nice!

Note that, by default, it looks like fluentd will automatically capture your syslog after installation. So if you’ve set up your application to log to syslog already, you won’t have to do any of the configuration above. In my case, I had another daemon I set up using systemd, so I just set StandardOutput=journal in its .servicefile, which pipes its stdout to syslog. systemd prepends the service name to log entries, so it’s easy to filter in Stackdriver.

Happy logging!

Google Cloud - Community

Google Cloud community articles and blogs

fpgaminer

Written by

fpgaminer

Engineer of all trades, I’ve worked in software, firmware, hardware, machine learning, computational imaging, web, servers, you name it.

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

fpgaminer

Written by

fpgaminer

Engineer of all trades, I’ve worked in software, firmware, hardware, machine learning, computational imaging, web, servers, you name it.

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store