Cloud Logging though a Python Log Handler

I’ve been doing Big Data on the Google Cloud Platform for a while now and I really saw the platform mature. And if you’re doing Big Data you probably love Python. It’s really a language to get things done.

One of those build in features is Logging. Out of the box Python has a few Log Handlers for logging to console, files and streams. But I was really surprised nobody wrote a decent log handler for Google Cloud Logging.

So I decided to write a build-in handler for my luigi-gcloud extension, giving the user full control for what they want to log though the logging configuration.

But because it’s so small and easy to use I want to share it so you can build it in your Python scripts that suppose to run on Compute. Include it in your project and you have the handler ready to use.

The only thing left is to configure it with a ini file. A caveat though is to make sure that the root logger doesn’t use our handler. This is because the client library also does logging and you will get a reclusive loop.

The important parts in the ini file is the [logger_app] and [handler_cloudLoggingHandler] section. This will route the loggers with name ‘my-app’ to a the respective handler. In the handler section you need to specify the project_id of our cloud project.

[loggers]
keys = root,app

[handlers]
keys = consoleHandler,cloudLoggingHandler

[formatters]
keys = simpleFormatter

[logger_root]
level = DEBUG
handlers = consoleHandler

[logger_app]
level = DEBUG
handlers = cloudLoggingHandler,consoleHandler
qualname = my-app
propagate = 0

[handler_consoleHandler]
class = StreamHandler
level = DEBUG
formatter = simpleFormatter
args = (sys.stdout,)

[handler_cloudLoggingHandler]
class = gcloud.log.CloudLoggingHandler
level = DEBUG
args = ('your-project-id',)

[formatter_simpleFormatter]
format = %(levelname)-8s %(name)-18s : %(message)s

Everything is configured, and now we can start logging directly to Cloud Logging. Make sure you look into the Custom Logs in the console.

I have to point out that for each log call an API call is done which can impact performance. But as a lot of Python scripts are written for automation task that are not performance critical this should not be a problem to them.

Happy Logging!