Automation with Ansible: Logentries

Using Logentries’ REST API to avoid manually managing logsets and logs

In some Docker Compose-based services I administer, I use Logentries to aggregate the log output from our containers. The token for the Logentries log is provided to the agent on the command line from the environment, something like this:

The LOGENTRIES_VERY_IMPORTANT_SERVICE environment variable is then populated through some Ansible we have. This approach works quite nicely, but leaves us with the burden of creating and naming new logs in Logentries when we deploy new instances of services, as well as transcribing the tokens for these new logs into our Ansible configuration. Lame. 🙅

In this article we’ll put together some Ansible tasks that will leverage the Logentries REST API to create a defined logset and list of logs if they don’t exist, and/or retrieve the tokens for those logs — ready to be plugged into something like the Docker Compose environment situation described above.

Generate Logentries API key

To get started we’ll need an API key for your Logentries account. Log in, navigate to your Account page, and click on the API Keys tab. Alternatively, navigate to this after logging in:

https://logentries.com/app/#/user-account/apikey

Click Generate Key for your Read/Write key, and record it somewhere, we’ll need it soon. Maybe test it out with a quick curl:

curl -H x-api-key:<API_KEY> https://rest.logentries.com/management/logsets

Define your API key as logentries_api_key somewhere (don’t forget to encrypt with Ansible Vault) in your Ansible variable configuration — personally I have a project-level group (project.ansibled) that fits perfectly.

Manage the Logentries REST API with Ansible

Now we’ll write some Ansible tasks that will manage our logs and log tokens. First a main task list that will retrieve our list of Logsets, create ours if necessary, and then determine whether our logs need to be created or not.

This main task list will then call out to two other small task lists for each log, either to create or look up each log:

Using the Logentries tokens

The idea here is that any host that runs these tasks just needs to provide the following:

• logentries_api_key (defined in our groups)
• logentries_logset_name (string)
• logentries_log_names (list of strings)

And after executing, the following will be defined:

• logentries_logset (Logset info returned by Logentries)
• logentries_log_ids (dictionary of name: id for the logs)
• logentries_log_tokens (dictionary of name: token for the logs — this is the important one!)

How you provide these tokens to your application depends entirely on how your stack is set up — but here is an example of a task that uses our new logentries_log_tokens dictionary and writes lines into a Docker Compose .env file:

I love automation. This series of articles, Automation with Ansible, is the documenting of some of the Ansible bits ‘n bobs that make my life easier when managing software infrastructure.

Check it all out on GitHub, or the other entries in the series:

• Automation with Ansible: Introduction
• Automation with Ansible: Building a VPC
• Automation with Ansible: AWS Elasticsearch
• Automation with Ansible: Aurora RDS Clusters
• Automation with Ansible: Logentries (this article)

tomwwright/ansibled