Encrypting Logstash data

Note, the patch described below is now merged into the official logstash-filter-cipher plugin as of January 2016, version 2.0.3

UPDATE: Note the pending patch to fix various issues and add random IV support for encrypting logstash event messages is located here here: https://github.com/logstash-plugins/logstash-filter-cipher/pull/3

Logstash can consume a myriad of data from many different sources and likewise send this to all sorts of different outputs across your network. However the question that sometimes comes up is: “how sensitive is this data?”

The answer to that question will vary as wildly as your inputs, however if it is sensitive in nature, you need to secure it at rest and in transit. There are many ways to do this and they can vary on the combination of your inputs/outputs and what protocols are supported and whether or not they use TLS or some other form of crypto. Your concerns might also be mitigated if you are transferring this data over a secured/trusted network, VPN or destination specific ssh tunnel.

However if none of the above apply, or you simply don’t have control over the parts of the infrastructure in between your Logstash agent and your final destination, there is one cool little filter plugin available to you in the Logstash Contrib Plugins project called the cipher filter.

If the only thing you have control over is the ability to touch your config file on both ends you can use this without having to fiddle with other infrastructure to provide a modest level of security for your data. This is assuming you have a topology where Logstash agents, send data to some queueing system, and then they are consumed by another Logstash system downstream, which subsequently filters the data and sends it to its final resting place. Using the cipher filter comes into play where your “event” field(s) are encrypted at your source agent, flows through the infrastructure, and decrypted by your Logstash processor on the other end.

The cipher filter has been around for some time, but recently I had a use case for it similar to the above so I attempted using it and ran into some problems. In lieu of any answers, I ended up fixing the issues and adding some new features in this pull request. So if you are reading this, and that pull request is not yet approved, you might need to use my fork instead.

IMPORTANT: The security of your “key” is only as secure as the readability of your logstash config file that contains it! If this config file is world readable, your encryption is worthless. When using this method, be sure to analyze how and who could potentially access this configuration file.

Here is a simple Logstash conf example of using the cipher filter for encryption

input {
 
    file {
        # to test, just starting writing
        # line after line to the log
        # this will become the 'message'
        # in your logstash event
        path = "/path/to/test.log"
        sincedb_path = "./.sincedb"
        start_position = "end"
    }
}
 
filter {
 
    cipher {
        algorithm = "aes-256-cbc"
        cipher_padding = 1
 
        # Use a static "iv"
        #iv = "1234567890123456"
 
        # OR use a random IV per encryption
        iv_random_length = 16
 
        key = "12345678901234567890123456789012"
        key_size = 32
 
        mode = "encrypt"
        source = "message"
        target = "message_crypted"
        base64 = true
 
        # the maximum number of times the
        # internal cipher object instance
        # should be re-used before creating
        # a new one, default to 1
        #
        # On high volume systems bump this up
        max_cipher_reuse = 1
    }
 
    cipher {
        algorithm = "aes-256-cbc"
        cipher_padding = 1
 
        # Use a static "iv"
        #iv = "1234567890123456"
 
        # OR use a random IV per encryption
        iv_random_length = 16
 
        key = "12345678901234567890123456789012"
        key_size = 32
 
        mode = "decrypt"
        source = "message_crypted"
        target = "message_decrypted"
        base64 = true
 
        # the maximum number of times the
        # internal cipher object instance
        # should be re-used before creating
        # a new one, default to 1
        #
        # On high volume systems bump this up
        max_cipher_reuse = 1
    }
 
}
 
output {
 
    # Once output to the console you should see three fields in each event
    # The original "message"
    # The "message_crypted"
    # The "message_decrypted" verifying that the decryption worked.
    # In a real setup, you would only send along an encrypted version to an OUTPUT
    stdout {
        codec = json
    }
 
}

View all posts by bitsofinfo

Originally published at bitsofinfo.wordpress.com on June 25, 2014.