Reading AWS SES encrypted emails with boto3
AWS Simple Email Service (SES) has this very neat feature that allows you to create a pipeline to process incoming emails.
Each step of the pipeline is called an Action, which can be to store the email in S3, to send an SNS notification, run a Lambda. A very simple example might be:
When you store incoming emails in S3, you also have the option to encrypt them with an SES client-side encryption using a KMS Key.
On the bright side, resulting S3 keys will benefit from both the server-side encryption from S3, and the client-side from SES, allowing for various RBAC models on the data.
On the dark side, the documentation about decryption is… cryptic to say the least. Even Google hasn't figured out how to decrypt the data in Python, @AWS still hasn't included client-side support in the aws-encryption-sdk, and people have to combine Java and Python to get this done, the world feels empty.
There is just this one genius who actually documented code that sorts of works here about "generic" client-side encryption on S3. Thanks a million @tedder42 for publishing this.
Unfortunately, it does not work on emails:
$ python get.py my-bucket incoming-emails/n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1 output.txt
Traceback (most recent call last):
File "get.py", line 63, in <module>
decrypt_file(decrypted_envelope_key['Plaintext'], dest_file, envelope_iv, int(original_size), "decrypted-" + dest_file)
File "get.py", line 29, in decrypt_file
decryptor = AES.new(key, AES.MODE_CBC, iv)
File "/usr/lib/python2.7/dist-packages/Crypto/Cipher/AES.py", line 94, in new
return AESCipher(key, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/Crypto/Cipher/AES.py", line 59, in __init__
blockalgo.BlockAlgo.__init__(self, _AES, key, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py", line 141, in __init__
self._cipher = factory.new(key, *args, **kwargs)
ValueError: IV must be 16 bytes longLooking at the code, it appears that the AES encryption used is MODE_CBC:
...
def decrypt_file(key, in_filename, iv, original_size, out_filename, chunksize=16*1024):
with open(in_filename, 'rb') as infile:
decryptor = AES.new(key, AES.MODE_CBC, iv)
...And if you look at the metadata of emails encrypted by SES, you can see that the algorithm is MODE_GCM and not MODE_CBC
$ aws s3api head-object --bucket my-bucket --key incoming-emails/n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1
{
"AcceptRanges": "bytes",
"ContentType": "application/octet-stream",
"LastModified": "Thu, 01 Nov 2018 14:18:34 GMT",
"ContentLength": 38519,
"ETag": "\"a7e63f50de8711e8aaf64b406710a414\"",
"ServerSideEncryption": "AES256",
"Metadata": {
"x-amz-unencrypted-content-length": "38503",
"x-amz-iv": "bB2T1y32dE87G1e8",
"x-amz-cek-alg": "AES/GCM/NoPadding",
"x-amz-wrap-alg": "kms",
"x-amz-matdesc": "{\"aws:ses:message-id\":\"n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1\",\"aws:ses:rule-name\":\"MoveToS3\",\"aws:ses:source-account\":\"012345678912\",\"kms_cmk_id\":\"arn:aws:kms:eu-west-1:012345678912:alias/aws/ses\"}",
"x-amz-key-v2": "AQIDAHghn...oraTng==",
"x-amz-tag-len": "128"
}
}Here is the trick: MODE_GCM does NOT exist in the default cryptography Python library. So if you use the provided code and just replace the mode, it will still fail with
Traceback (most recent call last):
File "get.py", line 63, in <module>
decrypt_file(decrypted_envelope_key['Plaintext'], dest_file, envelope_iv, int(original_size), "decrypted-" + dest_file)
File "get.py", line 29, in decrypt_file
decryptor = AES.new(key, AES.MODE_GCM, iv)
AttributeError: 'module' object has no attribute 'MODE_GCM'However, this mode DOES exist in PyCryptodome.
Let's install the standalone library:
$ pip install pycryptodomexand slightly update the get.py script to get the AES cipher from it instead of the default Crypto, then use the expected mode:
#!/usr/bin/env python3# Copyright 2015, MIT license, github.com/tedder.
# You know what the MIT license is, follow it.import base64
import json
from Cryptodome.Cipher import AES # pycryptodomex
import boto3
import sysif len(sys.argv) != 4:
print("usage: get.py bucket s3_key destination_filename")
sys.exit(-1)bucket_name = sys.argv[1]
key_name = sys.argv[2]
dest_file = sys.argv[3]def decrypt_file(key, in_filename, iv, original_size, out_filename, chunksize=16*1024):
with open(in_filename, 'rb') as infile:
decryptor = AES.new(key, AES.MODE_GCM, iv)with open(out_filename, 'wb') as outfile:
while True:
chunk = infile.read(chunksize)
if len(chunk) == 0:
break
outfile.write(decryptor.decrypt(chunk))
outfile.truncate(original_size)s3 = boto3.client('s3')
location_info = s3.get_bucket_location(Bucket=bucket_name)
bucket_region = location_info['LocationConstraint']
object_info = s3.head_object(Bucket=bucket_name, Key=key_name)metadata = object_info['Metadata']
material_json = object_info['Metadata']['x-amz-matdesc']envelope_key = base64.b64decode(metadata['x-amz-key-v2'])
envelope_iv = base64.b64decode(metadata['x-amz-iv'])
encrypt_ctx = json.loads(metadata['x-amz-matdesc'])
original_size = metadata['x-amz-unencrypted-content-length']kms = boto3.client('kms')
decrypted_envelope_key = kms.decrypt(CiphertextBlob=envelope_key,EncryptionContext=encrypt_ctx)s3.download_file(bucket_name, key_name, dest_file)
decrypt_file(decrypted_envelope_key['Plaintext'], dest_file, envelope_iv, int(original_size), "decrypted-" + dest_file)print("Flawless Victory!")
Suddenly it works much better:
$ python get.py <bucket> prefix/n14qsvqt4n767ap1kl5dlhbbf6cc50na5lif9ig1 output.txt
Flawless Victory!and the content of the email:
$ cat decrypted-output.txt
Return-Path: <origin@example.com>
Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [1.2.3.4])
by inbound-smtp.eu-west-1.amazonaws.com with SMTP id ...
for target@acme.com;
Fri, 02 Nov 2018 10:25:11 +0000 (UTC)
X-SES-Spam-Verdict: PASS
X-SES-Virus-Verdict: PASS
Received-SPF: none (spfCheck: 1.2.3.4 is neither permitted nor denied by domain of example.com) client-ip=1.2.3.4; envelope-from=origin@example.com; helo=mail-ed1-f54.google.com;
Authentication-Results: amazonses.com;
spf=none (spfCheck: 1.2.3.4 is neither permitted nor denied by domain of example.com) client-ip=1.2.3.4; envelope-from=origin@example.com; helo=mail-ed1-f54.google.com;
dkim=pass @example-com.20150623.gappssmtp.com">header.i=@example-com.20150623.gappssmtp.com;
dmarc=none header.from=example.com;
X-SES-RECEIPT:... ; c=relaxed/simple; s=...; d=amazonses.com; t=1541154311; v=1; bh=...; h=From:To:Cc:Bcc:Subject:Date:Message-ID:MIME-Version:Content-Type:X-SES-RECEIPT;
Received: by mail-ed1-f54.google.com with SMTP id u12-v6so1445426eds.4
for <target@acme.com>; Fri, 02 Nov 2018 03:25:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=example-com.20150623.gappssmtp.com; s=20150623;
h=mime-version:from:date:message-id:subject:to;
bh=...;
b=...
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
bh=...;
b=...
X-Gm-Message-State: ...
X-Google-Smtp-Source: ...
X-Received: by 2002:a17:906:6c9a:: with SMTP id s26-v6mr6344779ejr.239.1541154310145;
Fri, 02 Nov 2018 03:25:10 -0700 (PDT)
MIME-Version: 1.0
From: Samuel Cozannet <origin@example.com>
Date: Fri, 2 Nov 2018 11:24:59 +0100
Message-ID: <...@mail.gmail.com>
Subject: Hello World
To: target@acme.com
Content-Type: multipart/alternative; boundary="0000000000003362570579abf3c8"--0000000000003362570579abf3c8
Content-Type: text/plain; charset="UTF-8"I used to be an encrypted email--0000000000003362570579abf3c8
Content-Type: text/html; charset="UTF-8"<div dir="ltr">I used to be an encrypted email</div>--0000000000003362570579abf3c8--
There you go, you just successfully decrypted an email sent to an SES gateway, encrypted client-side with a KMS Key and stored to S3.
Again thanks Ted for sharing your code, it did put me on the right track! I hope this will now help others in the same situation as I was. Enjoy!