Processing PGP Encrypted Data in AWS Lambda using Python.

Business verticals dealing with highly critical and sensitive data, for compliance reasons, works with end to end encryption. The data is always encrypted before sharing with the business partners and the encrypted data received is processed in an end to end secured environment, ensuring no vulnerability or leakage throughout the pipeline.

PGP and GPG (pgp open source standard) is probably one of the most used Encryption solutions across industry. One prerequisite here is a basic knowledge of how PGP encryption works. To explain in brief, before requesting a file we generate key pairs (public and private). We share the public key with our business partners, who use the public key to encrypt the file before sharing. Once the file is received, we use the corresponding private key to decrypt it. Each key is associated with a user name and an email id.

There’s is another level of encryption on top of that where user provides a key (passphrase) to further encrypt the private key.

Now, let’s take a case where we have to process an encrypted file stored in AWS S3 in a serverless environment like Lambda. Below are the steps you can follow to process it :

• Extract the private key from your machine.
  In your machine (windows/mac/Linux), identify the private key:
  gpg --list-secret-keys user@some.com
Export the key:
  gpg --export-secret-keys YOUR_ID_HERE > private.key
Move the private.key file to your AWS S3 Bucket
  aws s3 mv private.key s3://your_s3_bucket/
• python-gnupg in Lambda:
  python-gnupg is the python library for working with gnupg, which allows you to encrypt or decrypt your data. AWS Lambda has no internal support for this library, so you’ll have to either create a deployment package (https://docs.aws.amazon.com/lambda/latest/dg/python-package.html) or a layer (https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) in lambda to bring it.
• Download files in your Lambda Function:
  Go to your Lambda function. /tmp is the only folder in AWS Lambda that is writable. Download your encrypted file and private.key file to /tmp folder.

s3Client=boto3.client(‘s3’)
s3Client.download_file(bucket,Key,’/tmp/myencryptedfile’)
s3Client.download_file(bucket,key,’/tmp/private.key’)

• Import the private Key:
  Set gnupg home directory and import the private key.

gpg = gnupg.GPG(gnupghome=’/tmp’) 
key_data = open(‘private.key’).read()
priv_key = gpg.import_keys(key_data)

• Decrypt the file:
  Passphrase is the user created secret key that encrypts the private key. We need to provide gpg with that secret key in order to decrypt the file.

with open(‘/tmp/’+myencryptedfile,‘rb’) as a_file:
  gpg.decrypt_file(a_file,passphrase=secret_key,output=someFileName)

• Process the decrypted file:
  Once the file is decrypted, you can process it as per your use case; load it to RDS, Redshift, DynamoDB or write to an SNS topic.

This can be achieved in other AWS Serverless service like GLUE. You can use Glue Pyshell. The only advantage it gives over Lambda is that it has no runtime limit of 15 mins.