Uploading images to AWS from remote devices securely via Rest API using AWS Serverless stack
Use Case: You may want the images from remote or edge devices to be stored in S3 for audit purposes, for backup or using machine learning services provided by AWS once the files are uploaded to S3
Serverless architecture allows ways to build applications without having to worry about operational complexity — provisioning, maintaining and scaling servers and that too at a very nominal cost because you only pay for using the services not the underlying infrastructure. For instance, the cost for API gateway is 3.5$ per million API requests, Data Out charges — 0.09$/GB for the first 10TB and it reduces thereafter.
What I intend to demonstrate in this post?
I will first build an API with public access that will allow images to be uploaded to S3 bucket and later add security feature that will allow access only to specific users. The focus would on using Python to configure Lambda and securing API using AWS Signature version4 and API gateway resource policies.
Building a publicly accessible API for uploading images to S3
I will not go through the steps of configuring API gateway, Lambda and S3 through AWS console as there are plenty of blogs available, the closest one being: https://medium.com/think-serverless/image-upload-and-retrieval-from-s3-using-aws-api-gateway-and-lambda-b4c2961e8d1
- Please configure API gateway, Lambda and S3 as mentioned above. Once the configuration is done, you will see the below diagram when you click on the lambda function that you have created above
- After configuring API Gateway as mentioned above, my API gateway endpoint is https://8b52km67bi.execute-api.us-east-1.amazonaws.com/prod/upload-image-to-s3 with region as ‘us-east-1’, stage as ‘prod’, resource as ‘upload-image-to-s3’ and method as ‘POST’. I will be using this endpoint throughout the post
- Configuring IAM role for Lambda — Under “Execution Role”, I am using ‘lambda_s3_full_access’ role that contains following policies: ‘AmazonS3FullAccess’ and ‘AWSLambdaBasicExecutionRole’ one for getting write access to S3 bucket and other for logging into CloudWatch logs
- Lambda Code — Instead of using the Node.js code as mentioned in the above blog post, you can use the Python code from the GIT for the lambda function
- Once all the above services are configured, you can use the code from GIT to invoke the API. Check if the image has been uploaded to the bucket. You can also choose to use Postman
- After the image has been successfully uploaded to the S3 bucket, the next step would be to secure your API because by default it is public
Securing your API using AWS Signature version 4 and API Gateway Resource policy
- Click on ‘your resource’ in API Gateway, Go to ‘Resources’ section on the left and click on the method — ‘POST’. You will then see a diagram with a workflow. Click on the ‘Method Request’ box. Under ‘Settings’, set Authorization field to ‘AWS_IAM’
- I. Create an IAM user — raspberry-pi and store the access key id and secret access key. Only this user will be able to invoke the API
- Click on ‘Resource Policy’ within your resource to add the policy. Click on ‘Save’. You can also use https://awspolicygen.s3.amazonaws.com/policygen.html to generate the policy. The policy that I’ve generated for the above user can be found in GIT. Note: ‘Principal’ would contain the list of users (ARN) to which the access is provided and ‘Resource’ is the API gateway endpoint (ARN) for which the access would be provided
- Once the new policy is added, you need to rebuild the API. Resources -> Action -> Deploy API. If it’s not rebuild the IAM authentication will not be enforced. ** I lost several hours before I figured out I needed to redeploy the API. Please note that the endpoint does not change provided you use the same ‘Deployment stage’ as earlier
- Run the earlier code above to invoke the API. You will get a 403 error — “Missing authentication token”. This means the API is not publicly accessible
- Now that the relevant policy has been added to API gateway, we will use API Signature version 4 to sign the request for the user ‘raspberry-pi’. The access key id and secret access key for this user needs to be provided for signing the request. Incase you want to read more about the process for signing the request the details can be found here
- The python code implementing the above process can be found in Git, use the code to sign AWS request and invoke the API. On successful invocation, a response code of 200, with a message ‘Lambda executed successfully’ will be displayed
- The image would have successfully been uploaded to the S3 bucket
Hope you enjoyed building the #API using #Serverless stack in #AWS for uploading an image to #S3 and securing it using #AWSSigaturev4 and #APIGateway #resourcepolicies. Any feedback or comments would be appreciated.