Boto3 and Flask Demystified

Photo by Hans-Peter Gauster on Unsplash

I was recently tasked with building and deploying a microservice that required interacting with S3. Everything was well until the moment it was deployed and running in EC2 (Amazon Linux 2 AMI). At this moment I realized that there is some information missing about how to properly configure your EC2 instance to work with AWS.

First let’s go over the official Boto3 installation and configuration instructions.

Installation

Install the latest Boto3 via pip

$ pip install boto3

Configuration

Set up your AWS authentication credentials using the AWS CLI

$ aws configure

The result from running the previous command should leave you with a similar credentials file and config file

~/.aws/credentials

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

~/.aws/config

[default]
region=us-east-1

Please also note that alternatively you can create these files yourself if the AWS CLI is not installed in your machine.

Ok so far so good :)

Once Boto3 credentials are configured we can import it into our application

import boto3

# Let's use Amazon S3
s3 = boto3.resource('s3')

At this point if the application is running in EC2 the following error might be the reason you are reading this post.

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

Everything was running just fine locally in my Macbook’s local environment. What could have gone wrong?

AWS will tell you over and over the solution is to configure your credentials properly with their AWS CLI.

Well let me tell you this is not the solution at all and you shouldn’t waste any of your time trying to fix it this way.

My preferred deployment platform for Python projects is WSGI which stands for “Web Server Gateway Interface”. It is used to forward requests from a web server (such as Apache or NGINX) to a backend Python web application or framework.

And this is where the problem begins, I scoured the internet and found my answer in some random blog hidden in the deepest depths of a google search.

Here is the blog for reference

This Is The Title Of This Page

So basically the reason Boto3 can’t find my credentials is because WSGI is not ready at the time the application initializes.

We have a few options to fix that:

First option we could create a .env file in our current working directory and save our AWS credentials there.

.
├── .env
├── run.py
└── config.py
# .env
AWS_ACCESS_KEY_ID=YOURACCESSKEY
AWS_SECRET_ACCESS_KEY=YOURSECRECTKEY

To make this work python-dotenv has to be installed first

$ pip install -U python-dotenv

And then we can load the env file to our config.py file.

# config.py
from dotenv import load_dotenv
load_dotenv()

# OR, the same with increased verbosity:
load_dotenv(verbose=True)

# OR, explicitly providing path to '.env'
from pathlib import Path  # python3 only
env_path = Path('.') / '.env'
load_dotenv(dotenv_path=env_path)

Remember not to commit your .env file to version control to keep sensitive information safe, in this instance it would our aws credentials.

A second option is to add the credentials to our wsgi file like so

# myapp.wsgi
from run import app as application
import sys
import os
sys.path.insert(0, '/var/www/html/app_name')
os.environ['AWS_ACCESS_KEY_ID'] = 'YOUR_ACCESS_KEY'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'YOUR_SECRECT_KEY'

Project structure example:

.
├── run.py
├── myapp.wsgi
└── config.py

Also remember this file should be committed to version control to protect sensitive information.

Ok so Boto3 is now able to find our AWS credentials, but there is still one issue. What if your project strictly requires you keep this sensitive information out of the directory that is being served to the internet, specifically /var/www/html/my_app . We are up to the challenge, let’s see how we can fix that.

The best way I was able to figure this one out was to pass my AWS credentials as Apache environment variables, this way they are set by the server environment. To do that we have to add them to the VirtualHost configuration.

In Ubuntu you would find this file in /etc/apache2/sites-enabled/000-default.conf

<VirtualHost *:80>
     ServerAdmin webmaster@example.com
     ServerName example.com
     ServerAlias example.com

     SetEnv AWS_ACCESS_KEY_ID your_access_key
     SetEnv AWS_SECRET_ACCESS_KEY your_secret_key

     DocumentRoot /var/www/html/
     WSGIScriptAlias / /var/www/html/my_app.wsgi

     ErrorLog /path/to/logs/error.log
     CustomLog /path/to/logs/access.log combined
</VirtualHost>

If you restart apache $ sudo service apache2 restart at this moment you will get the same no credentials found error. This is because wsgi does not pass the environment variables by default so we have to set them manually in our application’s wsgi file.

# myapp.wsgi
import sys
import os
sys.path.insert(0, '/var/www/html/app_name')

def application(environ, start_response):
    for key in ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY']:
        os.environ[key] = environ.get(key, '')

    from run import app as _application
    return _application(environ, start_response)

And now Boto3 can find the AWS credentials again and they are as secure as ever. Also make sure to restart apache to apply the changes $ sudo service apache2 restart

I hope this helps, it was really hard to find an answer on how to do this, I feel like there should be a tips or notes section either in the Boto3 docs or the AWS CLI docs.

Feel free to hit me up if there are any typos, errors of a way to improve this article, questions, also if you would like to learn how to deploy your Flask application to AWS EC2 ubuntu head to my tutorial Deploy a Flask app on AWS EC2