Tutorial on Using Gitlab CI/CD Pipelines to Deploy Your Python Flask Restful API With Postgres on Heroku

Ruan Bekker
Jan 5, 2019 · 6 min read
GitLab Logo from about.gitlab.com

Today we will build a Restful API using Python Flask, SQLAlchemy using Postgres as our Database, testing using Python Unittest, a CI/CD Pipeline on Gitlab, and Deploying to Heroku.

From our previous post, we demonstrated setting up a Custom Gitlab Runner on Your Own Server for Gitlab CI.

Heroku

If you don’t have an account already, Heroku offer’s 5 free applications in their free tier account. Once you have created your account, create 2 applications. I named mine flask-api-staging and flask-api-prod.

You can create the applications via cli or the ui, from the ui it will look more or less like this:

Select an app name and check if the name is available then select create. Note down the name and config as we will use it in our .gitlab-ci.yml config:

Heroku API Key

To allow the deployment of applications to Heroku from Gitlab, we need to generate a API Key on Heroku and save the config in Gitlab.

Head over to your Heroku Dashboard, select Account Settings, scroll to the API Key section and generate a API Key.

Head over to your Gitlab Repository, select Settings, CI/CD, then select Variables and enter the Key: HEROKU_API_KEY and the Secret of the API Key into the Value and select Save Variable.

We will reference this variable from our deploy steps.

Heroku Postgres Add-on

Heroku offers a free Postgres Add-On, to activate: Select your application, select Resources, search for the Add-on Heroku Postgres, select and select the Hobby Dev Free version and select provision.

Our Application Code

Clone your repository then let’s start by creating our Flask API. Note this is more on Gitlab CI/CD than going into detail into the Flask Application.

Create the files that we will need:

$ touch app.py config.cfg requirements.txt tests.py Procfile

Let’s start by populating our configuration for our flask app: config.cfg

#SQLALCHEMY_DATABASE_URI='sqlite:///database.db' SQLALCHEMY_TRACK_MODIFICATIONS=False

Our Flask Application: app.py

Note that we are using flask-heroku, with this package Heroku will automatically discover your configuration for your database using environment variables.

So if you have a postgres add-on, you don’t need to specify the location of your database.

If you want to use sqlite, you can remove the heroku instantiation and uncomment the SQLALCHEMY_DATABASE_URI property in your config.cfg

from flask import Flask, jsonify, request
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
from flask_heroku import Heroku
from passlib.hash import sha256_crypt
from datetime import datetime
app = Flask(__name__)
app.config.from_pyfile('config.cfg')
heroku = Heroku(app)
db = SQLAlchemy(app)
ma = Marshmallow(app)
## --Database Models--
class Member(db.Model):
__tablename__ = 'members'
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
email = db.Column(db.String(255), unique=True, nullable=False)
username = db.Column(db.String(50), unique=True)
password_hash = db.Column(db.String(100))
firstname = db.Column(db.String(50), unique=False)
lastname = db.Column(db.String(50), unique=False)
registered_on = db.Column(db.DateTime, nullable=False)
class MemberSchema(ma.ModelSchema):
class Meta:
model = Member
fields = ('id', 'username', 'email')
member_schema = MemberSchema(strict=True, only=('id', 'username'))
members_schema = MemberSchema(strict=True, many=True)
## --Views--
@app.route('/')
def index():
return jsonify({'message': 'ok'}), 200
# list users
@app.route('/api/user', methods=['GET'])
def list_users():
all_users = Member.query.all()
result = members_schema.dump(all_users)
return jsonify(result.data)
# get user
@app.route('/api/user/<int:id>', methods=['GET'])
def get_user(id):
user = Member.query.get(id)
result = member_schema.dump(user)
return jsonify(result.data)
# add user
@app.route('/api/user', methods=['POST'])
def add_user():
email = request.json['email']
username = request.json['username']
password_hash = sha256_crypt.encrypt(request.json['password'])
firstname = request.json['firstname']
lastname = request.json['lastname']
new_user = Member(email=email, username=username, password_hash=password_hash, firstname=firstname, lastname=lastname, registered_on=datetime.utcnow())
try:
db.session.add(new_user)
db.session.commit()
result = member_schema.dump(Member.query.get(new_user.id))
return jsonify({'member': result.data})
except:
db.session.rollback()
result = {'message': 'error'}
return jsonify(result)
# update user
@app.route('/api/user/<int:id>', methods=['PUT'])
def update_user(id):
user = Member.query.get(id)
username = request.json['username']
email = request.json['email']
user.email = email
user.username = username
db.session.commit()
return member_schema.jsonify(user)
# delete user
@app.route('/api/user/<int:id>', methods=['DELETE'])
def delete_user(id):
user = Member.query.get(id)
db.session.delete(user)
db.session.commit()
return jsonify({'message': '{} has been deleted'.format(user.username)})
if __name__ == '__main__':
app.run()

Our tests: tests.py

import unittest
import app as myapi
import json
import sys
class TestFlaskApi(unittest.TestCase):
def setUp(self):
self.app = myapi.app.test_client()
def test_hello_world(self):
response = self.app.get('/')
self.assertEqual(
json.loads(response.get_data().decode(sys.getdefaultencoding())),
{"message": "ok"}
)
if __name__ == '__main__':
unittest.main()

Our requirements file: requirements.txt

Click==7.0
Flask==1.0.2
flask-heroku==0.1.9
flask-marshmallow==0.9.0
Flask-SQLAlchemy==2.3.2
gunicorn==19.9.0
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
marshmallow==2.17.0
marshmallow-sqlalchemy==0.15.0
passlib==1.7.1
psycopg2-binary==2.7.6.1
six==1.12.0
SQLAlchemy==1.2.15
Werkzeug==0.14.1

Our Procfile for Heroku: Procfile

And lastly, our gitlab-ci configuration which will include our build, test and deploy steps. As soon as a commit to master is received the pipeline will be acticated.

Note that our production deploy step is a manual trigger.

Our config for .gitlab-ci.yml. Note to replace your Heroku app names.

image: rbekker87/build-tools:lateststages:
- ver
- init
- tests
- deploy
ver:
stage: ver
script:
- python --version
- whoami
init:
stage: init
script:
- apk add postgresql-dev --no-cache
- pip install psycopg2-binary
- pip install -r requirements.txt
run_tests:
stage: tests
script:
- apk add postgresql-dev --no-cache
- pip install psycopg2-binary
- pip install -r requirements.txt
- python tests.py
deploy_staging:
stage: deploy
script:
- git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/flask-api-staging.git
- git push heroku master
- echo "Deployed to Staging Server https://flask-api-staging.herokuapp.com"
environment:
name: staging
url: https://flask-api-staging.herokuapp.com/
only:
- master
deploy_production:
stage: deploy
script:
- git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/flask-api-prod.git
- git push heroku master
- echo "Deployed to Production Server https://flask-api-prod.herokuapp.com"
environment:
name: production
url: https://flask-api-prod.herokuapp.com/
when: manual
only:
- master

Send to Gitlab:

Once everything is populated, stage your changes, commit your work and push to master:

$ git add . 
$ git commit -m "blogpost demo commit"
$ git push origin master

Once the code has been pushed to master, gitlab will pick it up and trigger the pipeline to run.

Gitlab Pipelines

Head over to Gitlab, select CI/CD -> Pipelines, you should see a running pipeline, select it, then you should see the overview of all your jobs:

If everything has passed you should see the Passed status as shown above.

You will notice that the staging environment has been deployed. Now you can do some testing and when you are happy with it, you can select the play button which will deploy to production on the pipelines dashboard.

Creating the Tables on Postgres

Before we can interact with our API, we need to provision the postgres tables from the database models that we wrote in our application.

Open up a Python shell on Heroku and initialize the tables:

$ heroku run python -a flask-api-prod 
>>> from app import db
>>> db.create_all()
>>> exit()

Testing the API:

Now that everything is up and running, its time to test our API.

List the users:

$ curl https://flask-api-staging.herokuapp.com/api/user 
[]

Create a User:

$ curl -H 'Content-Type: application/json' \
-XPOST https://flask-api-staging.herokuapp.com/api/user \
-d '{"username": "ruanb", "password": "pass", "email": "r@r.com", "firstname": "ruan", "lastname": "bekker"}'
{
"member": {
"id": 1,
"username": "ruanb"
}
}

List Users:

$ curl -H 'Content-Type: application/json' \
-XGET https://flask-api-staging.herokuapp.com/api/user
[
{
"email": "ruan@r.com",
"id": 1,
"username": "ruanb"
}
]

Update a User’s email address:

$ curl -H 'Content-Type: application/json' \
-XPUT https://flask-api-staging.herokuapp.com/api/user/1 \
-d '{"username": "ruanb", "email": "ruan@r.com"}'
{
"id": 1,
"username": "ruanb"
}

Retrieve a single user:

$ curl -H 'Content-Type: application/json' \
-XGET https://flask-api-staging.herokuapp.com/api/user/1
{
"email": "ruan@r.com",
"id": 1,
"username": "ruanb"
}

Delete User:

$ curl -H 'Content-Type: application/json' \
-XDELETE https://flask-api-staging.herokuapp.com/api/user/1
{
"message": "ruanb has been deleted"
}

Troubleshooting

I had some issues with Heroku, where one was after I deployed, I received this error in Heroku’s logs:

code=H14 desc="No web processes running" method=GET path="/"

I just had to scale my web dyno to 1:

$ heroku ps:scale web=1 -a flask-api-staging 
Scaling dynos... done, now running web at 1:Free

Have a look at their documentation if you need help with the heroku cli.

And to troubleshoot within the dyno, you can exec into it by running this:

heroku ps:exec -a flask-api-staging

I seriously dig Gitlab-CI and with this demonstration, you can see how easy it is to set up a CI/CD Pipeline on Gitlab and Deploy them to Heroku.

Resources:

The code for this demo is available at: gitlab.com/rbekker87/demo-cicd-flask-heroku

For more blog posts on Gitlab, have a look at my gitlab category on blog.ruanbekker.com

Thank You

Please feel free to show support by, sharing this post, check out my other work or reach out to me on twitter:@ruanbekker if you want to say hi.

Thanks for reading!


Originally published at blog.ruanbekker.com on January 5, 2019.

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Ruan Bekker

Written by

Systems Development Engineer and Open Source Enthusiast

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts. Medium’s largest DevOps publication.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade