Building Full stack webApp using JS + Flask + Postgres + Firebase + Heroku

Published in

Analytics Vidhya

23 min readAug 21, 2021

Deploy a complete web app using frontend in vanilla JS, backend in Flask framework, Postgres for backend database connections, firebase for storage, sendgrid for sending emails and finally using Heroku cloud servers and Github workflow.

We will get through this using a very modular software engineering approach consisting of the following lifecycle :

Requirements analysis (Describe the requirements as per stakeholders)
Planning (team size estimate, cost estimate)
Designing (UI/UX flows, defining architechture)
Building (Designing the database models, developing the frontend, backend codebase)
Testing (Unit testing, github static code analysis, smoke testing)
Deployment (Deploy using Heroku servers in two staged — staging, production)

If you would like to skip directly to building/developement process, you are free to do so but I would really like you to atleast go over designing section before developement to get a feel of what we are going to build.

So, without waiting and respecting your time, let’s move directly to our first step :

➡️ Requirements gathering

This phase requires doing market research and collect all types of possible feedbacks and requirements from not only the end-users or chosen customers but for every stakeholder through various types of processes such as questionnaires, discussions, brainstorming sessions, workshops etc.

Obviously, we are not going to do that research and you can safely assume that I have done this research for you :)

Let’s say we are going to build a system where people can easily create a task on a board/card in just a click and keep it secure and safe and also we can track the user journey from the task creation to completion and encourage the user to finish the task as fast as possible and along with that don’t distract the user from their other work and especially give a great UI/UX feel.

This is the high level overview of the problem statement.
Now, as a Software developer we must ask lots of questions here but just wait a bit and all your questions will be answered soon.

Let’s write functional and non-functional requirements for our system —

Generally, we use SRS to write these requirements.

Functional —

User is able to sign up itself seamlessly through our sign up page / login page.
User is able to very easily create a task without any hinderance
User can apply push notifications for reminder through following means :
-> email
-> phone
User can change the status/tag of the task → by default we can show “todo” type tag
Schedular will send reminder to all users who have reminder set for those tasks
User is able to edit the task description, delete the task.

Non-functional —

User passwords should be secured safely
After logging out user, no one should be able to log back in some other user’s account using back button in browser.
Seamless UI/UX experience
great performance and no lagging between two operations
The system is up always for the user and handles the load if the user base grows.

🆕 We are also going to name our app : “flask-task-manager-app” or anything which you would like to have.

🌟 Now, awesome tools available for requirements gathering and analysis :

(We generally use UML diagrams for better modelling and visualization)

tools for better requirement analysis and modelling

➡️ Planning

This phase basically concerns with following four activities :

Project cost, cost and duration to develop the project, efforts and risks involved in the completion of the project.

The only costs you will be probably spending on are Internet connectivity, a laptop and most probably a few hours of your schedule and everything else will be freely hosted by various awesome service providers.
We can analyze and choose what’s the best option/choice for us at every stage of development.

We can’t really have these estimations right now so, probably we will skip this phase here but it’s good to know about them.

🌟 Now, awesome tools available for planning :

➡️ Design

So, now we come to design part where we will be describing how the architechture of the system looks like and how the UI/UX of the system will look like.

First, Keeping all our requirements in mind what all we have defined already above in the first phase, let’s start by designing our application by breaking it down into components on the basis of business logic because eventually, every component of the system has to serve some business logic that is related to it.

Now, let’s think about it one by one.

What type of component can be used for a scenario where a user who wants to go to his dashboard page wants to start creating those tasks?
Firstly, the user has to log in to the system, right ? or maybe if the user is new, then it can create its credentials using sign-up flow, right?

Yes, you got it, it has to be authorization module which can help the user do this above scenario.

What type of component can be used for a scenario where a user wants to see its personalized profile, I mean think about it, every user should be able to get its customized page, right?

So, here we can have some kind of users related module that can give us the required info about the user so that we can show the specific info to the logged-in user.

What type of component should be able to display all the already created tasks by the user, we need to show all the tasks which are already created by the user, right ? and we also need to show whatever control the user has on the tasks created like if it can edit, delete or subscribe to the notifications for that task? what’s the status of the task? Every task can be different.

So, here we can have some kind of task related module (let’s say we name it as product module) where it gives us all the task related info for that user.

What type of component should be able to give the user the ability to set notifications for a particular task by subscribing to that task notifications and also to take the user’s info for giving out notifications like email, phone, etc.

We have to allow the user to set the notifications or unsubscribe to them, right?

So, we need to have a notifications component for it which can help in subscribing / unsubscribing to the notifications for the particular tasks for that user.

Now, lastly, we can make some schedular component that can automatically send any automated mails which we can use to remind the user to complete their tasks until the user completes the task or it unsubscribes to the notifications.
Let’s call it CRON component (basically Unix based systems or in general every framework today has some kind of sceduling service which in a way is related to CRON jobs or scheduled jobs which is the name given to it).

👏 Well done if you reached here without skipping !

You now know your system requirements at a very high level.

Let’s name these as services as all these are serving some kind of business logic and providing a particular service — a work / job which it is meant to do without knowing what other components does.

As you can clearly see from the above picture, we need to create five components.

Now, the question is if these components have anything common between them ? Is there any relation between these components ?

It is logical to assume that these services can be created and made to work independently on its own without interfering with any other services.

Ofcourse at times you will find that we can leverage one service to intercommunicate and fetch the data to another service to fulfil the requirements for the latter service but in this case to keep it simple and flexible without having any current need for introducing dependency for any service over other, we can have all services independent.

This brings us to the world of microservices where each service can service acts independently and serves the purpose on its own but its loosely coupled so we can’t afford to have completely decoupled systems if we want to scale our system so we need to have some coupling — loose coupling.

So, let’s say we want our product service to know what user services can offer or what interface it has exposed for the outside world to communincate and leverage the information user service can give in order to complete its own requirement and in case of product service if any change is introduced then it can be easily done without affecting little to no change in other services.

We are going to allow all the services to be communicated either internally or from our frontend webapp using REST HTTP (s) protocol.

REST is useful for us as it is simple to implement and flexible to scale and deploy our services independently and enable communication through simple interfaces which clients can understand in terms of business goals associated with every service.

Tech stack we are going to choose as already defined above —

Python (Flask) for faster and simpler developement, pure vanillaJS for frontend as there is no much complexity involved but yes, we will surely see some improvements and learn why breaking our frontend app into small components can be useful — scalable idea that every new framework/library leverages into action, Firebase for secure storage of assets such as user profile pics, Heroku for server deployment on cloud and finally Postgres for database which easily integrates with Flask and Heroku supports direct connections with it which can make it very faster on production.

👉 Here comes our UI/UX part where we are going to see how our front end should look like.

Generally, there is a separate UI/UX design team which takes care of these components and there is a design system that is usually followed like all the buttons on every page should look almost same and this goes for every component that it should be consistent across pages of the webapp.

When you build a webapp, there are some common components like header, footer which we can write once and reuse it everywhere.
So, we should look for that and keep that in mind while writing our css.

I will share the link for my design components which I choose and probably you will like it too :)

Click for complete design describing the flows.

I will put some screenshots of the figma for you to have a feel for what we are building.

🌟 Now, awesome tools available for designing :

➡️ Building

Now, let’s start developement keeping in mind all the design that we have made for our system.

Let’s take certain decisions before we start.
How should we structure our directory to enable faster and easier developement ?

How should we go for structuring all our HTML, CSS, JS files and other assets ?

let’s first go for front-end part :

Also, How should we keep our python runtime environments independent of each other so that there are no packages / modules conflicts in terms of version or just separate our environment variables ?

let’s now go for back-end part :

Don’t worry ! We will go through each of the files why is it required for this webapp and how are we going to fulfill the requirement of the system using the following different components for frontend as well as backend.

Let’s start building front-end of our system and will focus on main components only as other components are naive to be implemented.

Home page — We need to build login forms and sign up modals.

2. Dashboard page — We need to build the task creation section, task buckets to be shown for already created task, subscribe/unsubscribe modals, uploading of profile pic modal, changing of task status modal and dropdown menu on header.

3. Change password page — We need to build the change password form in two parts.
Firstly, we need to authenticate that if the user is in our records in the first place, then only we proceed and give user option to change password.

We are going to build a sample function for fetching user data and setting the data on the page :

There’s a much better way to manage all your hosts for every microservice :

create a separate config file and put all your hosts there.

Break your code into separate components as some components are reusable and load them accordingly by writing the code once and reusing them as much.

Let’s design our data models.

Let’s ask us a question — what data should be stored for this webapp to fulfill the business requirement ?

Firstly, we want to store user’s information for login credentials and their firstname, lastname (while signing up).

Then, we may send them notifications on their phone, email so that has to be also stored at some point of time.

We will also need to uniquely identify the user using some id — “user_id”

Great, we identified all the requirements for the users information.

What format the data needs to be ? —

The data can be logically structured in the form of tables and also we give more importance to consistency over availability for this system (read about CAP).

Similarly, we can construct a relationship for task created by user and each task can be associated with a unique user_id.

Is there any relationship between the data stores ?

Therefore, we can see foreign key kind of relationship for user_id in both the data stores (we say them tables here from now onwards as we now are convinced to user RDMS).

Let’s convince ourselves that following tables will be used for our system.

user_id, list_id should be the primary keys as we need to uniquely identify the user in the users table and tasks in task_list table.

user_id is the foreign key in the task_list which references user_id in the users table.

It can be helpful in fetching data by performing complex joins if required.
Also, if you observe we can have some default values in task_list table as whenever we create our new task, it has to be tagged as “Todo”.

Similarly, we can tell we can have value of 0 for “is_email_pushed”, “is_phone_pushed” which signifes if the user has subscribed to the notifications through respective means for the task.

NOTE: We should always try not to store any confidential information in databases and try to get it from user only when user tries to use that feature and explicitly gives them permissions, that’s why we are also not storing any extra info while signing up, it may become obstacle to the naive user.
User privacy is must and we must respect that to maintain user’s trust.

Let’s start now our back-end of the system.

Build a complete API with Flask Blueprint

Before getting started, let’s create the directory structure as stated below :

Also, let’s use this awesome package manager “pipenv” which can help in keeping track of requirements.txt file (this file contains all dependencies with versions needed to run our flask app).

pipenv install flask

It will create a Pipfile containing all dependencies under [packages].

[packages]flask = "*"

Now, let’s understand Request and Response for our basic API and protocol method.

We are going to use JSON based messaging for sending both request and response and since we are only fetching some information we are going to use GET method.

We should define our endpoints logically so that any dev can understand without reading much documentation.

Before building exact API, usually we write a API contract which defines following :

endpoint : /user/fetch-infoMethod : GET  
request : user_id=1
response : {"user_id": "1", "firstname": "sourav", "lastname":  "kumar"}

Now, after deciding the contract, we write the following code (and let’s not dive into database connection with our app for now and use program in-memory) :

basic flask-app example

export FLASK_DEBUG=1
export FLASK_APP=app.pyflask run -p 8080

This will run our app on port 8080 and use default localhost with developement server running (debugger enabled), our app will be available on

http://127.0.0.1:8081/

And since we defined our route as /user/fetch-info which requires some args also, so we need to make a request from browser or terminal or better use Postman for testing :

http://127.0.0.1:8081/user/fetch-info?user_id=1

will fetch the following response :

response body: {    "firstname": "sourav",    "lastname": "kumar"}response status : 200

Is there something missing in here ? No not with the response but wouldn’t it be much better if we log our complete request arguments and also complete response object on the terminal for better information.

We can do this using following four decorators which you can read here :

@app.before_request
@app.after_request

Can you find a problem with this flow if we go on building more api’s with endpoints such as product/fetch-product-info, user/authorize-user etc…

If we build everything in app.py, then it will lead to very big problem of managing everything in one file and you will be lost while searching for something and also difficult to document everything in just one file.

The logical approach would be to divide the directories into services and then have separate app.py files for each of them — something flask Blueprint provides out of the box to manage big projects and scale developement process.

Let’s now go back to our earlier shown directory structure (but without other irrelevant files for now) :

A Blueprint is a way to organize a group of related views and other code. Rather than registering views and other code directly with an application, they are registered with a blueprint.

Now, let’s build the above API with use of blueprint.

user.py

app.py

See ? How easily now we can add more views and add more blueprints both defined separately and easily scale up the developement process.

Postgres connection with flask app

What is the fun without having your user’s data secure and stored in your database and giving user realtime experience by connecting your database to the flask app —

Let’s start then.

Firstly, let’s setup your database from here and get pgadmin.

pipenv install psycopg2-binary

Create your tables as shown in the design part or modify it according to your needs.

keeping our app.py same as before, change user.py to

This will also log the SQL query, number of rows affected, selected data and other much more useful info for debugging.

Don’t worry if you want to setup MYSQL and don’t want to setup Postgres —

You just need to setup Mysql from here and get Workbench.

Again you just need to change user.py —

pipenv install flask-mysql

Then,

from flaskext.mysql import MySQLmysql = MySQL()app.config['MYSQL_DATABASE_USER'] = "root"
app.config['MYSQL_DATABASE_PASSWORD'] = "password"
app.config['MYSQL_DATABASE_DB'] = "developement"
app.config['MYSQL_DATABASE_HOST'] = "localhost"mysql.init_app(app)

Small break here

Importance of environment variables

Did you saw some security issue in our code above ? We are exposing all our secret credentials in the source code ….. This shouldn’t be done at all and can compromise your database security.

So, what’s the solution ? Environment variables are at our rescue.

We should set our environment variables from terminal itself and generally avoid any secret credentials in any part of source code or create a .env file and ignore these files to be tracked by git by including in .gitignore .

export DATABASE_URL = "postgres://hhkjrxymeo:89hskhfks@kjhff/kjfskh"

and change in code be like

DATABASE_URL = os.getenv('DATABASE_URL')

So, from now onwards we will keep using these env variables as and whereever necessary.

Firebase integration

Now, we have our basic setup ready to be able to show basic information to our user — their personal details after they login from the app and their already created tasks in task bucket cards.

What if user wants to change / modify the default profile picture we provided ?

We need to have some secure storage for such objects — Firebase to our rescue, a faster and secure way to acheive our goal here.

Let’s go ahead and integrate firebase to our flask app.

Setup your firebase account from here and we are going to Pyrebase4 library.

We need to do very small changes to our app code wherever you have built your API to upload the profile pic.

Make sure you have following credentials (you can get these from firebase storage console) —

firebaseConfig = {"apiKey": "sdjhfdkjskdh43535hk","authDomain": "flask-app.firebaseapp.com","projectId": "flask-app","storageBucket": "flask-app.appspot.com","messagingSenderId": "98437593998","appId": "1:98437593998:web:983759kjh98759","measurementId": "H-43578678BHGH","databaseURL": ""}import pyrebasefirebase = pyrebase.initialize_app(firebaseConfig)
storage = firebase.storage()profile_pic = request.files['file']user_id = request.form["user_id"]APP_UPLOAD_FIREBASE_PATH = f"images/{user_id}.png"storage.child(APP_UPLOAD_FIREBASE_PATH).put(profile_pic)

You will see your profile pictures uploaded in firebase storage console under images directory with unique user_id identifier.

Sendgrid integrations for mailers and generating mail templates using Jinja2.

Let’s say we want to now increase user engagement through building our own automated mailer system — Sendgrid is at our rescue, efficient and secure way to reach your users through emails.

This will be helpful for two things —

Sending user notifications when user subscribes and unsubscribes to the task buckets.
To make CRON scheduler jobs.

Let’s first see how to integrate sendgrid. Open your verified sendgrid account from a trusted email and Setup sendgrid from here.

Keep your HTML emails under newly created templates directory and other directory structures remains same.
Btw, here is a great tool for creating awesome mails online.

pipenv install sendgrid

Remember to keep you sendgrid API keys safe and in .env

Jinja is a web template engine for the Python programming language and we will be using here to convert our HTML into template, something which flask understands and our web server can easily send the mail. It has special placeholders for dynamically generating the template on the fly using the variables we pass to it.

from sendgrid import SendGridAPIClientfrom sendgrid.helpers.mail import Mailimport jinja2

Now, you know how to send your custom mails through sendgrid.

Let’s also see how you can schedule your mails or any type of job for that matter using CRON scheduler in flask using Advanced Python Scheduler.

It’s very simple and efficient way to create CRON jobs using this flask extension and you can read more about it to create your own schedulers for heavy background tasks which doesn’t interfere with your main executable apps or services .

Finally, we are (almost) done with what all we decided to build in our design.

➡️ Testing

It’s now time to break our code !

Test it against all limits of the system. Think about edge cases.

Honestly, you should test your code throughly at every stage of developement, then it goes through code review process by your peers in your team and also we use static code analyzers to find any general errors and security issues in the code. We can use codeql on Github for our use cases.
Read about different types of testing.

➡️ Deployment

It’s time to finally deploy the system — the time we all are waiting to enable our users finally use the system made by us.

We will see all the processes for deploying our frontend system to Github pages and backend system to Heroku and how to manage all the services by integrating everything and keeping it up to serve the users requests and we will enable automatic deployment by creating a workflow pipeline which enables faster deployment.

Let’s push our code to Github using this and keep it public repository and then deploy your code using Github pages.

Everything else will be managed by Github and your page should be live on github alloted domain.

Before deploying our backend services to Heroku, we have to understand Heroku infrastructure and dynos — backbone of Heroku and also understand about pipelines.

Now, let’s prepare some files before deploying which helps Heroku to understand your app better.

Procfile

web: gunicorn wsgi:app

runtime.txt (change your version if there’s another version on your machine)

python-3.8.10

wsgi.py

from app import appif __name__ == '__main__':      port = int(os.environ.get('PORT'))      app.run(port=port)

Note: don’t set PORT variable for Heroku.

Now, firstly, we must upload our code to Github from here for better integration with Heroku (you can keep your repository private for better security if don’t want to expose it to outside world).

We are now going to deploy our backend on Heroku.

After first signing up, you will see following screen —

Let’s start by creating a new pipeline by clicking on “New” dropdown button on top right side of page as shown.

Now, fill in all the details and connect to your github account for better integration of deployment.

After integration with Github and creating pipeline, you can see we have various options for creating our workflow —

mainly, we will be using staging and production.

So, the idea is that we will enable automatic deployment from one of the branches on Github to Heroku’s staging app and we will test it, if everything looks good, then we can promote it to Production app and you can share your app with the outside world / users.

Now, you are at following screen. Click on Add app for both staging and production and name your apps.

Now, focus on the shown area below :

Now, click on the staging app and open dashboard for that staging app.

Then, open deploy tab of the page and enable automatic deployment by selecting this option and select the branch for automatic deployment.

If you have already pushed your code to Github, then for the first time you should go for manual deployment (after that every push to this branch will be automatically deployed to Heroku).

Note: Include your Pipefile as it will help Heroku during its build process to install all the dependencies.

You can see the build process logs in the activity tab of the app page.

Following are the most important tabs in the app dashboard page.

Now, most important use Heroku Postgres which it supports directly and so we need to get the credentials of the database.

Now, go to resources tab on the page.

Install Heroku addons — Heroku Postgres from this tab.

Then, open datastore by clicking on the postgres addons.

Here, you can see everything about your database and live connections.

Click on the settings and get your credentials.

Copy this URI value and keep it safe somewhere as you have to set it in your environment variable “DATABASE_URL” as we have used it in our code.

Now, Set your Environment variables (for database or any other secret keys used) using Config Vars in the settings tab on app page dashboard.
Also set buildpacks here — select it as heroku/python.

Finally, we can see our app live from this top right Open app.

If you have not set any view function for default routes, then clicking on this button and opening your website will give you “403 error — not found”.

You can now test your app using postman by hitting the URL for your API with defined requests and see if your response is 200 and body is same as we defined above in one of our API’s.

Also, you can set your default route “/”, for something like as shown below so that you should not get any error and also test your service.

@user.route('/', methods=['GET'])
def test_user_service():
    return ("<h1> User service is up and running !</h1>")

This will make sure that you will not get error on default route.

Now, go on and check if your app is available on following route :

https://flask-task-manager-app.herokuapp.com/user/fetch-info

with required requests.

Also, for schedulers you have to change your Procfile as shown below and read which scheduler you want to use from here :

worker: python app.py

Note: web process on Heroku will sleep after around 30 minutes of inactivity !

Don’t worry, it doesn’t means your app will die off, Once it receives the request it will spawn the dyno once again and will keep it active until again 30 minutes inactivity period.

This makes sure you are not charged unnecessarily and maybe a bad experience for first user who hits the request (as currently your user base will be small) but it’s not a problem as after that there will no problem for any other requests and also it spawns usually within 30 seconds again after sleeping.

Test your backend API’s and and if everything looks good promote it from staging to production.

Finally, open your config file for front-end system and set your hosts for fetch API’s to this Heroku domain hosts for the production (you can check by opening your production app dashboard page and click on open app).

Your frontend should be integrated with backend app now and your app should be live for end users to be able to use for their needs.

If you made it till here, you deserve million claps 👏👏👏👏

You really deserve something more — Let’s get it done !

How great would it be if you could simply see all your logs live on your machine inside your terminal ? (you can’t do it on Heroku using your free account) — It’s possible and actully very helpful while debugging your app.

Let’s start with very basic useful commands to get your app connected to terminal and see your live logs and server status.

First you have to login to Heroku and then you can start using further commands (login to your terminal as admin user).

heroku login

This will take you to new web page (if not then open the link manually which is shown on your terminal), click to login.

Come back to your terminal.

Check all your apps —

heroku apps

Check your server status and dyno types —

heroku ps -a flask-task-manager-app

Check your live server logs —

heroku logs --tail -a flask-task-manager-app

Even you can manage all your database connections and settings and see your live tables using more such awesome commands from here.

Great references for further reading and continuing your journey —

✔️ Very soon, I will be releasing all the code open source and will update it here once it is done as there are some issues which needs to be addressed before it is released.

UPDATE: Code is released and open-sourced.