Deploying Python, NodeJS & VueJS as Micro-Services
In this article I cover a number of different concepts and technologies; Web Sockets, Python, Node, Vue JS, Docker, and bring them all together in a microservice architecture.
Each component in this stack will be a microservice. I will take you through building each one, including running them in separate Docker containers.
Microservices bring with them immense flexibility, scalability, and fault tolerance. Today we will only cover the flexibility. Scalability and Fault tolerance require other technologies such as Kubernetes which are outside the scope of this article.
The flexibility of Microservices is created by each service being independent of the others and importantly only doing one job.
Taking the example we are looking at in this article, we could easily swap out the VueJS front end and replace it with react.
Or we could add a Mongo DB Microservice and store all tweets for analysis at a later date.
I have setup a demo site of what I am going to develop in this article. You can view it here http://medium-microservices.simoncarr.co.uk/
Of course this is just the VueJS front end, but underneath all the goodness of Python, Node, WebSockets and Docker are at work.
My approach to this project
Here is how I am going to approach this project
- Develop the Python app and confirm we are receiving tweets
- Develop the Web Socket service
- Connect the Python App to the Web Socket service
- Develop the Vue JS front end and connect it to the WebSocket Service
- Create docker images for each of the 3 Micro Services
- Create a Docker stack using docker-compose
- Deploy the stack
Code on GitHub
All of the code for this project is available on GitHub
Twitter Client (Python)
Websocket server (NodeJS)
Twitter stream UI (VueJS)
docker-compose file https://github.com/simonjcarr/medium_microservices_docker_compose
The images used in the docker-compose file need to be built first. You can use the Dockerfile’s in the above repos or follow the rest of this article to learn how to do it.
Creating a twitter app
Before we can receive tweets from Twitter, we will have to register an app at https://developer.twitter.com/
Once you have registered an App, a set of API keys will be generated that we can use to connect to the Twitter API.
- In your browser navigate to https://developer.twitter.com/. If you don't already have one register an account.
- After logging in, click on
- Hover your mouse over your username and from the dropdown menu, click
Create Appand fill in the form you're presented with.
- Once your app is created, you can retrieve your API credentials. There are two sets of credentials
Consumer API keysand
Access token & Access token secret. You will need both sets of keys shortly for use in the Python App.
Create a project file structure
Create a folder called
We will create folders for each of our individual microservices as we go along. For now, we will just need another folder for the Python twitter client.
In the microservices folder, create a subfolder called
Creating the Python Twitter client
Make sure you are using Python 3. As of writing this article, I am using
I would recommend that you use a virtual python environment. I’m going to use pipenv. If you don’t already have pipenv you can install it with
pip install --user pipenv
making sure you're in the
twitter_client folder and run
pipenv install tweepy python-dotenv
I’m going to be using Tweepy to connect to the twitter API. I am also installing
python-dotenv . This will allow me to put the Twitter API keys in a file called
.dot . This file will not be uploaded to git, so I know my Twitter API keys will remain secret.
Create two new files
.env file in your code editor of choice and add the following lines. Replace the place holders <…> with the relevant keys from the app you registered in your twitter developer account.
twitter_client.py and add the following code.
In the code above, I create an auth object from
tweepy then I create a class
TwitterListener that extends
on_status method simply prints the text of each tweet that is received. To start receiving tweets, I instantiate TwitterListener, create a new stream, and then add a filter to only receive tweets that contain one or all of the following values
If you run
twitter_client.py now you should see a stream of tweets scroll up the terminal. It will continue to display new tweets in real-time until you stop the script.
If like me, you are using pipenv, you can run the script like so
pipenv run python twitter_client.py
Creating the Web Socket service
Now we know we can receive tweets, I am going to create the Web Socket service. I will use NodeJS to create this service.
If you don’t have NodeJS already installed you will need to visit the NodeJS Website https://nodejs.org/ and follow the instructions to download and install it.
I am using NodeJS version
Create a new folder in the
microservices folder called
Make sure you're in the
websocket_server folder and enter the following command
npm init -y
This command will create a
package.json file that will hold the project dependencies.
Enter the following command to install the
npm i ws
Create a new file called
app.js and open it in your code editor and add the code below
The web socket server code is surprisingly simple. I am running the server on port
8088 (line 3)
Whenever the server receives a new message (line 13) it will resend the message to any clients that are connected by calling the broadcast function I have created. This simply loops through each connected client, sending the data to each (line 8)
Start the server by running the following command in the console in the
You won't see anything yet, as the server is not receiving any data. We are going to sort that out now by updating the Python Twitter client.
Connect the twitter client to the web socket server
open a new terminal and navigate to the root of the
twitter_client folder that holds the python code.
We need to install a python module that will create a WebSocket client and connect to the WebSocket server. Enter the following command
pipenv install websocket-client
Using the client is even simpler than the WebSocket server and only requires 3 lines of code. Update
twitter_client.py so it contains the following code.
The lines I have added are
- Line 4 import
- Line 8 creates a new connection and assign to a variable
- Line 17 Each time a new tweet arrives, send the text of the tweet to the server
I have also imported
json on line 5. The status object created by the tweepy module provides a number of items that represent each tweet. It also includes a
_json item that represents the original raw JSON received from Twitter. I’m using
json.dumps() to convert the JSON to a string that can be sent through the WebSocket connection.
You can now restart the
twitter_client.py with the command
pipenv run python twitter_client.py
If you open the terminal where the
python twitter client is running, you’ll see tweets scrolling up the screen.
I’m going to leave the console debug messages in the code until the VueJS client is complete, so I know everything is running.
Creating the VueJS frontend
This is where I start bringing it all together in a pretty front end, so tweets can be viewed in the web browser.
Open a new terminal window and navigate to the
If you don’t already have the VueJS CLI installed, enter the following command to install it.
npm i -g @vue/cli
You can find out more about VueJS at https://vuejs.org/
Now create a new Vue app by entering the following command.
vue create twitter_ui
You will first be prompted to pick a preset, use the up/down arrows on the keyboard to select
Manually select features and hit enter.
Now you're asked to select the features you want to install. You select/deselect features by using the up/down arrows and using the space bar to toggle a feature on or off.
Here are the features you should choose for this project, make sure to select the same in your project. If any of these features are not available in your list of options, try running
npm i -g @vue/cli to make sure you have the latest version.
Hit enter when you're finished. You will be asked to select the configuration for each of the features you selected. Here are the choices you should make
- Choose a version of Vue.js:
- Pick a CSS pre-processor:
Sass/SCSS (with node-sass)
- Pick a linter:
ESLint + Prettier
- Pick additional Lint features:
Lint on save
- Where to place config files:
In dedicated config files
- Save this as a preset:
Once the installation has completed, cd into the folder
twitter_ui created by the Vue CLI.
I am going to style this app using
tailwind CSS Installing and configuring tailwind in Vue is easy, simply enter the command below.
vue add tailwind
When prompted, choose
Minimal. Job done!
It’s helpful here to open the folder in your code editor. I use VS Code, so I just enter the following command.
Once the editor has opened, back in the terminal start the app with the following command
npm run serve
Then open a browser and navigate to the URL that the app says it is running on. In my case that is
You should see the default Vue app in your browser.
Create a new file
/src/components/Header.vue with the following code.
Tweets.vue, open the renamed file and replace the contents as below. (we will come back to this file shortly)
Open the file
/src/App.vue this file provides the layout for our app. We import the two components above and tell Vue where to display them and I also add a sprinkling of tailwind CSS classes. Update the code in
App.vue as below.
That is the basic structure of the app complete.
Now I will get the
Tweets component talking to the WebSocket Server and displaying the incoming tweets. Open
/src/components/Tweets.vue and update it as per the code below.
data() I have created two variables,
tweets which will hold the tweets received in an array and
connection which will hold the WebSocket connection object.
mounted() I connect to the WebSocket server and set up an
onmessage event. This is triggered whenever the server broadcasts data. When the event is triggered the function converts the data to a JSON object with
JSON.parse() and pushes it to the top of the
tweets array using
tweets array contains more than 20 tweets, the last tweet in the array is removed using
onopen event is for debugging and simply logs to the console one time when the client establishes a connection with the server
template loops through each
tweet held in
tweets and lays them out as a list. Each tweet includes a plethora of data. I’m pulling just a few items from each tweet
If you look at the app in your browser you will see the tweets scrolling through the page in real-time as the good people on twitter send them.
Try sending a tweet that includes one or all the words
python and watch your browser as appears a few seconds later. Feel free to mention me
@simonstweet along with a link to this article.
Dockerising each microservice
As the saying goes (in the UK at least), “there’s more than one way to skin a cat”. I am going to use docker because I believe that’s the best approach for a number of reasons. This is not a docker tutorial, however. If you have not used Docker before you might feel a little overwhelmed, I know I was the first time I came across it. There are a lot of resources on the internet that provide great introductions to docker, YouTube might be your best bet initially.
If you don’t have Docker installed, you take a look at the Docker official website.
The approach I will take
There are a number of different options for deploying containers
I am going to go with the simplest approach in this tutorial, which is to host them on my dev laptop.
The process will be
- Some small changes to each microservice to make them Docker friendly
- Create a Docker image for each container
- Create a docker-compose file that builds containers from each image and configures them to talk to each other over the Docker network stack.
Creating the Docker images
- Python Twitter Client
Open a terminal and navigate to your python twitter client folder, for me that is
Create a new file called
requirements.txt in the root of the folder. Our Docker container will be running Python 3 and will have PIP available. When we create a container from the Docker image, PIP will use the requirements.txt file to make sure the required dependencies are installed. In our case that is
Add the following lines to
Create a new file called
Dockerfile and add the code below
Before we create a Docker image from the Dockerfile, we need to tell Docker how to access the environment variables for the Twitter API. There’s also a problem with the URL to the WebSocket server, it’s hardcoded in
twitter_client.py This is an issue because each docker container is a self-contained system in its own right, so localhost refers to that container. I need to provide a way to tell the Docker container the address of the WebSocket container. I will do that later in a file called docker-compose.yml. For now, I need the Python script to be able to access the environment variables that will be in the docker-compose file.
twitter_client.py in your editor and make sure the code is updated as below.
os.environ, it provides access to the environment variables that will be stored in the docker-compose file.
For now, I’m just going to build the image from the Dockerfile by running
docker build -t microservices_twitter_client .
The above command tells Docker to create a new image called
microservices_twitter_client , the
. at the end tells Docker it can find the
Dockerfile in the current folder.
2. Websocket Server
In the terminal navigate to the folder holding the code for the Websocket server. For me that’s
Create a new file called
Dockerfile . Just like before we will define the Dockerfile image for this microservice in a Dockerfile and add the following code.
In this Dockerfile, I am using Node version 12 as the base image for the container. The command
WORKDIR create a new directory
/app in the image and tells Docker to use this as the working directory (base directory) for all further commands. So where you see
./ actually refers to
I then use
COPY to copy any files starting with
package and ending with
.json into the working directory. Then
RUN npm install which will install all the dependencies for our application. Once the install is complete
COPY app.js into the working directory.
Line 11, makes port 8088 available to be mapped to the outside world, so other containers or apps can connect to the WebSocket server. You will see how that is used later when we create a
docker-compose.yml file that will define the application stack.
Finally, on line 13, I tell Docker to run the command
You can now build this image so it’s available to use later with
docker build -t microservices_websocket_server .
3. VueJS Application
In a terminal navigate to your VueJS application folder. For me that is
As part of the
docker build command I will run
npm install this will create the
node_modules folder inside the container and ensure it has the latest updated dependencies. As such I don’t want the
node_modules folder on my development machine copying into the container. This is acomplished by creating a
.dockerignore file and listing the files that we want Docker to ignore.
Create a new file in the root of the application folder and call it
.dockerignore it only needs one line adding to it.
Environment variables in VueJS
Our app currently has a hardcoded URL to the WebSocket server. The best practice with VueJS is to create a
.env file in the root of the application for variables our application needs access to. There is a lot more to .env files when you get into the details of different environments such as Dev, Test, PreProd, and Prod. We will keep it simple here and just create a single .env file.
In the root of the VueJS application create a file called
Add this single line of code to the file.
Tweets.vue which is located in
process.env.VUE_APP_WEBSOCKET_SERVER_URL when you're done the whole line should look like this
Creating the Dockerfile
Create a new file in the root of the application folder called
Dockerfile and add the code shown below.
This Dockerfile is similar in structure to the others. A key difference here is that to deploy the application into production, we need to first build it. The process creates a
http-server which I do on line 3.
Following that, I go through a similar process as I did for the NodeJS WebSocket server. Once the files have been copied into the container, I build the application with
npm run build. This creates a
dist folder to hold all the build files.
Finally, I run the
http-server and tell it to serve the
Pull everything together with a docker-compose file
If you don’t have docker-compose installed (it does not come with docker) you can visit https://docs.docker.com/compose/install/ to find out how to install it on your OS.
Finally, we are almost done, just one last thing to do before we start our Microservices application stack.
We need a way to tell Docker what that stack comprises of, the relationship between each of the containers and the configuration for each container, i.e. Environment variables and what port each container should expose to the outside world.
Create another folder in the microservices folder at the same level as
websocket_server folders. Name it
Navigate into the new Docker folder and create a new file called
docker-compoes.yml and add the following code. Take care to maintain the correct indentation. yml files are sensitive to an indentation that is not consistent.
The docker-compose file lists the
services that I want to run in this stack.
I have created three services
Docker runs it’s own internal network and assigns it’s own IP Addresses internally. The service names are also essentially hostnames for each service on the docker network and will map to an IP Address inside docker.
You can see that I make use of this on
line 12 where I set the value for the environment variable
Environment variables for the Twitter App are set in the
twitterclient service. You can get these from the Twitter developer's website and the App that you created earlier.
We also have some dependencies in our stack. Both the VueJS App and the Python Twitter Client, rely on the WebSocket server being up and running before they start. If this wasn’t the case, there would be no server for them to connect to.
You can see that each service has an image. You should recognize this as the image we created when we ran docker build for each of our services.
websocketserver and the
twitterui both require that their internal ports be made available outside the container. This is achieved by mapping
internal_port In this case, both internal and external ports are the same, but they don’t have to be and often are not.
Running the stack
In order to make sure that everything is running correctly I will first run the stack in what is called
attached mode. This means the stack will only be available for as long as the terminal is open. This is not ideal for production use, but for testing it means I get to see any errors that might be generated. I also still have the
console.log statement logging to the console, which will help me know that everything is working.
Run the following command in the terminal in the
docker folder, the same folder that contains the
After a few seconds and if everything went well, you should see tweets in the form of JSON scrolling up the terminal.
Now open your browser and visit
http://localhost:8080 You should see tweets streaming into the app in real-time.
If you now go back into the terminal and hit Ctrl+c, this will shut down the stack. Now run the stack again detached mode using the
docker-compose up -d
Once the stack is running, you will be back at the terminal and the stack will be running in the background.
You can see which containers are running by issuing the command
If you need to shutdown the stack when in detach mode, open a terminal, make sure you're in the same folder as your
docker-compose.yml file and issue the command
I have covered a lot in this article and there is much that I missed. Some important best practices are missing but would have diverted from the concept I was trying to put across.
In summary, though, you have seen how it is possible to stream real-time from Python into VueJS. You have also learned about how to deploy a WebSocket server, which is the central technology that glues this stack together.
You also saw how you can create Docker containers for apps and use docker-compose to define and create application stacks.
I hope you enjoyed this article, I enjoyed writing it.
Please leave comments below if you are struggling or let me know if you think there is a better way of achieving anything that I discussed here.