Deploying Python, NodeJS & VueJS as Micro-Services

In this article I cover a number of different concepts and technologies; Web Sockets, Python, Node, Vue JS, Docker, and bring them all together in a microservice architecture.

3 Microservices

Each component in this stack will be a microservice. I will take you through building each one, including running them in separate Docker containers.

Why Microservices

Microservices bring with them immense flexibility, scalability, and fault tolerance. Today we will only cover the flexibility. Scalability and Fault tolerance require other technologies such as Kubernetes which are outside the scope of this article.

The flexibility of Microservices is created by each service being independent of the others and importantly only doing one job.

Taking the example we are looking at in this article, we could easily swap out the VueJS front end and replace it with react.

Or we could add a Mongo DB Microservice and store all tweets for analysis at a later date.

Demo site

I have setup a demo site of what I am going to develop in this article. You can view it here http://medium-microservices.simoncarr.co.uk/

Of course this is just the VueJS front end, but underneath all the goodness of Python, Node, WebSockets and Docker are at work.

My approach to this project

Here is how I am going to approach this project

• Develop the Python app and confirm we are receiving tweets
• Develop the Web Socket service
• Connect the Python App to the Web Socket service
• Develop the Vue JS front end and connect it to the WebSocket Service
• Create docker images for each of the 3 Micro Services
• Create a Docker stack using docker-compose
• Deploy the stack

Code on GitHub

All of the code for this project is available on GitHub

Twitter Client (Python)
https://github.com/simonjcarr/medium_microservices_twitter_client

Websocket server (NodeJS)
https://github.com/simonjcarr/medium_microservices_websocket_server

Twitter stream UI (VueJS)
https://github.com/simonjcarr/medium_microservices_twitter_ui

docker-compose file https://github.com/simonjcarr/medium_microservices_docker_compose

The images used in the docker-compose file need to be built first. You can use the Dockerfile’s in the above repos or follow the rest of this article to learn how to do it.

Creating a twitter app

Before we can receive tweets from Twitter, we will have to register an app at https://developer.twitter.com/

Once you have registered an App, a set of API keys will be generated that we can use to connect to the Twitter API.

1. In your browser navigate to https://developer.twitter.com/. If you don't already have one register an account.
2. After logging in, click on Developer portal
3. Hover your mouse over your username and from the dropdown menu, click Apps
4. Click Create App and fill in the form you're presented with.
5. Once your app is created, you can retrieve your API credentials. There are two sets of credentials Consumer API keys and Access token & Access token secret. You will need both sets of keys shortly for use in the Python App.

Create a project file structure

Create a folder called microservices

We will create folders for each of our individual microservices as we go along. For now, we will just need another folder for the Python twitter client.

In the microservices folder, create a subfolder called twitter_client

Creating the Python Twitter client

Make sure you are using Python 3. As of writing this article, I am using Python 3.7.6

I would recommend that you use a virtual python environment. I’m going to use pipenv. If you don’t already have pipenv you can install it with

pip install --user pipenv

making sure you're in the twitter_client folder and run

pipenv install tweepy python-dotenv

I’m going to be using Tweepy to connect to the twitter API. I am also installing python-dotenv . This will allow me to put the Twitter API keys in a file called .dot . This file will not be uploaded to git, so I know my Twitter API keys will remain secret.

Create two new files

• twitter_client.py
• .env

Open the .env file in your code editor of choice and add the following lines. Replace the place holders <…> with the relevant keys from the app you registered in your twitter developer account.

Open twitter_client.py and add the following code.

In the code above, I create an auth object from tweepy then I create a class TwitterListener that extends tweepy.StreamListener The on_status method simply prints the text of each tweet that is received. To start receiving tweets, I instantiate TwitterListener, create a new stream, and then add a filter to only receive tweets that contain one or all of the following values javascript, nodejs, or python

If you run twitter_client.py now you should see a stream of tweets scroll up the terminal. It will continue to display new tweets in real-time until you stop the script.

If like me, you are using pipenv, you can run the script like so

pipenv run python twitter_client.py

Creating the Web Socket service

Now we know we can receive tweets, I am going to create the Web Socket service. I will use NodeJS to create this service.

If you don’t have NodeJS already installed you will need to visit the NodeJS Website https://nodejs.org/ and follow the instructions to download and install it.

I am using NodeJS version 12.13.1

Create a new folder in the microservices folder called websocket_server

Make sure you're in the websocket_server folder and enter the following command

npm init -y

This command will create a package.json file that will hold the project dependencies.

Enter the following command to install the wsmodule

npm i ws

Create a new file called app.js and open it in your code editor and add the code below

The web socket server code is surprisingly simple. I am running the server on port 8088 (line 3)

Whenever the server receives a new message (line 13) it will resend the message to any clients that are connected by calling the broadcast function I have created. This simply loops through each connected client, sending the data to each (line 8)

Start the server by running the following command in the console in the websocket_server folder.

node app.js

You won't see anything yet, as the server is not receiving any data. We are going to sort that out now by updating the Python Twitter client.

Connect the twitter client to the web socket server

open a new terminal and navigate to the root of the twitter_client folder that holds the python code.

We need to install a python module that will create a WebSocket client and connect to the WebSocket server. Enter the following command

pipenv install websocket-client

Using the client is even simpler than the WebSocket server and only requires 3 lines of code. Update twitter_client.py so it contains the following code.

The lines I have added are

• Line 4 import create_connection
• Line 8 creates a new connection and assign to a variable ws
• Line 17 Each time a new tweet arrives, send the text of the tweet to the server

I have also imported json on line 5. The status object created by the tweepy module provides a number of items that represent each tweet. It also includes a _json item that represents the original raw JSON received from Twitter. I’m using json.dumps() to convert the JSON to a string that can be sent through the WebSocket connection.

You can now restart the twitter_client.py with the command

pipenv run python twitter_client.py

If you open the terminal where the python twitter client is running, you’ll see tweets scrolling up the screen.

I’m going to leave the console debug messages in the code until the VueJS client is complete, so I know everything is running.

Creating the VueJS frontend

This is where I start bringing it all together in a pretty front end, so tweets can be viewed in the web browser.

Open a new terminal window and navigate to the microservices folder.

If you don’t already have the VueJS CLI installed, enter the following command to install it.

npm i -g @vue/cli

You can find out more about VueJS at https://vuejs.org/

Now create a new Vue app by entering the following command.

vue create twitter_ui

You will first be prompted to pick a preset, use the up/down arrows on the keyboard to select Manually select features and hit enter.

Now you're asked to select the features you want to install. You select/deselect features by using the up/down arrows and using the space bar to toggle a feature on or off.

Here are the features you should choose for this project, make sure to select the same in your project. If any of these features are not available in your list of options, try running npm i -g @vue/cli to make sure you have the latest version.

Hit enter when you're finished. You will be asked to select the configuration for each of the features you selected. Here are the choices you should make

• Choose a version of Vue.js: 2.x
• Pick a CSS pre-processor: Sass/SCSS (with node-sass)
• Pick a linter: ESLint + Prettier
• Pick additional Lint features: Lint on save
• Where to place config files: In dedicated config files
• Save this as a preset: No

Once the installation has completed, cd into the folder twitter_ui created by the Vue CLI.

I am going to style this app using tailwind CSS Installing and configuring tailwind in Vue is easy, simply enter the command below.

vue add tailwind

When prompted, choose Minimal. Job done!

It’s helpful here to open the folder in your code editor. I use VS Code, so I just enter the following command.

code .

Once the editor has opened, back in the terminal start the app with the following command

npm run serve

Then open a browser and navigate to the URL that the app says it is running on. In my case that is http://localhost:8080/

You should see the default Vue app in your browser.

Create a new file /src/components/Header.vue with the following code.

Rename /src/components/HelloWorld.vue to Tweets.vue, open the renamed file and replace the contents as below. (we will come back to this file shortly)

Open the file /src/App.vue this file provides the layout for our app. We import the two components above and tell Vue where to display them and I also add a sprinkling of tailwind CSS classes. Update the code in App.vue as below.

That is the basic structure of the app complete.

Now I will get the Tweets component talking to the WebSocket Server and displaying the incoming tweets. Open /src/components/Tweets.vue and update it as per the code below.

In data() I have created two variables, tweets which will hold the tweets received in an array and connection which will hold the WebSocket connection object.

In mounted() I connect to the WebSocket server and set up an onmessage event. This is triggered whenever the server broadcasts data. When the event is triggered the function converts the data to a JSON object with JSON.parse() and pushes it to the top of the tweets array using unshift.

If the tweets array contains more than 20 tweets, the last tweet in the array is removed using pop()

The onopen event is for debugging and simply logs to the console one time when the client establishes a connection with the server

The component template loops through each tweet held in tweets and lays them out as a list. Each tweet includes a plethora of data. I’m pulling just a few items from each tweet screen_name, profile_image, text, followers, and following.

If you look at the app in your browser you will see the tweets scrolling through the page in real-time as the good people on twitter send them.

Try sending a tweet that includes one or all the words javascript, nodejs, or python and watch your browser as appears a few seconds later. Feel free to mention me @simonstweet along with a link to this article.

Dockerising each microservice

As the saying goes (in the UK at least), “there’s more than one way to skin a cat”. I am going to use docker because I believe that’s the best approach for a number of reasons. This is not a docker tutorial, however. If you have not used Docker before you might feel a little overwhelmed, I know I was the first time I came across it. There are a lot of resources on the internet that provide great introductions to docker, YouTube might be your best bet initially.

If you don’t have Docker installed, you take a look at the Docker official website.

The approach I will take

There are a number of different options for deploying containers

I am going to go with the simplest approach in this tutorial, which is to host them on my dev laptop.

The process will be

• Some small changes to each microservice to make them Docker friendly
• Create a Docker image for each container
• Create a docker-compose file that builds containers from each image and configures them to talk to each other over the Docker network stack.

Creating the Docker images

1. Python Twitter Client

Open a terminal and navigate to your python twitter client folder, for me that is /microservices/twitter_client

Create a new file called requirements.txt in the root of the folder. Our Docker container will be running Python 3 and will have PIP available. When we create a container from the Docker image, PIP will use the requirements.txt file to make sure the required dependencies are installed. In our case that is python-dotenv, tweepy, and websocket

Add the following lines to requirements.txt

python-dotenv
tweepy
websocket-client

Create a new file called Dockerfile and add the code below

Environment Variables

Before we create a Docker image from the Dockerfile, we need to tell Docker how to access the environment variables for the Twitter API. There’s also a problem with the URL to the WebSocket server, it’s hardcoded in twitter_client.py This is an issue because each docker container is a self-contained system in its own right, so localhost refers to that container. I need to provide a way to tell the Docker container the address of the WebSocket container. I will do that later in a file called docker-compose.yml. For now, I need the Python script to be able to access the environment variables that will be in the docker-compose file.

Open twitter_client.py in your editor and make sure the code is updated as below.

Notice the os.environ[], it provides access to the environment variables that will be stored in the docker-compose file.

For now, I’m just going to build the image from the Dockerfile by running

docker build -t microservices_twitter_client .

The above command tells Docker to create a new image called microservices_twitter_client , the . at the end tells Docker it can find the Dockerfile in the current folder.

2. Websocket Server

In the terminal navigate to the folder holding the code for the Websocket server. For me that’s /microservices/websocket_server

Create a new file called Dockerfile . Just like before we will define the Dockerfile image for this microservice in a Dockerfile and add the following code.

In this Dockerfile, I am using Node version 12 as the base image for the container. The command WORKDIR create a new directory /app in the image and tells Docker to use this as the working directory (base directory) for all further commands. So where you see ./ actually refers to /app

I then use COPY to copy any files starting with package and ending with .json into the working directory. Then RUN npm install which will install all the dependencies for our application. Once the install is complete COPY app.js into the working directory.

Line 11, makes port 8088 available to be mapped to the outside world, so other containers or apps can connect to the WebSocket server. You will see how that is used later when we create a docker-compose.yml file that will define the application stack.

Finally, on line 13, I tell Docker to run the command node app.js

You can now build this image so it’s available to use later with

docker build -t microservices_websocket_server .

3. VueJS Application

In a terminal navigate to your VueJS application folder. For me that is /microservices/twitter_ui

.dockerignore file

As part of the docker build command I will run npm install this will create the node_modules folder inside the container and ensure it has the latest updated dependencies. As such I don’t want the node_modules folder on my development machine copying into the container. This is acomplished by creating a .dockerignore file and listing the files that we want Docker to ignore.

Create a new file in the root of the application folder and call it .dockerignore it only needs one line adding to it.

node_modules

Environment variables in VueJS

With frontend Javascript apps we have to consider that the app is running in the browser rather than on the server. The implications of this are that environment variables on the server are not available to the app running in the browser.

Our app currently has a hardcoded URL to the WebSocket server. The best practice with VueJS is to create a .env file in the root of the application for variables our application needs access to. There is a lot more to .env files when you get into the details of different environments such as Dev, Test, PreProd, and Prod. We will keep it simple here and just create a single .env file.

In the root of the VueJS application create a file called .env

Add this single line of code to the file.

VUE_APP_WEBSOCKET_SERVER_URL=ws://192.168.30.100:8088

Now open Tweets.vue which is located in /src/components/Tweets.vue

Replace ws://localhost:8088 with process.env.VUE_APP_WEBSOCKET_SERVER_URL when you're done the whole line should look like this

Creating the Dockerfile

Create a new file in the root of the application folder called Dockerfile and add the code shown below.

This Dockerfile is similar in structure to the others. A key difference here is that to deploy the application into production, we need to first build it. The process creates a index.html file that contains references to minified javascript. That index.html file needs to be made available via a webserver. We could have set up a NGINX server container, but a simpler approach for this use case is to install an npm package http-server which I do on line 3.

Following that, I go through a similar process as I did for the NodeJS WebSocket server. Once the files have been copied into the container, I build the application with npm run build. This creates a dist folder to hold all the build files.

Finally, I run the http-server and tell it to serve the dist folder.

Pull everything together with a docker-compose file

If you don’t have docker-compose installed (it does not come with docker) you can visit https://docs.docker.com/compose/install/ to find out how to install it on your OS.

Finally, we are almost done, just one last thing to do before we start our Microservices application stack.

We need a way to tell Docker what that stack comprises of, the relationship between each of the containers and the configuration for each container, i.e. Environment variables and what port each container should expose to the outside world.

Create another folder in the microservices folder at the same level as twitter_client, twitter_ui, and websocket_server folders. Name it docker.

Navigate into the new Docker folder and create a new file called docker-compoes.yml and add the following code. Take care to maintain the correct indentation. yml files are sensitive to an indentation that is not consistent.

The docker-compose file lists the services that I want to run in this stack.

I have created three services

• websocketserver
• twitterclient
• twitterui

Docker runs it’s own internal network and assigns it’s own IP Addresses internally. The service names are also essentially hostnames for each service on the docker network and will map to an IP Address inside docker.

You can see that I make use of this on line 12 where I set the value for the environment variable WEBSOCKET_SERVER_URL to ws://websocketserver:8088

Environment variables for the Twitter App are set in the twitterclient service. You can get these from the Twitter developer's website and the App that you created earlier.

We also have some dependencies in our stack. Both the VueJS App and the Python Twitter Client, rely on the WebSocket server being up and running before they start. If this wasn’t the case, there would be no server for them to connect to.

You can see that each service has an image. You should recognize this as the image we created when we ran docker build for each of our services.

Finally the websocketserver and the twitterui both require that their internal ports be made available outside the container. This is achieved by mapping export_port: internal_port In this case, both internal and external ports are the same, but they don’t have to be and often are not.

Running the stack

In order to make sure that everything is running correctly I will first run the stack in what is called attached mode. This means the stack will only be available for as long as the terminal is open. This is not ideal for production use, but for testing it means I get to see any errors that might be generated. I also still have the console.log statement logging to the console, which will help me know that everything is working.

Run the following command in the terminal in the docker folder, the same folder that contains the docker-compose.yml file.

docker-compose up

After a few seconds and if everything went well, you should see tweets in the form of JSON scrolling up the terminal.

Now open your browser and visit http://localhost:8080 You should see tweets streaming into the app in real-time.

If you now go back into the terminal and hit Ctrl+c, this will shut down the stack. Now run the stack again detached mode using the -d switch

docker-compose up -d

Once the stack is running, you will be back at the terminal and the stack will be running in the background.

You can see which containers are running by issuing the command

docker ps

If you need to shutdown the stack when in detach mode, open a terminal, make sure you're in the same folder as your docker-compose.yml file and issue the command

docker-compose down

Conclusion

I have covered a lot in this article and there is much that I missed. Some important best practices are missing but would have diverted from the concept I was trying to put across.

In summary, though, you have seen how it is possible to stream real-time from Python into VueJS. You have also learned about how to deploy a WebSocket server, which is the central technology that glues this stack together.

You also saw how you can create Docker containers for apps and use docker-compose to define and create application stacks.

I hope you enjoyed this article, I enjoyed writing it.

Please leave comments below if you are struggling or let me know if you think there is a better way of achieving anything that I discussed here.