Building Microservices with Python , Part I
But at the end, I am a Software Engineer interested in many different areas. This post it is going to be focus on Backend Development and how I am building microservices on a personal project I am working.
You can find all the code in this repo: https://github.com/ssola/python-flask-microservice
Nowadays it is a common practice to work in smaller applications, sharing the responsibility among many different services. I believe it is critical to have some standard tools between the teams working solving those problems.
In my opinion a Microservices should have some basic features:
- Easy to start coding your logic, stop worrying about tools/patterns.
- Documentation, an essential feature to share how your service it is going to work. In this case, Swagger works pretty well.
- Serializing your input/output in a way shared among all applications. You need to chose a technology like Avro/Protobuf. This is mandatory to be sure all services are sharing the same entities.
- Events, probably your application it is going to generate events that can be consumed by others. For instance, in my project, every time a new room is inserted into the system, that piece of information is serialized with avro and spread among all the other services.
Stack and Patterns
In this basic setup we are going to include those packages:
- Flask (as a Framework)
- connexion (helpful tool to generate routes and Swagger docs)
- Flask-Injector (Dependency Injection package)
- Avro (or any data serialization package)
In this case, I chose Flask because I find it super useful to build small services without all the learning curve of Django. I just need something to help to do the routing and I will include whatever I need on it.
Connexion is a project from Zalando that adds a layer on top of Flask to help you building your RESTFul API in a simpler manner, with the great benefit to have at the end Swagger docs. Connexion gives you as well an elegant solution to protect your service behind oAuth2 and a way to versioning your API.
Dependency Injection is a nice way to inject the dependencies your need in your methods/classes. For this purpose, I chose Flask-Injector, as you can see by the name it’s completely integrated on Flask. With this tool using the decorator
@inject I can have the service I need, for instance, ElasticSearch or SQLAlchemy.
Did you have problems with services sending corrupted data or some payload with the wrong schema? Well, that can be solved using some Serialization tool like Avro or Protobuf. These tools help you to ensure whatever you are receiving or sending has the proper schema.
I am not covering all the topics related to Microservices, but this is a good starting point, and you are covering some of the important pieces of the puzzle.
Building your more than a Hello World service
Starting your environment
I am going to specify a step-by-step guide with al the steps you need to do to have a simple service up-and-running on your machine.
First of all, create a new virtual environment with Python 3.6.0. If you are not used to working with virtual environments in Python, stop reading and jump to this other article.
Then create a new project on your favorite IDE, I like to work with PyCharm. In the root of your project create a new
requirements.txt file with these dependencies:
Starting with Connexion
After installing the dependencies, we can start coding our
app.py script on the same path we placed the requirements file.
As you can see, we are already using the package connexion. It is pretty simple what it is doing there.
- Creates a new application defining to which port it will start (later on we can change which server we want to use to execute the service)
- Defines to which directory your swagger configurations are placed.
- You add a new api (you can have more than one in the same app), mainly the yaml file where you are going to write your routes and definitions.
At this point, we need to read a little bit how the connexion yaml file works.
Let’s create our first endpoint on the
In this example, you can start understanding some features of connexion.
- We have a
getendpoint on the
- When this endpoit it’s called it will execute the method
searchon the module
- This module is exposed on the
__init__.pyfile in the
- This endpoint will return an array of objects of type
Item, we defined this item as an object with an
Here you can see the code on the
Executing this endpoint, it will return a json like this:
"name": "First item"
And going to your
http://localhost:9090/ui you will find the Swagger docs.
Without that much effort, we have an up-and-running simple API returning items. Obviously, this is a silly example. So the next step is to make use of injection to inject an
ItemProvider in our
Working with Flask Injector
Dependency Injection it is a great pattern to help us to organize our dependencies among objects.
In software engineering, dependency injection is a technique whereby one object supplies the dependencies of another object. A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it. The service is made part of the client’s state. Passing the service to the client, rather than allowing a client to build or find the service, is the fundamental requirement of the pattern.
In our previous example, we have in the very same file a list of
Items , as you can imagine that is not the way we should retrieve data. We can connect to
databases to get this information. To achieve that, we can use the
Repository Pattern or create a
Data Service to make this example simple enough; we are going to create a Data Service where we can retrieve some data.
This simple example can be easily extended to get the data from any database or third party API.
Now, let’s bind this service and inject it in our search method. First of all, we need to do some changes on our
app.py , we need to add the bindings on the Flask Injector.
Here, as you can see, we have a new method called
configure , this will handle all the bindings to create the container. In this case, we are creating a new
ItemsProvider with some default items.
Now, in our
api.items we can inject this
This is great as well for testing. When we need to test our
search method we can create new bindings to a
mocked ItemsProvider. We are simplifying our dependencies and improving our code for testing purposes.
As I said, this is not a complicated example. But probably you can see now how you can inject SQLAlchemy into your ItemsProvider to get dat from a database.
Serializing your payloads
A common pitfall building services are sharing wrong payloads. For instance, you expect an Item object with a name and an id. But working with plain JSON is not possible to enforce it.
We can do it right now with your connexion yaml, it will throw exceptions if something is wrong. But in this case, we need to ensure the data between different services using for instance RabbitMQ.
We can use JSON as I said, but then for each application, you will need to have all the required validations. If your platform works with several programming languages that can be a hassle.
With some tools like Avro or Protobuf, we can have a common serialization objects among different programming languages. A good practice would be to have a repo where you have all your Avro/Protobuf serializers. Those can be used almost for any programming language.
In this case, Avro gives us:
- Rich data structures.
- A compact, fast, binary data format.
- A container file, to store persistent data.
- Remote procedure call (RPC).
- Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.
A simple way to work with Avro and Python is using this library called fastavro.
You can find many examples in the GitHub repo.
With these small steps we can have a microservice up-and-running pretty quickly. With some nice features as:
- Dependency Injection
- Global serializer
As you can imagine, this is not everything we need to do to have a production ready microservice. Here we have the steps to start building a solid skeleton.
In the next post I am going to write about other important features on microservices, like:
- Logging & Metrics
- Service Discovery
- Dev environment with Docker
Please, let me know with your comments if you find any flaw or you have better ideas about how to improve those ideas.
This is a very interesting book explaining some of the basic of microservices concepts.
Very useful to start getting all the concepts clear in your mind.