Python conference Alicante 2019

In early October, Ruben Gallardo and Jesús Avellaneda attended the PyConES Python conference, where the Spanish community meets every year in different locations: this year it was based in Alicante. One of the best things in this event is that you can meet a wide range of people coming from different areas and professional backgrounds, such as data science, big data, machine learning, web development, etc…

PyConES is a three day conference where you can choose between talks and workshops. After attending a few talks and a couple of workshops, let us introduce you, sorted in general terms, what we learnt there:

General python

In this part, we will cover some of the talks connected to the Python language itself:

• Hashable Objects: by implementing our own hash table (AKA python dicts or Java Maps) we learnt about some important concepts and design decision for this fundamental data structure.
• f-strings: or string interpolation, that it’s a new core component that appears after Python 3.6. It is a new and easier way of formatting strings - you just need to prefix your string with a “f’” as below:

print(f'HOLA {username.upper()}')
> HOLA JESUS

• Design patterns: this was a talk to avoid spaghetti code and follow some best practices. The speaker covered some patterns such as dependency injection, factory method, facade, adapter…and he said that you don’t have to follow just one! The best thing is to try to orchestrate them making your code readable, robust and flexible.
• Exceptional exceptions: the speaker presented us some best practices and anti patterns regarding exception handling. We encourage everyone to watch the English version of this talk.

Asynchronous Python

The async programming model, is becoming more popular in Python after adding the asyncio module to the Python standard library. Since then we have seen the release of async web frameworks like Startlette ( which is used by other many web frameworks as their base ). Another important technology we saw is ASGI , an async replacement for WSGI, that sets a standard interface for web servers and async web apps. Likewise aiohttp is the replacement for requests. Lastly, we knew about the existence of trio: an async I/O library alternative to the built-in asyncio

Architecture

We attended two talks about micro-service frameworks as below:

• Tamarco: is compatible with asyncio and includes some features like lifecycle management, messaging patterns using AMQP and Kafka. One of the most important aspects is that it manages the dependencies sharing a core for all the micro-services avoiding repeated code and unnecessary coupling between them. It’s easy to extend via plugins and also it is instrumented using Prometheus
• Nameko: it has been built in support for RPC over AMQP, Asynchronous events over AMQP, web-socket RPC and subscriptions.

Airflow and Apache Bean

The last day of PyCon we could attend a 3-hours long workshop called “Flow is in the air” (code in https://github.com/javierillo88/flow-is-in-the-air) about Apache Beam and Airflow. Apache Beam defines a unified programming model to create data processing pipelines that eventually get executed by an engine as Spark or Google Cloud Dataflow. Since NiFi and Apache Beam are both based in dataflow programming, we asked how they differ in their use cases and the answer given by them was, Apache Beam/Spark is better for transformations while Apache NiFi works best for moving data.

They gave a talk about Airflow apart from the aforementioned workshop, there we could confirm we are in a good track regarding best practices and common trade-offs for this technology (e.g. don’t abuse XCOM inter-task messages, get a right granularity of task is tricky…).

Big Data

Sometimes when the amount of data is too large, we cannot use pandas due to the memory of our local machine is limited. In that case, we can use Dask. It is a tool with an API similar to Pandas but being able to process data in a distributed cluster.

In the talk called “Big Data En Live Coding” we learnt how to create a scalable pipeline to get and process streaming data using technologies from Python ecosystem instead of Scala and Spark (e.g ASGI, Starlette, uvicorn, NNG, Trio, Skein, Dask and streamz). We recommend having a look to the GitHub repo for this talk with a set of projects based on the popular game Flappy Bird.

Data science

Another area where Python shines is data science, thanks to the PyData Stack. The first day of the conference we attended the workshop called FTW2019 Introduction to Data Science in Python which covered the libraries that many people of the data science community use, such as Numpy, Pandas and Dask for data manipulation, Matplotlib for data visualisation and Scikit-learn to perform machine learning. We could also see an example of a NLP (Natural Language Processing) in Production for the Barcelona City Council.

Others

There were some others interesting talks that weren’t covered with the previous areas, so let us talk a bit about them:

• Opsdroid: the presenter showed a common issue that happened in his company but which could happen in any other company. A person of the devops or other team is on holidays and an action that only he or she has permissions is required, then this person has to be disturbed and maybe he/she doesn’t have a laptop near. The proposed solution is using a chatbot for chatops so that we can use our mobile phone to perform the action. Opsdroid and an external service (e.g. DialogFlow) to map natural text to intention can be used to create a functional bot in no time.
• pyenv: a version management tool that, allows install multiple Python versions, change the global version and letting the user change the version with an environment variable.

• nteract: it’s a complex app ecosystem that includes papermill (tool to parameterise, execute and analyse Jupyter Notebooks) among others. The presenter also added a concept connected to documentation coverage like creating doc templates for code comments in GitHub, making sure your documentation cover almost every method.

Finally, after spending 3 days learning and acquiring new knowledge about Python world, we came back to Malaga with a bunch of new and fresh ideas in order to apply in our daily work, helping our teams and the company.

We recommend anyone to attend this event that will take place in Granada next year. See you there!