Announcing Relé!

Andrew Graham-Yooll
Nov 5, 2019 · 3 min read
Image for post
Image for post
Relé — A powerful Python integration for Google Cloud’s Pub/Sub

We are pleased to announce Relé, a comprehensive Python solution to get you up and running with Google Cloud’s Pub/Sub as quickly and easily as possible. Meant to ease development workflow, Relé comes built-in with key components for managing a project with Pub/Sub enabled services.

Out-of-the-box, Relé provides:

  • Powerful Publishing API
  • Highly Scalable Worker
  • Intuitive Subscription Management
  • Easily Extensible Middleware
  • Optional Django Integration
  • And much more!

Allowing our products to scale with ease, Pub/Sub is a vital piece of infrastructure at Mercadona Tech and we lean on Relé to manage publishing and subscribing to millions of messages.

Use Case

For many years, Python’s go-to task queuing library has either been Celery or RQ. Although on the face of it, Pub/Sub’s design may look similar, it is solving a different problem.

While Celery and RQ solve task queues within an application, Pub/Sub shines when it comes to managing task queues between many services. Leveraging Relé with Pub/Sub maintains an elegant API much like Celery and RQ’s, but instead allows for scalable messaging within a service oriented architecture.

Reasoning

Since the Google Pub/Sub client does not come built-in with solutions to database connection management, publisher management, declaring subscribers, and running workers, the Mercadona Tech team decided to coalesce around an open source solution.

To reduce complexity and risk when adopting Pub/Sub, we organized our experiences from the Python client into one comprehensive library to solve common use cases and patterns suitable for a highly-available production environment.

Getting Started

pip install rele

Or with Django:

pip install rele[django]

Configuring

For Relé to work, we must have a Cloud Pub/Sub topic. Via the Google Cloud Console we create one, named photo-uploaded.

Relé configuration looks like:

# settings.pyimport rele
from google.oauth2 import service_account

RELE = {
'GC_CREDENTIALS': service_account.Credentials.from_service_account_file(
'credentials.json'
),
'GC_PROJECT_ID': 'photo-uploading-app',
}
config = rele.config.setup(RELE)

Once the topic is created and our application has the proper configuration defined, we can start publishing to that topic.

Publishing

To publish, we simply pass in the topic to which we want to publish too, followed by a valid JSON serialize-able Python object.

import rele
from settings import config

data = {
'customer_id': 123,
'location': '/google-bucket/photos/123.jpg'
}
rele.publish(topic='photo-uploaded', data=data)

Subscribing

Once we can publish to a topic, we can subscribe to the topic from a Relé worker.

# subs.pyfrom rele import sub

@sub(topic='photo-uploaded')
def photo_uploaded(data, **kwargs):
print(f"Customer {data['customer_id']} has uploaded an image "
f"to our service, and we stored it "
f"at { data['location'] }.")

You will notice the simple API for both publishing and subscribing. We have taken inspiration from numerous other libraries such as Celery.

Consuming

# worker.pyfrom time import sleep
from rele import Worker
from settings import config
from subs import photo_uploaded
if __name__ == '__main__':
worker = Worker(
[photo_uploaded],
config.gc_project_id,
config.credentials,
config.ack_deadline,
)
worker.setup()
worker.start()
sleep(120)

Once the sub is implemented, we can initialize our worker. This will register the subscriber with Google Cloud and will handle pulling messages from the topic python worker.py.

If you are integrating Relé into your Django project, declaring workers are not necessary. Since the provided python manage.py runrele command will auto-discover your subs. The only thing necessary is modifying your settings.py file.

# settings.pyINSTALLED_APPS = [
...,
'rele'
]
RELE = {
'GC_CREDENTIALS': service_account.Credentials.from_service_account_file(
'credentials.json'
),
'GC_PROJECT_ID': 'photo-uploading-app',
'MIDDLEWARE': [
'rele.contrib.LoggingMiddleware',
'rele.contrib.DjangoDBMiddleware',
],
}

The code for these example can be found here.

We hope that Relé provides you and your project the ability to leverage Google Cloud’s Pub/Sub with ease.

For more information and guides, please see our docs or submit an issue.

And do not forget that contributions are always welcome!

Mercadona Tech

Somos el equipo detrás de la nueva experiencia online de…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store