Announcing Relé!
We are pleased to announce Relé, a comprehensive Python solution to get you up and running with Google Cloud’s Pub/Sub as quickly and easily as possible. Meant to ease development workflow, Relé comes built-in with key components for managing a project with Pub/Sub enabled services.
Out-of-the-box, Relé provides:
- Powerful Publishing API
- Highly Scalable Worker
- Intuitive Subscription Management
- Easily Extensible Middleware
- Optional Django Integration
- And much more!
Allowing our products to scale with ease, Pub/Sub is a vital piece of infrastructure at Mercadona Tech and we lean on Relé to manage publishing and subscribing to millions of messages.
Use Case
For many years, Python’s go-to task queuing library has either been Celery or RQ. Although on the face of it, Pub/Sub’s design may look similar, it is solving a different problem.
While Celery and RQ solve task queues within an application, Pub/Sub shines when it comes to managing task queues between many services. Leveraging Relé with Pub/Sub maintains an elegant API much like Celery and RQ’s, but instead allows for scalable messaging within a service oriented architecture.
Reasoning
Since the Google Pub/Sub client does not come built-in with solutions to database connection management, publisher management, declaring subscribers, and running workers, the Mercadona Tech team decided to coalesce around an open source solution.
To reduce complexity and risk when adopting Pub/Sub, we organized our experiences from the Python client into one comprehensive library to solve common use cases and patterns suitable for a highly-available production environment.
Getting Started
pip install rele
Or with Django:
pip install rele[django]
Configuring
For Relé to work, we must have a Cloud Pub/Sub topic. Via the Google Cloud Console we create one, named photo-uploaded.
Relé configuration looks like:
# settings.pyimport rele
from google.oauth2 import service_account
RELE = {
'GC_CREDENTIALS': service_account.Credentials.from_service_account_file(
'credentials.json'
),
'GC_PROJECT_ID': 'photo-uploading-app',
}
config = rele.config.setup(RELE)
Once the topic is created and our application has the proper configuration defined, we can start publishing to that topic.
Publishing
To publish, we simply pass in the topic to which we want to publish too, followed by a valid JSON serialize-able Python object.
import rele
from settings import config
data = {
'customer_id': 123,
'location': '/google-bucket/photos/123.jpg'
}
rele.publish(topic='photo-uploaded', data=data)Subscribing
Once we can publish to a topic, we can subscribe to the topic from a Relé worker.
# subs.pyfrom rele import sub
@sub(topic='photo-uploaded')
def photo_uploaded(data, **kwargs):
print(f"Customer {data['customer_id']} has uploaded an image "
f"to our service, and we stored it "
f"at { data['location'] }.")
You will notice the simple API for both publishing and subscribing. We have taken inspiration from numerous other libraries such as Celery.
Consuming
# worker.pyfrom time import sleep
from rele import Workerfrom settings import config
from subs import photo_uploadedif __name__ == '__main__':
worker = Worker(
[photo_uploaded],
config.gc_project_id,
config.credentials,
config.ack_deadline,
)
worker.setup()
worker.start()
sleep(120)
Once the sub is implemented, we can initialize our worker. This will register the subscriber with Google Cloud and will handle pulling messages from the topic python worker.py.
If you are integrating Relé into your Django project, declaring workers are not necessary. Since the provided python manage.py runrele command will auto-discover your subs. The only thing necessary is modifying your settings.py file.
# settings.pyINSTALLED_APPS = [
...,
'rele'
]RELE = {
'GC_CREDENTIALS': service_account.Credentials.from_service_account_file(
'credentials.json'
),
'GC_PROJECT_ID': 'photo-uploading-app',
'MIDDLEWARE': [
'rele.contrib.LoggingMiddleware',
'rele.contrib.DjangoDBMiddleware',
],}
The code for these example can be found here.
