How to split up a Django monolith without using microservices
Let’s queue(bie) our business logic
TL;DR
Facing too much complexity in your Django application as a developer or system architect with linear, imperative programming? Try out django-queuebie, a synchronous message queue working with commands and events.
Why queuebie?
A while back I was implementing a piece of business logic where lots of stuff was happening in one place. My regular approach, writing service classes, created entangled and hard-to-read code. There were just too many things that had to be implemented in “one place”.
Since I’ve been interested in event-driven architecture and domain-driven design, I looked into these patterns but concluded that they add too much overhead. In the end, I wasn’t implementing a corporate-size application. Nevertheless, splitting up the business logic written in imperative programming into events seemed like a valid option.
Therefore, I’ve decided to go with “Commands” and “Events”, where commands are allowed to persist data in the database while events listen to commands that have happened and trigger new commands. Having in mind that events are supposed to “only” trigger new commands, they are well-shaped to cross domain boundaries — in the Django world, internal Django apps. This decouples the code and enables the developer to avoid big-ball-of-mud’ding business logic all over your application.
Note, that I’ve decided to implement a synchronous message queue — in contrast to most async solutions out there. The reason is simple: Async adds a lot of complexity to any application which didn’t seem necessary for this approach.
How does a synchronous queue work?
The queuebie message queue takes n messages and will process them. Processing means that a message handler, a function, can be registered to listen to one or more messages. Once this message is in the queue, the handler code will be executed. Handlers can return 0-n messages, which will then be enqueued. Since the entire process is synchronous, the queue will block the thread until every message was processed.
This queue can process two types of messages. “Commands” and “events”. Both are dataclasses with typed attributes (pic. C1) and inheriting from “Command” resp. “Event”. These attributes act as the context of this message. It’s easy to distinguish between commands and events since commands are always written in present tense while events are written in past tense.
Our message queue is always initialised from an external-facing endpoint. This can be a Django view or an API REST endpoint (pic C2).
There, you’ll add a command which contains the first atomic step of your business logic. The whole idea is to split up the block of business logic into small and independent messages which can be processed individually.
A message will be picked up by every handler — a Python function taking the message context as the only input parameter — that has been registered to listen to this message. The registration works like registering a Celery task — via a decorator.
Since the handlers are usually quite small and all input is well-defined, it’s easy and straight-forward to write unit-tests for them. Furthermore, having an explicit and typed context leads to good IDE support since there is no magic happening at any point.
Every command handler will return 0-n events, and every event handler 0-n commands. Every message being returned from a handler will be enqueued and then processed. Once no new messages are enqueued, the queue stops.
To ensure data consistency, the whole event loop runs in a single database transaction. If something breaks, everything that has been persisted will be reverted.
Let’s talk code
Great, we’ve established the general pattern. I’ve created a demo repository at GitHub called queuebie-test. This implements a simple web shop which has two “buy” buttons in the shopping cart. One uses imperative programming utilising a service class, the second one uses the novel “django-queuebie” package.
Imperative programming
Here is the code, written in one service class. To be able to put everything on one page, we’ve refrained from encapsulating logic in services or helpers.
The code is quite straight forward. The order contains products for which we’ll calculate the price. Then we’ll create a payment transaction with some magic since this might fail for multiple reasons. Next, we separate between bulky (like a refrigerator) and regular products and create delivery notes accordingly.
Message-based implementation
In the next step, we’ll try to implement the exact same behaviour as seen in C4 — but using messages to split up the code.
When using Queuebie, we don’t have one central place where our code lives. Instead of calling a Python class containing our business logic, we inject our first command into the message queue, like shown in snippet C2. The command is filled with all required context and then being processed by the queue.
Once this command is finished, an event handler is listening to the event “OrderCreated” (pic. C6)
But what if we additionally want to log to our logging system that the order has been created? Easy, just register a new event handler (pic. C7)
Working with Queuebie
We’ve established that the queue will run synchronously once initialised and will not return any data. Implementing a create endpoint is a common use-case. In a classical Django application, you’d call Model.objects.create() and you can then return the newly created object ID. When you want to achieve the same with Queuebie, you must take one of the well-known workarounds for distributed systems, like marking every request with a UUID and query for a new object with that UUID once the queue has been processed.
In the previous paragraphs we’ve shown how Queuebie might be superior to a classical service pattern in certain cases. Note that it’s perfectly fine and reasonable to combine these approaches. Think of a handler which contains a complex piece of business logic. No reason not to wrap this in a service class.
We’ve talked about decoupling the domains and avoid creating a big ball of mud. People coming from a truly decoupled architecture looking at our demo repo might point out that we still use Django’s foreign keys which enable the developer to cross domain boundaries “silently”. That is a valid remark but keep in mind that the whole approach of queuebie was to find a way to avoid the mudball without buying into all the overhead of a decoupled system.
If you’re feeling bold, I’ve played around with the thought of making the context completely independent of the Django ORM. This would mean that we never use model instances in the context but either just object IDs or custom data objects. IDs would probably lead to a lot of extra queries since every handler would have to fetch the data that it needs. Custom data objects would contain exactly the data that is required to process the given and any following events. It will lead to some implementation overhead but this might be a plus.
Automated testing is paramount in software engineering. Since we add a layer of abstraction, we must ensure programmatically that all those messages are fired and listened to. Therefore, having integration tests on top of the handler unittests is a key success factor.
But why not just use Django signals?
Some dear readers might ask themselves why we bother creating this pattern which will be a hurdle for onboarding new developers to a project. So why not just go with Django signals, a well-known and battle-tested feature of Django?
The answer is quite simple. Signals are bound to a model which we explicitly don’t want. The message should reflect the business concern and not the persistence layer.
Secondly, they don’t have an explicit and customisable context. They accept a model instance and that’s it.
Lastly, I feel that separating commands and events adds a great benefit to your system and the way this forces the developers to think about the business logic. That’s not possible with Django signals, too.
Benefits
To sum up this article: Why should you try out Queuebie?
- Separating your business logic into small and understandable chunks.
- A diagram of your commands and events is understandable for non-tech people and it can be used to discuss business concerns with the stakeholders / product owner.
- Using reusable messages leads to reuseable code. Look at pic. C5 how often the “CreateLogEntry” command is triggered.
- We have a simple pattern at hand to keep domains separated without buying into full DDD or losing all the advantages of the Django framework.
Critical review
Given all the benefits and reasoning, I have do disclose that this package, though its 100% code coverage, has never been tried out in production yet (May 2025). So, there might be some downsides I’m not (yet) aware of. This is indicated by the 0.x version on PyPI.
In a nutshell, I’m not reinventing the wheel here. I’ve taken some ideas from event-driven architecture and tried to marry it with the Django framework without losing most of its benefits.
If you’ve manged to reach this point and ask yourself “Do I need Queuebie?”, I’d answer with the following checklist:
- Will your project stay at an appropriate size (meaning not a giant corporate application)?
- Do you have a lot of deep (multi-step) business logic?
If you answer these questions with a “yes”, I’d suggest you’ll have a look. If you’ve never worked with events and therefore never thought of your business logic as a chain of real-world “things” that happen in some kind of order, you might face there your first hurdle. But trust me, even if you don’t use this package in the end, it will be a benefit for you and the project to be aware of this pattern.
Thanks for reading and please feel free to drop a comment or two. I’m eager to get some feedback!
