Pub/Sub messaging within Billie
Billie soon turns 3 years old from an IT perspective: since the very first repository has been created, and the first ship has set sail. And 3 years ago there were not much communication means in our monolithic application: some HTTP API endpoints, some RabbitMq shenanigans.
Coming back to nowadays (article publication date-time), Billie became a set of microservices. And how would you imagine them — microservices — talk to each other? What about a publish/subscribe pattern? And, last but not least, how could all that be practically implemented with Symfony?
Let’s examine those questions. Under the cut, people!
Monolithic application
Before we tap the pedal to the metal, let’s time travel 3 years in the past again. Back then, we quickly came up with an idea that a solid backend would do best to shave some time off. We were only 3 software & one system engineer and a CTO to create an API, cloud infrastructure and an SPA to decouple responsibilities, agree on contracts and happily work in parallel to make candy.
And as it was expected, a monolithic application started to develop all problems possible. The bootstrap file became pretty much massive, which is still ok when it comes to FPM but think of all PHP consumer processes. Memory usage — that was massive.
Anyways, such success/tragedy didn’t necessarily mean it’s the end of greenfield software for us. In fact, it was a great push & motivator to make things better, without blocking the business processes.
Segregation of responsibilities and technologies
/me thumb clicks and we are back to our present time! While PHP is a primary programming language in our company, some tasks are better to be done with e.g. Java or R, etc.
Cutting our monolithic codebase down into independent services pursued other goals as well. For example:
- Developers can work independently on their own codebases;
- Microservices are faster to deploy;
- Forces you to be idempotent, generally means better code quality (hella important for fin-tech);
- You save resources when you need to scale your infrastructure: add only services that are required for the spike or general load.
We can talk a lot about service-oriented architecture, it deserves a separate article here. But let’s focus on the cornerstone. A set of microservices is absolutely meaningless if they can’t communicate, delivering what our business has to do.
How would microservices communicate with each other?
The first idea that comes to mind would be the usage of HTTP calls (which are basically RPC calls in our case as we don’t follow RESTful) with application/json payload or query parameters.
That brings us to a question though: what if a 2nd party service is not available (I mean we could use Guzzle’s retry middleware e.g., but what if a service is still down after a particular threshold)? Or your upstream fails with 429 HTTP codes?
What if we need to call two services with the same payload? How do we discover a service’s endpoint?
Not that these problems are unsolvable, but clearly it requires some decisive actions to be taken. Why can’t we delegate all those responsibilities to an AMQP message broker?
Our Pub/Sub architecture in RabbitMQ broker
When we do an RPC call — that’s nothing else, but a command to the service. When a service replies—that’s basically an event, something that has happened and can’t be changed, it has its timestamp.
Similar pattern to pub/sub would be Observer. However, in that case your event within a subject eventually triggers a command to tell your observers what happened. Recruiter knows their employees.
In publish/subscribe architecture there are no commands, only events. That means, we just need to spread news across our services and… nothing else.
Events here make your life simple and complicated at the same time. Simple, because in your app you don’t care what happens later, who’s subscribed to the topic (e.g. invoice approved). Complicated, because you don’t know what happens later (yet you can track it down by request id in the logs).
To implement the pattern, we take advantage of the AMQP model. Whenever something happens in a microservice, a message goes straight to the front exchange of fanout type. That’s it. The rest of the job is done for you by a broker, ensuring that the message gets delivered to the services that are interested in such an event.
The following requirements must be met:
- Topic name must be a domain, e.g.
InvoiceorTransaction; - Destination queue must be bound to the topic exchange;
- A queue itself must be
durable. We don’t want to lose any messages; - In the future, it might be
Invoice→InvoiceApproved→queues. It's more semantic and clearer to understand topology, but even if we don't do it now, it doesn't affect queues (no need to disturb teams), so that could be done within the scope of Technical Friday;
- Each queue must follow a naming convention, such as
app_name.domain.event. E.g.payments.invoice.invoice_approved; catching such an event will initiate a payout procedure, which may cause another event that also heads out to the fanout exchange.
It’s important that within a microservice, 1 queue can have only one port. That port, however, may have as many consumers as possible. This also leads to the conclusion: if a system needs to perform 2 actions on 1 event, your microservice must do it internally (pipelines or events e.g.).
At this point, we have producers, subscribers. They produce and consume payload. How do they know what kind of a payload is there? We need to define a contract! That might be plain text, JSON or…
Serializing data with Protobuf
As we know, an event must be immutable: data—producer emitted—must reach subscriber in exactly the same state. Both systems, independent of technology, must be able to deserialize a payload and represent it as an object, schema of which is known to each system whether it’s written in PHP or Java.
With protobuf we define our payloads in proto3 syntax. So basically once you have need to communicate between PHP and Java, you just generate classes for the corresponding language and import those to your project.
Compiled protobufs offer you serialization whether in JSON or binary out of the box. Finally, having our data, it’s right about time to wrap the payload up in RabbitMQ message, accompanying it with a few headers:
X-Request-IDTo trace request flow-through systems. Read more here “No log shall be lost”;X-Message-TypeTo tell subscribers what payload is in there.
You can read an awesome article regarding protobuf usage in one of Billie’s R services.
Practical example with Symfony4
Since I’ve mentioned PHP, why don’t we take a look at a practical example of pub/sub with Symfony4? There we use the messenger component in conjunction with Enqueue.
To get started, one just requires a composer package from our private repository composer require ozean12/amqp-pack. It will pull all dependencies, including the protobuf generated PHP classes. Once it’s there, you’ll witness a stock config/packages/messenger.yaml configuration file.
Each microservice has to declare transports: whether it is for publishing or consuming. Sample configuration might be like this
It’s possible to publish/consume messages via transport. It doesn’t look developer-friendly, however, you can clearly see a pattern there. Something for future improvements.
Before we create our first producer or consumer, we need a transport factory.
It is a stock configuration with one exception, the debug flag (3rd argument) is set to false. Debugging doesn’t do anything useful, except for creation of queues and exchanges on the fly. Unfortunately, it has no idea about fanout exchange, so any attempt to publish a message will always crash as the bundle expects topic exchange. Then again, why would you like to have queues creation in runtime?
We are all set!
Publishing a message is trivial
So is subscribing & consuming
Now just run your consumer bin/console messenger:consume invoice.invoice_approved -vvv and enjoy happy communication!
Circuit breaker for PHP consumers
For PHP it’s natural process to get executed and quit the process with exit code zero. Of course, you could make use of process control, but in our case consumers terminate after they ack a bunch of messages. Unhandled exceptions cause a process to quit with non-zero exit code.
At some point, you may face the situation when the process always fails and the message gets redelivered. First of all, your logging facilities gonna be cluttered and you may do unnecessary calls to 3rd parties. It’s handy to have a circuit breaker.
The basic idea behind the circuit breaker is very simple. You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all. Usually you’ll also want some kind of monitor alert if the circuit breaker trips.
In our case, we just track exit codes. If it’s not zero — something went wrong. Having such a trivial tool helps a lot to cut costs.
Conclusion
Pub/Sub helps a lot with service discovery and reliability, scaling. It makes your code idempotent and robust. However it can’t replace remote procedure calls, we still have old good HTTP APIs, which someday gotta be replaced with gRPC. There’s a whole new world of communication: transactions, streams.
Thank you for your attention! Have any questions? A comment section is below!
Best regards, Nikolai.
