PlayBook
Published in

PlayBook

Functional testing microservices using Google Pub/Sub emulator

By Rolandos Koulas - Principal Software Engineer in Test

When building the automated-testing strategy on the levels of the test-pyramid, the need to mock the integration points during functional testing becomes a usual challenge for the teams. The most common cases are REST or GraphQL networks calls with a lot of tooling available, but they are not the only.

While asynchronous communication between services is getting more and more popular, modern platforms tend to adopt such mechanisms for their microservices communication. In such architectures, one service publishes a message and one or more others are waiting to consume it (subscribe).

The two basic roles in Pub/Sub model are:

● Publisher: Publishes messages to topics

● Subscriber: Listens to a topic’s subscriptions and consumes published messages

The basic Google Pub/Sub structure contains:

● Topics: Where messages are published

● Subscriptions: Each topic contains subscriptions, where subscribers attach to, in order to consume the events of the topic

Testing an architecture like this, requires all of the services to communicate with an actual Google Pub/Sub instance, which most times is not already in place and automated functional testing is not easy to be applied in isolation.

Google gives a solution to this by providing the ability to use its Google Pub/Sub emulator and reach a good level of confidence before moving to end-to-end testing.

Let’s see how it works

To start the emulator locally, Google Cloud SDK should be installed as a prerequisite (https://cloud.google.com/sdk/docs/install)

Then the emulator component installation follows

The emulator requires PUBSUB_PROJECT_ID environment variable to be set to indicate the project the emulator is going to connect to, plus PUBSUB_EMULATOR_HOST variable pointing to the host and the port the emulator runs. The latter is also required to instruct the application under test to connect in emulator mode. Otherwise it is considered as a real Google Pub/Sub, requiring authentication with credentials.

Everything is installed let’s start the emulator:

You see, it is up & running waiting for our services to integrate with it.

Now let’s write some code to play with it

Google and other 3rd parties provides libraries that can be used to interact with the emulator (create topics and subscriptions, publish and consume messages etc).

We are gonna use Google’s python client to create a basic script that communicates with our local emulator. Many snippers are available as an entry point.

https://github.com/googleapis/python-pubsub/tree/main/samples/snippets

Creating topics and subscriptions:

Publishing and receiving messages:

Moving a bit fast-forward, we can have something like this as our Python client (e.g.: pubsub.py):

Of course, this can be extended to support dead-letter queues and many other Google Pub/Sub features.

With some basic JSON parsing given a custom Pub/Sub topology structure it can be really easy to interact with the demo emulator

All these however, needs to be applied on each engineer’s local environment and on CI when automated tests are about to run. Let’s make it a bit easier!

What’s better than utilizing Docker’s power to make our tooling environment agnostic and avoid phrases like “On my machine it works, I do not care about yours”?

Firstly, we will need a really basic script (feel free to make yours more sophisticated). Let’s call it “setup.sh”.

Then comes the Dockerfile to bundle them all together and make it a single command thing.

Remember, we need PUBSUB_EMULATOR_HOST to be set for our python client to avoid Google Cloud authentication.

Building and running the image…

And voila!!! The emulator is up, along with a topic and a subscription waiting for our messages!

Now everything is ready and automated, how can it be integrated with the functional tests of the publisher or subscriber services?

The emulator image can start along with the service-under-test, before the automated test’s execution, using any preferred way (docker commands, docker-compose, Java TestContainers etc).

Then, based on the Test Automation Framework’s language, a Google Pub/Sub client can be used to either update the topology (create extra topic/subscriptions) or publish/receive messages.

In worst case scenarios, where such a client is not an option, docker commands can be executed directly from test’s code to run the built-in python scripts presented before inside the container.

It may be easier to present it with examples…

Testing the publisher

Given a publisher service that exposes a REST endpoint and publishes events that are generated upon POST requests.

An integration test could be like:

● Start the emulator with a topic/subscription topology (Suppose it is already done like explained above)

● Execute a REST call to the service to generate and publish messages and assert the response status code.

● Through test code (using a Google Pub/Sub client), verify the expected messages have reached the emulator

Using NodeJS, Jest, supertest and the Javascript’s Google Pub/Sub client library (https://www.npmjs.com/package/@google-cloud/pubsub) the test could be like:

Testing the subscriber

Given a subscriber service that consumes Google Pub/Sub events, processes and stores them in a database

An integration test could be like:

● Start the emulator with a topic/subscription topology

● Through test code (using a Google Pub/Sub client) publish messages to the emulator

● Verify the database is updated with the expected entries as soon as the messages are consumed by the service

For JVM fans let’s see the opposite using SpringBootTest and Spock framework

Our client for this example will be the one from Spring Cloud (https://docs.spring.io/spring-cloud-gcp/docs/1.0.0.M3/reference/html/_spring_cloud_gcp_for_pub_sub.html)

This way each service can be tested independently, without a real Google Pub/Sub deployment, in any local environment and CI.

The same emulator instance can be also used with both the services up & running for testing end-to-end.

Mocking and tooling on the integration points, always gives the teams a greater level of confidence to produce bug-free code and write automated tests, avoiding any bottlenecks from the infrastructure or teams that implement the other end, especially on microservices architectures.

If you like tackling with tech challenges and want to join a tech community that promotes best practices, creativity and work with motivated colleagues, have a look on our current openings and apply. Join our team of ambitious and talented tech professionals and give back to the community!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store