Better process tracking with the workflow engine

Let’s imagine you want to digitize a business. In our case, it’s real estate. To do so, you want to take routine tasks like collecting documents, setting viewing appointments, evaluating the property, etc, and put it inside a web platform to track the progress and make sure the process is moving.

How would you tackle this?

What challenges could you meet on your way?

At Homeday we created Connect — a Rails Monolith app that describes the journey of selling your real estate in some sort of “state machine” with deal statuses that describes where the customer is with the process in the current moment. One of the statuses we have is “Getting Energy Certificate”, which describes the process of getting this document.

Because this process is usually handled by our backoffice team, when we saw the deal status is “Getting EC”, we understood that now they are doing their job and we should receive this document soon. Software engineers were seeing the same status which seemed to be a “backbox” for them and didn’t provide many details.

With time we started to notice some flaws of this “encapsulated” approach:

1. Processes implicit in code make it hard to reason about them (especially when seen by new colleagues).
2. Risk of the business processes diverging over time from the implementation as a result of users finding workarounds.
3. Process optimization also becomes harder due to a lack of systematic insights from hard, measured data.

Photo from design inspiration

To understand something better, you need to take it apart and see how it works. This is exactly what we decided to do with our processes, and for this, we created a team that mapped these into Miro diagrams. It gave us visibility into our processes and also, makes onboarding new employees easier.

But diagrams in Miro are only good as a presentation tool. If you want to make it interactive and see and track the progress in every process you need something different.

BPMN & Camunda

While evaluating different approaches and solutions we quickly discovered that there is already a widely used standard to model business processes called BPMN (Business Process Model and Notation) and a way to run and execute these processes in a tool called Camunda.

From a software engineer's point of view, BPMN can be compared to HTML as it is designed to describe the processes similar to HTML describing a web page. The source code of a BPMN diagram is also an adjusted XML that you can easily read and visualize like HTML in a web browser.

BPMN diagram and its XML code

<?xml version=”1.0" encoding=”UTF-8"?>
 <bpmn:collaboration id=”Collaboration_17o7yv7">
 <bpmn:participant id=”Participant_1nq5l3k” name=”Warehouse” processRef=”ProcessWarehouseAmazon” />
 </bpmn:collaboration>
 <bpmn:process id=”ProcessWarehouseAmazon” isExecutable=”true”>
 <bpmn:serviceTask id=”FetchProduct” name=”Fetch the product” camunda:type=”external” camunda:topic=”Warehouse”>
 <bpmn:incoming>Flow_0cfzktk</bpmn:incoming>
 <bpmn:outgoing>Flow_15ms304</bpmn:outgoing>
 </bpmn:serviceTask>
 <bpmn:startEvent id=”StartEvent_1" name=”Product requested”>
 <bpmn:outgoing>Flow_0cfzktk</bpmn:outgoing>
 </bpmn:startEvent>
 <bpmn:endEvent id=”Event_1d7kyyo” name=”Product fetched”>
 <bpmn:incoming>Flow_15ms304</bpmn:incoming>
 </bpmn:endEvent>
 .....

To do so, we chose Camunda Modeler. Modeler is a BPMN editor which enables to model business processes, make them executable, and deploy them to the Camunda Workflow Engine for running.

Camunda Modeler window

Camunda workflow engine is a standalone Java app the main purpose of which is to execute BPMN models, store information about all the running as well as already completed processes.

Camunda workflow engine running as console Java app

Also, there’s Camunda cockpit which is a web app where we can track the current progress of processes running in the camunda engine and control/change these processes.

Camunda Cockpit, “Invoice Receipt” process

Camunda + Monolith = ❤ ?

Let’s talk about how we can make this workflow engine a part of our monolith app. Camunda provides you with different ways to integrate with your application. If your app is written in Java — you can easily integrate Camunda as a java library and communicate directly with the Camunda engine.

In our case, we have a Ruby/Rails Monolith app and for this Camunda team provides you with API integration.

Let’s go back to the Energy Certificate process we mentioned earlier. For a better visual representation, I simplified the diagram describing the process of getting this document.

Getting Energy Certificate process diagram

On this diagram, you can see start and end events, different types of activities, and XOR gateways for branching. The activity marked with ⚙ symbol like “Determine type of EC” is called service task, something we can perform via the codebase in Connect (fetching the data from the database, calculating certain indexes, etc). And the activity with 👨🏻‍💼 symbol is called user task, something where we need an input from people, ex: sending the letter with custom content or making a call to get certain information.

class DetermineTypeOfEC
  def self.bpmn_perform(external_task)
    deal_summary = load_deal_summary(external_task.businessKey)
    { dataType: { value: deal_summary.energy_certificate_type } }
  end
end

This is how we’d store our code for the “Determine type of EC” external task. It will be later invoked by Sidekiq workers which in turn receive a list of external tasks from Camunda and execute it sequentially.

class AskOwnerToSendMeeterReading < UserTasks
  def config
    {
      taskable_type: :deal,
      camunda_task_process_and_task_name: 'EnergyCertificate::AskOwnerToSendMeeterReading',
      request_for: :backoffice,
      due_at: 24.hours,
      time_critical: true,
      unique: false,
      smart: false
    }.freeze
  end
end

And for the User task, we use the class AskOwnerToSendMeeterReading as a source of instruction on what to put in the ticket of our Connect CRM for backoffice agents. Our UserTasks class knows how to read this config so in your app it might look completely different and maybe even more similar to ExternalTask.

Now a Sidekiq worker will be fetching the list of external tasks from the Camunda workflow engine, executing the corresponding Ruby class, and moving the process forward. The current state of the process we’d be able to find in Camunda cockpit or right in your app via the alternative BPMN visualizer like BPMN.IO

Conclusions

We went through a lot of modeling sessions to model the Energy Certificate process in a way that reflects its real complexity. Now with the workflow engine, our backoffice team received a proper tool to have an overview of their work where they can see which processes are in what stage and help more efficiently if something stuck. As a developer, we now also have a better understanding of the whole process journey which helps with optimizing the processes even if you’re not familiar with all business details. Because BPMN diagrams are easy to read.

As a result, we now have a single source of truth for the processes we migrated to the Camunda workflow engine. We were able to see every process in detail and optimized steps that could be automated or somehow extracted to reusable subprocess. Nevertheless, we faced quite a challenge to not make a Camunda model very granular and complex — you need to see how detailed your application interprets each step and if you can track them via external metrics.

It was also interesting to see how our colleagues found Camunda useful even for describing process tasks without integrating it with code, just for the representation purpose. Our colleague Carsten Wirth used it to describe the shortened function in the article.