Connecting data and processes in audit — some considerations about the use of process mining

Today’s potential for capturing and processing data digitally, on an unprecedented scale and at hitherto unattainable speeds, uncovers new opportunities that public auditors can ill afford to neglect. But incorporating analytics into audit is not without challenges. How can auditors move in this new scenario? Gilberto Moggia, responsible for knowledge management projects at the ECA, and Zsolt Varga, data scientist in the ECALab, report about some current uses of process mining in audit and suggest avenues to be explored in the quest for data-led audits.

By Gilberto Moggia and Zsolt Varga, Information, Workplace and Innovation Directorate

There are no magic recipes

Before focusing our attention on the use of process mining in audit, it is worth starting with a helicopter view on the impact that data technologies are having on audit. Whether we like it or not, the transformation by technology is taking place before our very eyes. In fact, the massive volumes of data available and the power of data analytics technologies, now affordable for a growing number of organisations, are fundamentally changing audit and assurance. But the availability of data as a new source of information is not only a blessing for auditors, it is also an arduous operational challenge for audit organisations.

‘Without data you’re just another person with an opinion.’ ‘If you can’t describe what you are doing as a process, you don’t know what you’re doing.’

Two quotes by W. Edwards Deming (1900–1993)

The capacity for auditors to work with great bodies of data is no longer an optional add-on in the future, but has rather become an essential requirement for the profession to remain relevant and continue having an impact. In the not too distant future, ‘increasingly capable systems’ (Susskind &. Susskind, 2017) will take over many of the routine tasks now performed by auditors. To avoid any risk of losing relevance, it is vital for auditors to adopt an experimental mind-set and start developing new (data-led) audit practices.

An increasing number of public audit institutions have started developing their own analytical capabilities, but incorporating data-centric methods into audit is not without its challenges. For sure, the change will not happen overnight and certainly not by a simple implementation of new technology. As Tytti Yli-Viikari, Auditor General of Finland, put it, ‘we should not await some external magical solution to show us the way forward.’ In other terms, there are ‘no magic recipes’ ready for use, but rather several technological options that we should start testing, experimenting with and embedding in our audit practice (Yli-Viikari, 2018).

Process mining is one of the most mature technological options for auditors to extract evidential value from data and data flows, especially (but not only) for financial and compliance audit purposes (Jans et al., 2014; Werner, 2017).

Box 1 — What is process mining?

Process mining is designed to discover, monitor and improve real processes (i.e. not assumed processes) by extracting knowledge from event logs readily available in today’s information systems. Process mining includes automated process discovery (i.e. extracting process models from an event log); conformance checking (i.e. monitoring deviations by comparing model and log); social network/organisational mining; automated construction of simulation models; model extension; model repair; case prediction; and history-based recommendations.

(Gartner, Market Guide for Process Mining, June 2019, Analyst: Marc Kerremans)

See also:

The Process Mining Manifesto, IEEE Task Force on Process Mining. http://www.win.tue.nl/ieeetfpm/downloads/Process Mining Manifesto.pdf

Process Mining: Data Science in Action — Online Course — Instructor Prof Wil van der Aalst: https://www.coursera.org/learn/process-mining

Process Mining http://www.processmining.org/

Process mining for audit was one of the main topics of the ECA 2019 Summer School on digital audit. The presentations by renown experts, including Wil van der Aalst and Miklos A. Vasarhelyi, are available at https://ecademy.eca.europa.eu/course/view.php?id=8

Process mining — a practicable way to go

Process mining is designed to discover, monitor and improve real processes (i.e., not assumed processes) by extracting knowledge from event logs readily available in today’s information systems (van der Aalst, 2016). Despite the omnipresence of such data, it is still uncommon for auditors to use this source of data for a fact-based identification of problems in their auditee’s business processes.

We believe, however, that there are very good reasons for auditors to make extensive use of it:

• First, process mining enables us to carry out what is called ‘conformance checking,’ which makes it possible to compare, in terms of compliance, a process model (expected or approved behaviour) with an event log of the same process. It is used to check if the real execution of a business process, as recorded in the event log, conforms to the model and vice versa. For instance, there may be a process model indicating that a procurement chain requires three or more bids. Analysis of the event log will show whether this rule is followed or not. Another example is the checking of what is known as the ‘four-eyes’ principle, which stipulates that a particular activity should not be executed by only one person. By scanning the event log against a model specifying these requirements, auditors will discover and visualise deviations that deserve closer scrutiny. To sum up, conformance checking can be used by auditors to detect, locate and explain deviations from expected behaviour, and to measure the severity of these deviations.
• Second, process mining can already be applied to entire populations of real-life event logs. Given the time and cost constraints, sampling in audit (testing only a representative sample of items and extrapolating the results to the entire population) has been the accepted practice for several decades until now. Based on the use of event logs to analyse business processes such as payment chains, process mining is a mature digital technology revealing a new audit opportunity that can make the goal of 100 percent testing (using all the data and not just a representative sample of it) affordable and achievable.
• Third, process mining can help auditors focus on performance by auditing the underlying processes in terms of their economy, efficiency and effectiveness. Moreover, it can facilitate the task of auditing the control systems in place in order to assess their adequacy and efficiency. Audit findings and consequent recommendations obtained in this way can not only discover and monitor, but also help improve real processes and thus overall performance.

Box 2 –Linking data and processes — the importance of Process Mining (*)

The interest in data science is rapidly growing. Many consider data science as the profession of the future. […] Data (“big” or “small”) are essential for people and organisations and their importance will only increase. However, it is not sufficient to focus on data storage and data analysis. A data scientist also needs to relate data to operational processes and be able to ask the right questions. This requires an understanding of end-to-end processes. Process mining bridges the gap between traditional model-based process analysis (e.g. simulation and other business process management techniques) and data-centric analysis techniques such as machine learning and data mining.

A quote by Prof. Wil van der Aalst, considered the ‘father’ of process mining, and the author of a seminal book on the subject (see details in Box 1).

To sum up, process mining offers auditors actionable ways to dive into data flows without getting lost. It provides auditors with a tested and mature technology for extracting useful information about business processes, enabling them to navigate throughout the huge quantity of data stored and displayed in financial and management information systems. Process mining for audit was one of the main topics of the ECA 2019 Summer School on digital audit (see also p. 130). Let us now analyse some uses of process mining in audit.

Patterns in data — connecting data and processes

One of the most important questions when auditing an entire population is how the data actually looks. Are there any unusual patterns? What is the proportion of exceptions? The event logs of business logic and database applications contain valuable information about how the business processes are implemented in reality. By examining these logs with ‘big data’ methods one can identify audit-relevant information, such as payments without approval or violation of controls.

Process mining is a very different approach to evidence collection and analysis, as it does not focus on the ‘content,’ e.g. the value of the transactions and its aggregations, but on the path of transactions and the transactional processes themselves. It is thus a powerful tool for tests of controls, such as those for segregation of duties. Process mining is also applied to the population of data rather than to a sample as in traditional auditing procedures. Even though the focus is not on the content of the transactions, the ‘sub-populations of interest’ identified through automated tests of controls can be cross-referenced with data stored in financial systems for tests of details.

Figure 1 — Differences between the ‘de jure’ (left) and ‘de facto’ (right) process of an auditee based on

2 million events

Process mining consists of the analysis and detection of patterns and irregularities in procedures. It is a combination of data analytics and data visualisation as processes are reconstructed and visualised through the analysis of event logs. Event or transaction logs comprise a database that underpins the what, when, how and why of a process. This includes timestamps as well as information related to involved parties, resources and more. Although process mining has been traditionally used for improving the performance of manufacturing and business processes, it also has great potential to be used for compliance and performance audit. The digitalisation of an auditee’s processes and records means that log data is now more accessible, leaving auditors with rather detailed audit trails to examine. An increased amount of event data has created a new kind of audit trail and process mining is an innovative way to explore it.

Some current uses of process mining at the ECA

The ECA’s ECALab started its first process mining experiments in 2018 and we are now ready to carry out two pilot projects relating to the 2019 financial year. Process mining can be used on ‘small’ and ‘big’ datasets alike, ranging from hundreds to millions of items. We have used process mining to model the steps of European public consultations, payment processes within a European agency and the claims administration procedures of an auditee. The visual nature of process analysis makes it very intuitive in respect of spotting deviations, exceptions and bottlenecks.

Figure 2 — Process model of public consultations

Figure 2 shows the different stages of public consultations along the timeline in order of their most frequent occurrence. Deviations from the most frequent process path are shown in red fragmented lines. For example, we can see that nine public consultations did not have a roadmap while three did not complete a synopsis report. Likewise, the shade of blue assigned to a given event is indicative of its frequency, with events missing from some consultation processes presented in a lighter shade of blue.

Figure 3 — Process model with ‘exceptions’ in red

Figure 4 — Process model with median transition times

Figures 3 and 4 depict the payment process of a European agency. It is based on the entire population of their invoice payments for a financial year (4 thousand events and 700 payments). Figure 3 shows the most frequent activities and their interconnections, the blue rectangles indicate the ‘ideal’ process, while the exceptions are highlighted in the red rectangles. By observing the entire population, we can see exactly how many invoices were rejected at what steps in the process, by which staff members. Figure 4 shows the same process, but from a performance point of view. In this figure the wider the red arrow is, the slower the transition between two process steps. We can see that the main bottleneck of this process is the waiting time before the initiating agent or the authorising officer rejects or approves the invoice.

Process mining also allows additional details to be checked, such as segregation of duties, payment deadlines, the completion of the required authorisation steps, etc. A process model based on large amounts of historic data could even be used to make predictions about the possible outcome and duration of a transaction.

Process Mining and the audit standards

The emergence of process mining techniques better tailored to audit and a more widespread adoption of the use of event data for audit purposes have the potential to challenge accepted audit standards and the traditional role of the auditors. If we examine the entire population instead of sampling, more exceptions requiring follow-up actions are likely to be detected, and this could increase the time and effort spent on audit. New methods, such as visual analytics, will be necessary to deal with a greater number of outliers.

Figure 5 — Invoice payments vs commitments in an entire population

Figure 6 — Outliers in the entire population of payments

Visual analytics applications allow the auditors to explore entire populations or sub-populations to identify risks and areas of interest, while traditional reporting formats and spreadsheets listing rows of exceptions are more likely to create potential blind spots for those seeking a full picture of their data. The examples in the figures above show different views of the entire population of invoice payments for an auditee. Figure 5 gives an overall view of the consumption of commitments by framework contract, allowing the auditor to intuitively identify the biggest counterparties and potential cases of overconsumption. Figure 6 shows the distribution of payment amounts to business partners, so that the auditor can easily spot outliers and points of interest for further tests of details. The combination of visual analytics and process mining can also easily identify data integrity issues, such as missing or non-matching identifiers, unusual activities by privileged users and suspiciously short processing times.

Preparing ourselves for the future

As experts confirm, artificial intelligence will certainly play a decisive role in internal and external audit in the not too distant future (Financial Reporting Lab, 2019). This will happen, however, in forms and ways that we cannot precisely predict now. The shift towards data-led audit, however, has started already. We believe that process mining is a smart way for auditors to achieve mastery over the data and turn it into actionable insights.

Box 3 — References

Aalst, Wil van der (2016), Process Mining — data science in action, Springer, 2016, 2nd ed.

Financial Reporting Lab (the Lab) (2019), Artificial Intelligence — How does it measure up?, January 2019 https://www.frc.org.uk/document-library/financial-reporting-lab/2019/ai-and-corporate-reporting.

Gartner (2019), Market Guide for Process Mining, June 2019 (Analyst: Marc Kerremans).

Jans, Mieke, Alles Michael and Vasarhely, Miklos A. (2014), A field study on the use of Process Mining of event logs as an analytical procedure in auditing, The accounting review, Vol. 89, 5 pp. 1751–1773.

Susskind, Richard and Daniel (2017), The Future of the Professions — How Technology Will Transform the Work of Human Experts, Oxford, Oxford University Press, 2017 (1st edition: 2015).

Werner, Michael (2016), Financial process mining — Accounting data structure dependent control flow inference, International Journal of Accounting Information Systems 25 (2017) 57–80.

Yli-Viikari, Tytti (2018), The Future of Audit — No Magic Recipes. INTOSAI — Capacity Building Committee https://www.intosaicbc.org/the-future-of-audit-no-magic-recipes/

This article was first published on the 1/2020 issue of the ECA Journal. The contents of the interviews and the articles are the sole responsibility of the interviewees and authors and do not necessarily reflect the opinion of the European Court of Auditors.