Auditing COVID‑19 recovery and resilience plans: some initial experiences and findings relating to Flanders
Member States’ National Recovery and Resilience Plans (NRRPs) have now been designed, submitted and approved, and the first steps have been taken to implement them. This has extended the domain public auditors have to cover, not only in terms of size but also in terms of the new disbursement criteria for EU funds, under which achieving milestones and targets is fundamental. This has not been left solely to their own initiative: striving for maximum transparency under such a new instrument, the Flemish parliament has requested the Belgian Court of Audit to regularly assess the quality of reporting on the implementation of the Flemish Resilience Plan. The Plan includes some projects that also form part of the Belgian NRRP. Vital Put is a councillor and a member of the Dutch‑speaking chamber of the Belgian Court of Audit, while Bart Andriessens is a senior auditor supervisor in the same institution. Below, they explain the first auditing initiatives taken by their Court of Audit in terms to help maximise accountability under the Flemish Resilience Plan, describing both the audit approach used and initial discussions following public scrutiny of the audit results.
By Bart Andriessens and Vital Put, Belgian Court of Audit
Flemish Resilience Plan — more than just part of the National Plan
The COVID‑19 pandemic has had a very profound impact on the economy and society in general. The Flemish government rapidly decided to launch a major plan for recovery and resilience. Two advisory committees were created, both including experts and government officials: one for economic, and one for societal recovery and resilience. In addition, different stakeholders were consulted. Based on this input and on the Flemish coalition agreement, the Flemish Resilience Plan (‘Vlaamse Veerkracht’), comprising 180 projects, was drawn up. Of those 180 projects, 158 will be financed through an extra budget of €4.3 billion euros. These 158 projects include 55 that will receive grants from the European Recovery and Resilience Facility (RRF); the other 22 projects will be financed through the standard annual budget.
Apart from this Flemish initiative, the Belgian government has submitted a National Recovery and Resilience Plan including 140 federal and regional projects to the European Commission. Belgium has a federal state structure, Flanders being one of the regional entities (1). As Belgium has been promised €5.9 billion in grants from the EU, negotiations have been held between the federal and regional entities to divide the total grant. Flanders has been allocated €2.255 billion.
It is therefore important to emphasise that, as shown in Figure 1 below, the Flemish Plan goes well beyond the projects included in the National Plan (only 55 of its 180 projects are RRF projects), but that those projects in turn only form part of the Belgian Recovery and Resilience Plan, which has a total of 140 projects.
Figure 1 — Two recovery plans with overlapping projects
The Flemish government decided to aim for maximum transparency in the Resilience Plan and its execution, committing itself to three progress reports per year, made available on the government’s website. These reports provide an insight into the Plan’s overall implementation as well as a more detailed (2) overview of each project.
Two audits on recovery and resilience, but with a different scope
The Flemish parliament asked the Court of Audit (3) to evaluate the quality of the performance and financial information in these progress reports annually, thus resulting in a recurrent, annual audit. This audit examines overall progress in implementing the Flemish Resilience Plan and the specific progress made in the 74 projects with a budget of €20 million or more.
In order to adequately audit these progress reports, we needed to gain better insight into the Plan’s governance, and monitoring systems and the project management for individual projects. Indeed, without good overall goals, project goals, and project milestones as well as good insight into all these elements, it is difficult to evaluate the quality of the reporting. To this end, we also performed a one‑off audit. Whereas the recurrent annual audit concerns the quality of project progress reporting (indicators relating to project performance and financial progress), the one‑off audit dealt with the quality of the Flemish Resilience Plan, the monitoring system, and project management. Both audits fall within the scope of our audit work on COVID‑19 (4).
When planning both audits, we frequently consulted other European Supreme Audit Institutions (SAIs), including the ECA, about the audit approach. Such consultation took place in the context of RRF‑related SAI cooperation at auditor/working‑group level. This is an informal and light form of cooperation that offers the added value of international cooperation (such as peer learning) without the cost and administrative burden of more formal types of cooperation. These exchanges were particularly helpful in planning the audit work.
Main audit findings relate to framework, monitoring and project management
Our main findings from both audits are:
Policy framework and governance
The Flemish Resilience Plan sets out seven objectives, but we found them to be vaguely worded, while the underlying analysis of the economic and societal challenges was not explicit. The overarching vision can only be derived from reading the Plan and the Flemish Coalition Agreement, including the advice on which the plan is based, together. The choice to include many projects, rather than taking a more focused approach, was not substantiated. The government only stated that this approach allowed it to cover all aspects of the economy and society.
It is unclear how the budget was distributed among the seven objectives and their projects. While a task force supervises the Plan’s implementation, we observed little steering of the Plan as a whole, as steering is mainly done at project level. We found that the diversity and the high number of projects complicate overall steering. Management of risks to the Plan’s implementation (e.g. developments in society, financial risks) is also generally lacking.
The numerous administrations involved are responsible for monitoring and reporting on their projects. To this end, the Chancellery has developed a high‑performance online reporting tool, which functions as a source database for all project information. The Chancellery also monitors the quality of the data entered. Quarterly progress reports are produced in several stages and provide an overview of the Plan’s overall implementation and the progress of each project.
Project management and progress information
We assessed the selected 74 projects on a number of key elements for good project management and high‑quality reporting to Parliament. Figure 2 below shows the conclusion for each of the key elements for project management. Elements shown in green are clearly defined, while elements in orange and red could be improved.
Figure 2 — Main findings by project stage
For most projects, the scope, target group, responsibilities and desired output were clearly defined. In terms of objectives, output indicators, milestones, intermediate targets, project risks, planning, action plans, and making explicit links with other projects, there was still room for improvement. The Court of Audit established, however, that the Flemish government is gradually trying to remedy these shortcomings and has already taken steps in the right direction. We concluded that the facts and (financial) figures were rather qualitative in nature, but that the reported status of projects was more difficult to evaluate due to persistent shortcomings in project management.
Reports were well discussed, before and after publication
Having completed the audit work, we sent our two draft reports to the government and all audited entities for their comments. For each of the reports, we had a video meeting with representatives of the audited entities to facilitate the adversarial procedure. We asked for a coordinated reply from the auditees on the facts and figures, and an overall high‑level government reply in writing. Despite the numerous entities involved in the audit, the Flemish government managed to provide a coordinated reply to both audit reports within a short time. The replies were quite positive, although the government also made some critical comments, e.g. that we did not sufficiently take into account the extraordinary circumstances in which the Plan had been drawn up and implemented, such as the need for quick action, uncertainty about the pandemic’s evolution, and the high workload in the various administrations. The replies and comments received led to some changes in the final reports.
The reports were published on our website on 17 March 2022. Media coverage on 17–18 March was rather limited and mostly based on a press release issued by the Belgian News Agency Belga. It focused on the budgetary aspects, namely that the Plan could cost more than originally estimated.
On 22 March, the Flemish Prime Minister and the Minister of Finances presented their latest progress report to the Flemish Parliament. They also announced some new aspects, namely:
- all reporting data will be made available as open data on the government’s website;
- further investments will be made to improve the quality of the indicators;
- a clearer narrative will make the connection between the projects more explicit.
We then presented the results of both audits and answered some technical questions from Members of Parliament, who addressed most of their more general questions to the government. It was an interesting and stimulating parliamentary debate, during which the added value of the audit work for better governance and monitoring of the Plan was recognised (see below). The day following the debate in Parliament, there was another newspaper article that also focused on the progress of the projects and mainly expressed the position of the Green party.
Audit criteria and audit methods are most relevant
Scoring the 74 projects based on the various criteria proved to be a methodological challenge. Firstly, it required a substantive knowledge of the policy domain in which the projects are carried out. Moreover, assessing the quality of objectives, milestones, indicators etc. is subjective to a certain extent. We asked our auditors who monitor each specific policy area to provide a score for the projects in their policy domain. In order to ensure uniformity, we produced guidelines including all questions to be answered. Based on these guidelines, the audit team then reviewed the scores and assessed the answers again, adjusting them where appropriate.
At this early stage in the Plan’s implementation, our audits have not yet focused on the raison d’être of the Recovery and Resilience Plan — the achievement of its goals. They have largely been systemic audits, in which we paid attention to conditions that are important for achieving these goals, such as clear objectives, evidence‑based policy, and good project management. The assumption behind this is that a rational, result‑oriented approach to policymaking and management contributes to the effectiveness of the policy, as well as providing better accountability to Parliament.
One might even wonder whether a SAI can assess the effectiveness of a recovery and resilience plan as a whole at all. It is of course feasible to assess the effectiveness of certain projects or clusters of projects. However, in view of their desired global impact on the economy and society, these plans should also be assessed at macro level. This seems better suited to the expertise of institutions such as central banks, planning offices or statistical institutes, which have macro‑level data and models at their disposal. As SAIs, we focus more on the micro and meso level (organisations, programmes, etc.).
Political uptake of audit findings
Although it is too early to comment on the impact of the audit reports, can we already draw some conclusions on the way they were received, and on how our audit findings were taken up at political level (see Figure 3)?
Figure 3 — Audit logic and political logic
During the parliamentary debate following the audit presentation, the members of parliament expressed their appreciation for the government’s comprehensive and transparent reporting and for our thorough audit reports. They used the audit reports to ask the government a number of questions, such as:
- Have there been any delays?
- What would happen if the financing were insufficient?
- Will the crisis in Ukraine have an impact on the recovery and resilience policy?
- Can the indicators be better defined?
- Does the Flemish government have sufficient capacity to carry out all projects on time?
The Flemish government gave a clear answer to a number of concerns raised: even if European financing were to decrease, the plan would still be fully implemented and debt financing would be used if necessary. Extra resources required for the crisis in Ukraine will not be deducted from the resources for the Plan. MPs also delved deeper into the political choices behind the Plan, with questions such as:
- Why did the plan not focus more on climate policy?
- Which political considerations played a role in the choice of projects?
- Were the projects selected the ones with the most impact?
The audits were also partly used to reinforce political positions: one of the opposition parties that had previously criticised the recovery and resilience policy had its position confirmed by the Court of Audit’s findings. The technical language of the audit reports was reframed into more colourful wording, e.g. ‘the choice to include many projects’ became ‘political and financial confetti’ or ‘the choice to strengthen existing policy with additional resources’ was reformulated as ‘to compensate for the inadequate financing of the coalition agreement’. On the other hand, the tone of the coalition parties was much softer. The government emphasised several times that it is a learning organisation and that our audits provide an incentive to do even better. At the end of the debate, the Flemish government once again committed itself to thoroughly studying our recommendations and to implementing them or to justifying why it would not do so, in line with the ‘comply or explain’ principle.
As described above, our performance audits often start from a result‑oriented framework on how government should work:
- a policy should be based on clear policy objectives;
- it should be substantiated by a thorough ex ante analysis of the problem;
- it should be evaluated ex post and adjusted if necessary;
- there should be transparent accountability for the implementation, costs and results of the policy pursued;
- projects must be managed in accordance with best practice.
This audit logic is often at odds with political logic:
- During the adversarial meeting, the Flemish government pointed out that the Plan had been created and was being implemented under extraordinary circumstances. This concern was reiterated in the parliamentary debate. Indeed, politicians often have to work with incomplete and uncertain information (bounded rationality) (5) and it is often not completely clear what works best in dealing with ‘wicked problems’ (6).
- The dilemma of transparency was also discussed during the parliamentary debate: on the one hand, transparent reporting by the government is very important for parliamentary scrutiny, but on the other, it also makes the government more vulnerable. Indeed, clear objectives can be a threat to politicians because the opposition and the media will judge them on these objectives.
- Policymakers not only want to be effective, but also expressive (i.e. they want to be able to show visible results in the short term), hence the inclusion of smaller, ‘quick win’ projects.
This report is the start of a journey for the Belgian Court of Audit, as the Flemish parliament has asked us to report annually on the progress of the Plan’s implementation. This also implies that we will have a long‑term relationship with both the government and administrations on this subject, which will also allow us to increase the impact of our work. We are confident that our reports will not end up being a message in a bottle.
(1) The Belgian Court of Audit is competent for both federal and regional levels.
(2) The last progress report (December 2021) was 703 pages long.
(3) Parliaments can ask the Belgian Court of Audit to do an audit, but most audits are performed on our own initiative and are selected on the basis of a number of criteria such as risks, importance for society, and added value.
(4) Other audits on COVID‑19: Support Measures for Businesses and Individuals during Coronavirus (October 2021) — this was a ‘transversal’ or whole‑government audit, examining federal and regional levels of government and paying attention to coordination between these levels of government; and COVID‑19 and continuity of the federal public service — Implementation of staff measures (January 2022).
(5) The fact that rationality is limited when making decisions, due to e.g. the limited time available to make a decision or a lack of information.
(6) ‘Wicked problems’ are inherently resistant to a clear and agreed solution: difficult to clearly define; no clear and correct solution; many interdependencies and multi‑causal aspects; problems are socially complex with many stakeholders; proposed measures may have unforeseen effects. See Australian Public Service Commission, Tackling Wicked Problems: A Public Policy Perspective, Canberra, 2007.
This article was first published on the 1/2022 issue of the ECA Journal. The contents of the interviews and the articles are the sole responsibility of the interviewees and authors and do not necessarily reflect the opinion of the European Court of Auditors.