Unit 7: Measuring results
Summary
Policy makers, academics and practitioners share the view that public procurement can be a strategic tool and pursue a mixed array of relevant social and environmental objectives. This unit will introduce you to the conceptual differences between short-term outputs, mid-term outcomes and long-term impact.
Looking back to understand what has and has not worked is fundamental to improving decision-making and policy design for future projects. To draw valid conclusions it is necessary to have comparable results against which to compare our interpretations of events. This unit provides frameworks and sample indicators to measure results in two intertwined dimensions: 1) value created by implemented solutions and 2) value created by the procurement process.
Why it is important to measure results
No matter your approach to procurement, once a solution is implemented it is essential to analyze results in order to enhance processes, improve decision-making and inform policy.
How can you ensure that you are going in the right direction? Conducting an evaluative analysis will allow you to understand what has worked and identify what can be improved. This will trigger a shift from a linear to a cyclical approach by treating procurement as “an ongoing process” rather than a series of isolated “one offs”.
Measuring your results will help you to evaluate two correlated dimensions of your procurement:
- First dimension value: created by implemented solutions. Have the procured goods or services addressed the need you originally identified? Has it directly created value for the target group (beneficiaries or end-users) and indirectly for the rest of citizens (local community)?
- Second dimension value: created by the procurement process. Did your procurement process find solutions you may not otherwise have discovered? Has it involved bidders who may not normally have been eligible or aware of the opportunity? Has your procurement process become more effective?
Some results, of both first and second dimension value, are directly related to your specific interventions, while others might be influenced by external factors. In other words, some results may be more clearly attributable to your work than others. What is the difference between outputs, outcomes and impact? What follows is a simple definition of those terms and a set of examples for each of the two dimensions of your procurement.
Examples of first and second dimension outputs, outcomes and impact
First dimension value: created by implemented solutions
Technological tools to monitor senior citizens’ nutritional intake
- Output: Provision of a user-friendly application to monitor daily nutritional intake
- Outcome: Decreased number of doctor’s visits / reduced time of care-takers due to improvements in senior citizen nutrition
- Impact: Increased and sustainable health benefits of senior citizens while reducing healthcare costs
Second dimension: value created by the procurement process
Making the procurement process more open and engaging
- Output: sourcing document describes the problem instead of specifying the solution — it is clear, informative and shorter than traditional sourcing documents
- Outcome: increased number of bids from SMEs/startups, new approaches and disruptive ideas found
- Impact: sustainable cost reduction, increase in quality and economic development through local job creation
Measuring value created by implemented solutions
How has the implemented solution affected citizens’ lives? What is their perception of the effectiveness of public spending? Do they consider the problem solved?
Remember that your KPIs should be measurable and related to the targeted service or solution. They should fall under: direct/immediate outputs, mid-term outcomes and long-term impacts. As explained in Unit 2, Key Performance Indicators (KPIs) should be established at the problem definition stage. This will allow you to define what success looks like from the beginning and collect the relevant data in a systematic way.
When implementing a solution it is important to understand the value created not only for direct beneficiaries, but also for the wider community, the economy, the environment, and the overall resilience of your city. As illustrated in Stockholm’s digital assistance case study, where a solution developed in Stockholm to enable visually impaired citizens to gain employment, solutions can have indirect benefits for the local government, local economy, and even the global market. The City of Stockholm collected relevant data in order to understand the initiative’s additional benefits. Certain interventions, however, may require a deeper analysis over a specific period through an impact evaluation.
Impact evaluation
Once the procurement cycle is closed and the solution is implemented, it is important to find a specialized institution to conduct an impact evaluation. The learning derived from impact evaluations helps your City be managed by evidence, continually improving and allocating resources better.
Furthermore, many cities around the world are tackling similar global issues like poverty and climate change. Unless cities create solid and trusted impact evaluations, we will not have the evidence to scale solutions that work.
An impact evaluation is an assessment of how the intervention affects outcomes, whether those effects are intended or unintended. A proper impact analysis also requires a counterfactual statement of what those outcomes would have been in the absence of the intervention. It is important to distinguish between monitoring outcomes, which is a description of the factual, and using counterfactual statements to attribute observed outcomes to the intervention.
The impact measurement consists of quantifying the impact variables for the treatment and control (or comparison) groups, then analyzing the differences between them.
The following table outlines the questions that an impact assessment should follow, the objectives it should pursue and the methodologies that can be used. Also see the subsection ‘evaluation framework’ in Unit 5 for more information on deciding criteria.
Measuring value created by the procurement process
A sound procurement system is built on a set of processes that maximize the likelihood of achieving the City’s objectives while minimizing the use of resources. The following framework provides examples of indicators related to strategic leadership, objectives, processes and relationships with stakeholders, which may contribute to an overall improvement of your procurement system.
Performance management
The indicators used to measure procurement results can be used as a part of performance management. Although originally designed for and by the private sector, several local governments have adopted such practices to increase accountability.
Measuring performance is an integral part of achieving continuous improvement in the management and delivery of public services. Performance management is a mechanism for ensuring that the government’s overall vision, objectives and priorities are executed.
An adaptation of the “balanced scorecard” to the local government context can allow you to focus on strategy and vision. The scorecard can aid your understanding of several interrelationships that “transcend traditional notions about functional barriers and ultimately lead to improved decision making and problem solving”.
Example
The City of Sant Cugat (Spain) was in a difficult situation. Close to bankruptcy and plagued by low staff morale, the City needed to find a way to increase resource effectiveness and change its political and public management culture and practices.
Advantages of the balanced scorecard are:
- Focus on strategic targets.
- Increase transparency of public management.
- Useful communication and motivation tool
- Develop a culture of continuous improvement
Adopting these measures will help you align your procurement outcomes with the overall strategic objectives of the city administration.
Key Takeaways
- Treat procurement as an ongoing process rather than a series of “one-offs”. Understand what has worked and identify what can be improved.
- Monitoring and evaluating the results of an implemented solution will allow you to improve decision-making and inform policy. Do this by matching results against Key Performance Indicators (KPIs) established from the problem statement definition.
- Cross-check your results measurement and problem statement to ensure alignment.
- When measuring results, differentiate between the results of the implemented solution and the results of the procurement process itself.
- Measuring the results of the procurement process means evaluating whether each step was successful in achieving what it was designed to do.
Now do your homework!
Copy the Worksheet to differentiate outputs, outcomes and impact. To understand the importance of collecting results data systematically in order to inform and improve future projects.
Congrats!
You can proceed to Unit 8: Communicating and engaging markets and communities
