Unit 5: Defining the evaluation framework
Summary
Once the appropriate public procurement path is selected and before issuing the sourcing document, procurement professionals and project stakeholders must establish the criteria by which proposals will be evaluated. Defining an evaluation framework is, together with the problem definition, the most important stage of procurement as it sets out how the solution will be selected. A good evaluation framework will help ensure transparency and reduce risks and delays.
In recent years there has been a clear shift away from traditional fixed procurement criteria toward more open and flexible criteria. Increasingly, procurement uses criteria that emphasize function, overall value and results. This unit introduces the purpose and principles of the evaluation process, defines and illustrates selection criteria, presents an overview on methods to award contracts, and provides best practice recommendations.
Purpose and principles
The evaluation process is a complete review of the received bids based on pre-defined evaluation criteria. The main aim of any effective evaluation in procurement should be to determine the bid that best meets the identified needs and offers best value for money. Value for money refers to the ideal combination of lifecycle costs and quality (or fitness for purpose) to meet your requirements.
This phase is highly dependent on the quality of your preceding preparations: the selection will only be satisfactory if requirements are well defined and procurement objectives and targets are clear from the beginning. For this reason, when choosing your overall evaluation framework, it is essential to focus on the problem statement and desired outcomes, as well as to take into account your analysis of risks and market insights.
Evaluation is a sensitive process as it is how contracts are awarded. It is therefore critical to ensure that the three fundamental principles for the procurement process in general are embedded in this stage too:
- Transparency
- Fairness
- Equal treatment (or non-discrimination)
Building these values into the design of your evaluation framework means you are likely not just to eliminate the risks of your procurement’s outcome being challenged, but also create a more robust and reputable process. By proactively promoting these principles, you can ensure better results and make a significant contribution to a variety of your City’s goals.
Transparency
Ensures a level playing field, which in turn reduces potential risks and legal challenges. This is not simply about disclosure and openness, but also avoiding discretionary decisions and recognizing subjectivity.
Fairness
According to the U.S. Federal Acquisition Regulation (FAR), all bidders should be treated fairly and impartially. The selection and award criteria as well as the evidence required from bidders must be relevant to the subject matter of the contract.
Equal treatment
All bidders and potential providers must be given the same opportunity, based on the same information and criteria, and evaluated in a non-discriminatory manner. Information should be clear and consistent so that the procurement process is well understood and applied as equitably as possible. These principles prevent manipulation, corruption and enhance accountability and integrity, which is essential to preserving public trust.
Evaluation framework
Definitions
There are two types of criteria: one refers to the qualification of the bidder (the selection criteria) and the other to the bid itself (the award criteria). The following table describes each in more detail:
According to EU Legislation, in an Open Procedure the contracting authority will assess a bid against both selection and award criteria in a single process. In other procedure types — restricted, competitive dialogue, negotiated, competitive with negotiation, and innovation partnerships — the evaluation is a two-stage process which permits the contracting authority to shortlist the bidders against certain specified criteria (selection criteria). Shortlisted bidders are then invited to submit their proposals, which are evaluated against the award criteria.
Most commonly, public authorities must publish the criteria used to select the providers invited to bid, in addition to the contract award criteria. These, along with the respective weightings, should be disclosed before the sourcing document is published. If it is not possible to establish weightings, then criteria should be disclosed in descending order of importance.
Evaluation criteria
All criteria should follow the three principles outlined at the beginning of this unit: they should be transparent, fair (relevant) and non-discriminatory (objective). Good criteria will allow the evaluation panel to make a valid comparison of the bids received and will help to determine the best value solution. It should also enable effective competition.
Example 1/ US
The US Federal Government established the Small Business Innovation Research Program under which only SMEs can compete.
Example 2/ UK
While SMEs are vital to a resilient economy, their role in procurement is often limited to subcontracts passed on by prime contractors. This usually involves a disproportionate amount of risk and limited financial gain and independence. In the UK, the Department for Work & Pensions has developed a code of conduct for SME subcontracting (Merlin Standard) aimed at ensuring a more equitable relationship. This allows prime contractors bidding for public contracts to be assessed and evaluated according to these standards, ensuring public money is invested sustainably.
“The evaluation criteria that tend not to work well are those that are overly complex and… do not match the core need”.
Adrian Walker, Global Head of Infrastructure, Energy, Resources and Projects, Hogan Lovells
In terms of the selection criteria, it is good practice to reduce entry barriers for small and diverse providers and not design them in a way that favors a specific provider. With respect to award criteria, a special emphasis should be placed on how the solutions meet your objectives or achieve performance goals.
Note that while you should pay close attention to improving the evaluation criteria in your own procurement efforts, you can also promote the uptake of relevant criteria across the organization to influence broader decision-making.
Awarding contracts
The EU Public Contracts Regulations introduced in 2015 excluded the lowest price as an option to award public contracts, allowing only awards based on the most economically advantageous tender (regulation 67(1)). The ‘most economically advantageous tender’ (MEAT) approach requires the authority to decide the relative weighting it should attribute to quality and price (or cost). In a MEAT evaluation, the quality and price scores are converted into percentages in accordance with the pre-set weightings to create a combined score that should identify the successful bidder.
Looking at the government’s strategic objectives as well as the objectives of your procurement project will help you decide the appropriate price/quality ratio to apply. It is worth remembering that assigning value is not a mathematical exercise but rather a subjective task, and as such should be cross-validated by stakeholders.
“Good public procurement requires imagination and problem-solving… value is a balance that should take into account the characteristics of the proposal and the responsiveness to the requirements. Best value is a compromise”.
John Adler, Vice President of Procurement at Dallas Area Rapid Transit
When weighting the criteria is important to get your priorities right:
- Is the solution relevant? I.e. will it address your need?
- Consider a design approach with the user / citizen at the center (‘user-centered design’). How will the solution improve citizens’ lives? Which citizens will it impact most/least?
- Envision the scale and broader impact. Will it create environmental and economic benefits?
- Anticipate risks associated with the solution.
“The government has to first identify its priorities in terms of resilience and then utilize the procurement function to find how that strategy connects with best value”.
Rick Grimm, Chief Executive of NIGP (The Institute for Public Procurement)
Best practice recommendations
Evaluation panel
The evaluation process should be entrusted to a multidisciplinary panel representing all stakeholders involved, internal and external, in order to obtain a balanced and diverse assessment.
“Develop the human capital required to assess solutions by involving private sector knowledge and experience.”
Adrian Walker, Global Co-Head of Infrastructure, Energy, Resources and Projects, Hogan Lovells
Those involved in the process must maintain integrity and professionalism in all aspects of evaluation. The evaluation panel should:
- Be competent: have the necessary skills or experience to be able to assess bids against the award criteria.
- Reflect the diversity of the target group (age, gender, ethnicity…) and/or have representation of someone who is or will be directly affected by the problem or need identified.
- Be fully briefed in the evaluation methodology and trained to understand the criteria.
- Consider the bids individually with sufficient time to reflect on them before coming together to compare thoughts and agree scores.
This approach ensures that the risk of potential bias is reduced as far as possible. Furthermore, it prevents falling into the biggest risk, which is procuring a solution that fails to address the need or solve the problem.
Example 3/
The City of Eindhoven looked for ways to fight child obesity through a pre-procurement project. The City designed an evaluation panel that included an 11-year old girl to assess the solutions that were sourced.
Example 4/
The Outer Hebrides, a remote island chain in Scotland, involved citizens in developing the evaluation framework as well as lead the quality evaluation for their public bus service contract.
Conflicts of interest
An ‘organizational conflict of interest’ exists when a contractor is unable or unwilling to provide the government with impartial or objective assistance or advice. At Citymart we are defining a ‘conflict of interest’ is a set of circumstances that creates the risk that objective professional judgment could be influenced by other competing personal or financial interests, and managing conflicts of interest is important to ensure that fair competition is not undermined. Taking appropriate measures to prevent, identify and remedy conflicts of interest will allow you to treat all bidders equally.
Impartiality may also be compromised when staff involved in the procurement process or evaluation panel members have, directly or indirectly, any conflicting financial or personal interest. Avoiding those situations will increase trust, transparency and efficiency. It is good practice to brief both staff and the evaluation panel about conflicts of interest and give instructions on how to deal with any that might emerge during the process.
Terms and conditions
Terms of contract determine under which conditions a service or product may be provided. Most cities have standard terms which are often in excess of 50 pages even for a small service or consulting contract.
Pay attention to the terms of contract as these may have direct consequences on who participates and what type of solution or business model they might propose. Get copies of your standard terms and familiarize yourself with them. Highlight any terms that may undermine your objectives and start conversations early on about how they could be changed as this may be a lengthy process.
It is good practice to consult on the terms of contract externally to ensure they do not present a generalized difficulty to the market. As a rule, bids must be made on the basis that no substantial amendments to the terms and conditions are required. A bid that required serious deviation could be considered non-compliant and would therefore not be considered.
Key Takeaways
- In evaluation it is critical to ensure the integrity of three fundamental principles: transparency, fairness, and equal (or non-discriminatory) treatment.
- Cross-check award criteria with your problem statement, overall strategy, your City’s strategic objectives and its broader social, economic, and environmental impact, to ensure they are fully in line.
- Develop selection criteria that do not exclude small providers. For instance, allow them to work in partnership with bigger providers, or lower the economic or liability requirements.
- When designing your criteria think from a user/citizen perspective. Consider design, functionality, and results. Think of the communities that are directly affected by the problem.
- Select a diverse and inclusive evaluation panel that combines expertise from public, private, and academic sectors. They should be briefed and trained on the evaluation criteria, methodology and ethical issues, such as avoiding conflicts of interest.
Now do your homework!
Get a copy of the Worksheet to reflect on the importance of evaluation principles as well as the importance of defining criteria that takes into account overall (‘big picture’) goals.
Congrats!
Proceed to Unit 6: Managing risk
