What Works Cities Assessment Guide

Certification Criteria

  • Data Governance
  • Evaluations
  • General Management
  • Open Data
  • Performance & Analytics
  • Repurposing
  • Results-Driven Contracting
  • Stakeholder Engagement

Data Governance

  • Your local government has a documented methodology for routinely collecting and updating the data inventory.
  • In the past 12 months, the inventory has been updated.
  • The inventory covers datasets from more than 80% of the departments in the local government.
  • A version of the data inventory is published online.
  • A copy of the data inventory.
  • A documented process for collecting and updating the data inventory.
  • Your data governance process is guided by a statement of purpose or documented set of goals and objectives.
  • Committees engaged in data governance meet to carry out data governance responsibilities at least quarterly.
  • Minutes or agenda of your two most recent data governance meeting.
  • Open data or data governance documents that describe the roles and responsibilities of the data governance team.
  • Statement of purpose, relevant clauses of enabling policy that identify the goals and objectives of a data governance body, or other official documents identifying the goals of your data governance process.
  • A schedule of your upcoming data governance meetings.
  • Your local government identifies data quality problems by conducting a data quality audit.
  • Your local government implements processes to improve identified data quality problems.
  • Your local government documents standards for internal data so that certain common fields (e.g., street address) are consistent across the organization.
  • Your local government establishes feedback channels for data users to identify data quality issues.
  • A summary of your local government’s process to improve data quality.
  • A description of what happens when local government staff identify a low-quality dataset and the process for improving the dataset before it is published.
  • An example of a dataset that has been improved through this process.
  • Your local government has a documented process to reduce the possibility of inadvertently releasing private or confidential data.
  • Your local government offers staff information or training about how to assess and reduce security risks associated with your data.
  • Your local government has a process for engaging residents and other external stakeholders in the government’s security risk management program.
  • Internal memos or training materials that inform local government staff who manage and use data about how to protect the privacy and confidentiality of residents.
  • Your local government has a written process documenting how to share data internally or with trusted partners.
  • Your local government has a user-friendly method of collecting requests to share data and requires data-holders to respond in a timely way.
  • Your local government limits data-sharing agreements to datasets that are non-eligible for open data; otherwise, datasets are publicly shared.
  • Where there is a risk of liability, your local government uses data-sharing or data use agreements.
  • Copy of a current data sharing agreement.
  • Copy of relevant policy or process document.
  • An inventory of data sharing agreements.

Evaluations

  • The policy or ordinance requires a justification for why programs should or should not be evaluated.
  • Documents containing the policy or ordinance.
  • Documented defined standards, methodologies, or tools to help staff rigorously evaluate practices, programs, and/or policies.
  • Departments are required to explain how they will measure the impact of a new investment using a rigorous evaluation method.
  • A budget memo or other relevant document that demonstrates new or renewed programs will be rigorously evaluated as a condition of funding.
  • Experimental or quasi-experimental evaluations can be launched with or without the support of external organizations (e.g., academic institutions, vendors).
  • The evaluation has been launched if the intervention has been implemented or if analysis of evaluation results has been completed.
  • A brief summary of two experimental or quasi-experimental evaluations that your local government has launched.
  • If evaluations are conducted in-house, examples and evidence of evaluations (e.g., report summarizing the evaluation and results) should be provided.
  • If evaluations are conducted with external organizations, formalized documentation that defines how the scope of the evaluation aligns with the goals of the local government should be provided.
  • An example of a different, or newly justified, decision and how it was informed by evaluations.

General Management

  • In the past three years or within the current administration, your local government’s mayor, chief executive, and/or executive team has communicated to staff, at least twice, the commitment to data-driven governance.
  • In the past three years or within the current administration, your local government’s mayor, chief executive, and/or executive team has required data analysis, evaluation findings, or other evidence to inform at least two of her/his major decisions.
  • Examples of internal staff emails, memos, or council presentations that clearly state leadership’s commitment to data-driven governance and establishes it as the organizational expectation.
  • Your mayor or chief executive communicated at least one such example in the most recent State of the City or equivalent address.
  • In the past 12 months, your mayor or chief executive has used data and evidence to publicly communicate the work and impact of government on four different topics, programs, and/or announcements. For each instance, unique examples for at least two forms of communications (ie. earned media, owned media, social media, public appearances) must be demonstrated.
  • A link to the most recent State of the City or equivalent address during which the mayor and/or chief executive used data and evidence to publicly communicate the work and impact of government.
  • A document with eight links to media examples, divided by topic, program, and/or announcement.
  • In the past 12 months, your local government has publicly communicated examples of how it is governing using data and evidence and/or shared stories of progress made as a result for four different topics, programs, and/or announcements. For each instance, unique examples for at least two forms of communications (ie. earned media, owned media, social media, public appearances) must be demonstrated.
  • A document with eight links to media examples, divided by topic, program, and/or announcement.
  • The performance management team is responsible for the following activities: collecting and managing the data used in the performance management program, preparing analyses and reports of departmental data, overseeing departments/team members through the performance management and analytics process
  • Performance management documents should describe team members and their roles and responsibilities. Designated team members should also be able to clearly articulate the role they play in the local government’s performance management and analytics efforts. When possible, organizational charts and job descriptions could suffice.
  • The data governance team is responsible for the following activities: managing a comprehensive data inventory, improving data quality, overseeing departments/team members through the data management process.
  • Data Governance documents should describe team members and their roles and responsibilities. Designated team members should also be able to clearly articulate the role they play in the local government’s data governance efforts. When possible, organizational charts and job descriptions could suffice.
  • Your local government has at least one individual to lead the design, implementation, and analysis of experimental or quasi-experimental evaluations. Alternatively, if your local government has an established relationship with external organizations to conduct evaluations, there is a designated leader or team within the local government to manage the projects and relationship.
  • Your local government has a process to match the leader or team’s expertise with evaluation topics that enable them to work on priority projects across the local government (i.e., across departments).
  • Names of relevant staff and their roles and responsibilities. Designated team members should also be able to clearly articulate the role they play in the local government’s evaluation efforts. Organizational charts and job descriptions could suffice.
  • Your local government has at least one individual to lead results-driven contracting work. Designated leaders and/or the team will either be in a senior position in a centralized office or must have succeeded broadly in applying results-driven contracting to their most important procurements at the department level with assistance from a centralized office.
  • Names of relevant staff and their roles and responsibilities. Designated team members should also be able to clearly articulate the role they play in the local government’s results-driven contracting efforts. Organizational charts and job descriptions could suffice.
  • Your local government communicates inside and outside of government about newly available data or opportunities to get involved with its data program.
  • Your local government holds in-person or online events that feature organizational data or insights from organizational data.
  • Your local government seeks relationships with external partners who help train staff to use organizational data to provide the local government with useful insights.
  • Your local government conducts internal trainings to ensure that staff can make good use of organizational data in their daily work.
  • A summary of the trainings available to local government staff that help them use data and evidence to make decisions and how you communicate the availability of these trainings to all staff.
  • In the past three years, your local government made accessible to city staff a value proposition and/or scope of services for the use of data and evidence in decision-making.
  • Your local government has a codified policy or strategy document mandating the development, maintenance, and utilization of data and evidence in decision-making.
  • In the past three years, your local government shared examples of success for using data and evidence internally from at least three separate departments or agencies.
  • In the past three years, your local government developed guides, skill development plans, and/or other products to upskill local government staff on the use of data and evidence to make decisions.
  • Documents that contain the relevant value proposition, skill development plans, internal communications, policies and/or strategy documents that support your local government in carrying out these strategies.

Open Data

  • The policy creates a governance structure for the proactive release of public information.
  • The policy includes plans to collect regular public input on the open data program.
  • The policy ensures periodic review of policy language and implementation practice through at least annual reporting.
  • The policy commits to public disclosure of decision-making pertaining to collection, prioritization, storage, use, analysis, and publication of public data and information, including explanation for sensitive, protected, or other information ineligible for release.
  • Link to the publicly available open data policy.
  • The process includes ensuring consistent high-quality and complete metadata.
  • The process forces local government employees to review data quality before it is published.
  • The process prevents the inadvertent release of protected data.
  • The process assesses community needs in prioritizing data releases.
  • Open data governance documents that detail the open data publishing process.
  • Publicly available data is appropriately contextualized with clear language, contact information, relevant policies and/or reports and other documents. If there are any types of data that are on separate sites, then the main open data portal clearly links to those.
  • Publicly available data is machine-readable, downloadable, and freely available.
  • Link to your local government’s open data portal.
  • Your local government uses at least one common data standard (e.g., GTFS) to publish data.
  • Link to a sample dataset, along with naming the conforming standard.

Performance & Analytics

  • Your local government documents its key citywide priorities and/or goals (e.g., strategic plan, visioning statement, etc.).
  • Your local government has assigned targets, metrics, and a timeframe for measuring progress toward the achievement of a publicly stated strategic goal(s).
  • Link to strategic goals that are citywide, time-bound, specific, and measurable.
  • Senior leadership attends at least 50% of the performance management meetings.
  • Your local government has a designated process for conducting analysis, preparation, and follow-up for each performance management meeting.
  • Your local government has a set schedule for performance management meetings.
  • Minutes or agenda of your two most recent performance management meetings.
  • Schedule of upcoming performance management meetings.
  • Your local government has documented targets, metrics, and a timeframe for measuring progress toward the achievement of a publicly stated strategic goal(s).
  • Strategic goals and progress towards them are shared with the public at minimum annually.
  • Link to where progress toward strategic goals is publicly shared.
  • Your local government has a Chief Information Security Officer or a similar role and/or function.
  • Your local government has an audit mechanism, which can be done by a third party or other independent agency.
  • Your local government provides trainings to update city staff on policies and procedures to manage the risk of data breach, loss, or unauthorized manipulation.
  • Name, title and email address of the Chief Information Security Officer or similar role and/or function for your local government.
  • Copy of relevant documented policy and/or practice.
  • Evidence that trainings to city staff on policies and procedures took place.
  • Your local government documents where and how automated decisions are taking place, what risks they present, and the strategies in place to mitigate those risks.
  • Your local government engages residents at least annually to gather feedback and concerns about how data is used by the local government and its partners.
  • Your local government has documented policies or practices to attempt to reduce the impact of demographic or geographic bias.
  • A summary of the practices, included specific examples, used by your local government to reduce risks associated with automated decisions and artificial intelligence (AI).
  • Descriptive analysis (averages, counts, maximums/minimums)
  • Diagnostic analysis (regression, correlation, data mining)
  • Geospatial analysis
  • Prescriptive analysis (algorithms)
  • Two examples of data analysis that have been shared with your local government leader.
  • An explanation of how leaders get access to data analysis for decision-making.
  • An example from the past 12 months of a specific instance where policy decisions were made as a result of your local government’s performance and/or analytics program.
  • Examples of analysis conducted, including memos and presentations from performance management meetings. Local governments should point to specific instances where policy decisions were made.

Repurposing

  • The process requires budget proposals, requests, and/or renewals be tied to strategic priorities, performance, data, and/or evidence.
  • A budget request form, budget renewal form, or budget memo.
  • An internal process or external policy, resolution, and/or ordinance that is used to determine when a program should be discontinued. If relevant processes or policies exist at the department or agency level, this must be demonstrated in a minimum of three departments or agencies.
  • Your local government has a documented process, informed by data analysis and resident feedback, to make different or newly justified budget decisions.
  • An example of a different or newly justified budget decision that was made based on analyzed data and resident feedback that utilized the documented process demonstrated by the city for RP2.
  • Your local government has shifted (or begun the process of shifting) funding across programs.
  • The program was shown to be ineffective through data analysis or evaluation, and resident feedback.
  • An outline of the process you are pursuing, or have pursued, including the identification of the particular practice, program, or policy that failed to achieve its desired outcomes, and the new program the funds are being shifted to.

Results-Driven Contracting

  • Three unique examples of either procurements (e.g.), RFPs, RFQs). contracts, or grants to acquire two different goods or services (i.e., they cannot be three contracts for shelter services, but could be an RFP for shelter services, an RFP for supportive housing, and a contract for transitional housing).
  • For each submitted procurement, list the page number on which we can find defined strategic goals.
  • Your local government uses a performance tracking system to flag when vendor performance is off-track during the course of the contract.
  • Evidence must be supported by the same examples used in the first Results-Driven Contracting criterion.
  • For each submitted procurement, list the page number on which we can find metrics aligned with the strategic goals.
  • Materials used to track data for comparing vendor performance such as vendor report cards, dashboards comparing vendors across a consistent set of metrics, and/or reports with process and outcome metrics for at least one product or service type.
  • Examples of contracts with vendors providing similar products or services that outline consistent metrics and/or performance reports of similar vendors.
  • To qualify, you may demonstrate structuring the procurement, contract, and/or grant in terms of any of the following, based on desired outcomes:
    1. Procurement vehicle (e.g., problem-based, agile, or design build structures)
    2. Contract type/Payment mechanism (e.g., fixed-price, cost-type, performance-based, etc.)
    3. Vendor selection criteria
    4. Scope of work
  • Evidence must be supported by the same examples used in the first Results-Driven Contracting criterion.
  • For each submitted procurement, list the page number on which we can find how the procurement aligns the vendor’s incentives with the local government’s strategic goals.
  • Your local government collects and shares performance data with and from vendors.
  • Your local government engages contractors at least quarterly to course correct when challenges are identified. Meetings for primarily compliance purposes do not qualify.
  • Evidence should be supported by the same examples used in the first Results-Driven Contracting criterion. If the information does not exist in the submitted contracts, please provide examples of how performance data was used to identify a challenge and how the local government then engaged with the vendor to design a course correction during the contract term.
  • For each submitted procurement, list the page number on which we can find how performance data is used to identify challenges.
  • An explanation of how the materials uploaded for RDC3 are shared across departments.
  • Your local government shares data on contracting using Open Contracting Data Standard requirements.
  • Your local government shares a downloadable and machine readable catalog of previous contracts, as well as procurements set to expire or upcoming in the next 6–12 months. For contracts set to expire, data fields must include project name, project description, estimated timing, estimated dollar size.
  • Link to shared data, documents, information about contracts, procurement and/or vendor performance.

Stakeholder Engagement

  • Your local government analyzes data download statistics, website usage statistics, public records request counts, or other measures of information demand to inform open data releases or open data portal design.
  • Your local government has established online channels for residents to provide questions or comments around open data, such as online feedback forms or GitHub repositories.
  • Your local government has user personas and/or use cases for open data. These should be developed using research on examples of internal or external data use, or by using collaborative workshop techniques.
  • A link to an online feedback form for your local government’s open data.
  • A document of your user personas and/or use cases for open data.
  • A link to a dataset of download statistics and/or an explanation of how this data is used.
  • Your local government’s how-to guidance helps residents analyze, engage, or use at least one open dataset or tool.
  • Your local government produces high-quality metadata for a majority of available open datasets that conform to international best practices.
  • Your local government publishes contact information for a person who can support the use of available open data.
  • A link to your local government’s how-to guidance to help residents access and use city data.
  • Your local government’s staff participate in in-person community events where staff present open data program updates or announce newly available open data relevant to community groups’ needs and activities.
  • Your local government regularly collaborates with community members to use data on projects and/or supports local civic technologists’ use of open data by publishing open-source code for open data tools or products.
  • Evidence of local government staff participating in in-person community events to communicate open data needs and activities.
  • Results of community-driven collaboration using city open data and/or examples of open source code.
  • Your local government has provided at least one annual in-person and/or online training to support data users in accessing, understanding, or using publicly available datasets. Alternatively, your local government partners with local open data experts or assigns dedicated City staff to answer questions about open data or conduct community training for local data users.
  • Your open data team conducted in-person demonstrations of at least one open data product or tool in the past year using local government resources, local government space, or local government financial sponsorship for the demo.
  • A summary of efforts made to educate, activate, or upskill partners to better understand and utilize administrative and performance data.

--

--

What Works Cities Certification is the national standard of excellence for well-managed, data-driven local government. Learn more: http://bloombg.org/2o72SzG.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store