What to Assess When You’re Assessing
Electric bicycles, scooters, and mopeds are still very new to the transport ecosystem. Though the technologies have matured rapidly and there is incontrovertible evidence of their popularity, many cities still struggle with how the devices should be appropriately managed and what role they will play in the future. Many cities are trying to be iterative, learning from each other and from their own programs in an effort to gradually improve the experience of both riders and non-riders.
For many cities, the most effective way to manage shared mobility is first through the use of “pilots” or “trials”. Whether formal or informal, these pilots are meant to create a feedback loop for both regulators and the general public. The point of this process is typically to understand what role shared mobility could play in the future of the transport network, to engage stakeholders and the general public, and to determine if the regulatory approach taken meets the needs of the city.
Cities should strongly consider documenting the pilot through regular evaluations, ideally annually or semi-annually. In this white paper we will explore several key components of successful pilot evaluations.
Why Prepare an Evaluation?
In order to improve shared mobility regulations in the future, it is critical to take time to evaluate successes and failures. While operators can certainly provide the public with updates on their operations in cities, trial evaluations should ideally be led by cities in order to reflect the priorities of the general public and to ensure integration into the broader regional transport network. When findings are made public, they promote accountability of both the operators and the city, and build confidence in the management of the program.
It is ok if some of the findings in an evaluation indicate areas of improvement- in fact it would be surprising if they did not. Approaching a pilot evaluation with objectivity and humility will lead to better program outcomes in the long run.
A program evaluation is not the same as simply providing open-data. A well-crafted evaluation can be complimented by publishing open-data sets. Opening certain data sets to the public will allow members of the general public to build new applications and analyses. However, to the general public, open data can be difficult to manipulate. A program evaluation offers citizens the opportunity to engage with key aspects of the data in a digestible way.
The Components of Strong Evaluations
The best pilot evaluations are those that begin the preparation before the pilot even starts. Ideally, a city begins their shared mobility program by clearly and explicitly laying out the goals they hope to achieve, the questions they intend to ask about the pilot, and the metrics that they are going to collect in order to determine progress and report out to the public. If these objectives are articulated at the beginning of the program, it is much easier for operators and cities to work together to achieve them.
It is helpful to think about these terms in reverse order:
Presenting Metrics
During the pilot phase, and afterwards, you are likely to have a lot of data in the form of metrics. If you are using our Cityscope product, you will have several key performance indicators, including but not limited to:
- Counts of trips
- Average number of devices
- Distance and duration of trips on average
- Popular (and unpopular) origins and destinations
- Numbers of infringements or policy violations
- The number of trips per device per day (also called “device rotation”)
The scale of these numbers may be interesting on their own (many are seeing more than a million trips every month), but an evaluation should seek to provide context to the specific readers you are trying to engage. Merely reporting the numbers is unlikely to capture the attention of the general public, nor is it likely to support the drawing of conclusions. When looking at metrics, consider presenting them within a helpful context. For example:
The context of metrics is critically important to ensuring that they are easily understood and appropriately interpreted. If shared mobility is to become a feature of the transportation system in the long-term, practitioners must begin thinking about how to report metrics in ways similar to other modes to allow for ongoing and consistent comparisons.
It is also helpful to identify and name certain trends that may be evident in the data. For example, in evaluating a time series, you may see increases and decreases in metrics. Simple analyses that combine metrics together can be helpful for identifying the key trends. For example, it may be unsurprising that the number of trips is increasing when the number of devices is increasing. However, it may be interesting to present if the rate of change in the number of trips is larger or smaller than the rate of change in the number of devices (if you use Cityscope, we have already done this for you by presenting the device rotation)!
Answering Questions
A useful rhetorical device is to pose and answer a set of questions. Over the course of the pilot, it is likely that the program managers have received certain questions from their department leadership, elected officials, the press, and the public at large. In some cases, these questions can be answered in the moment by using existing data. But in other cases, the questions are best collected and then answered at the conclusion of the pilot period.
While at Vianova, we strongly advocate for the use of standardized mobility data in formats such as the Mobility Data Specification (MDS), it is important to acknowledge that “big” mobility data may not be able to answer every question. The City may want to consider employing other strategies to answer questions, for example:
- Asking for specific information from operators that is not present in existing specifications (for example, the share of trips that were taken using a low income plan)
- Asking or requiring operators to offer an anonymous post-ride survey to users to help answer the city’s questions
- Interviewing key stakeholders
- Conducting virtual surveys, or “pop-up” meetings at public events such as festivals or markets
- Conducting a statistically valid survey of shared mobility users and non-users by mail or telephone
Your pilot evaluation may likely include a range of subjective and objective questions. Do your best to answer them, but also acknowledge where you have insignificant information to draw conclusions.
Evaluating Goals
Articulating program goals and evaluating progress towards them is arguably one of the hardest parts of a shared mobility program evaluation. These goals should not be limited simply to measures of compliance (though they should be included). But cities should also include goals about the effectiveness of the program overall- how is it helping cities achieve their objectives around sustainability, social access, intermodality, and other public policies. A generic set of program goals for a shared mobility pilot may look like the following:
Goals should be explicit in their intention (what did you want to see happen?) and should be supported by several metrics or questions. Because the point of a trial is to evaluate an unknown and unproven technology, a successful evaluation need not have a specific, numeric target. But a city should be able to indicate (quantitatively or qualitatively) if progress has been made toward the goal, or be able to articulate why sufficient data does not exist to answer the question.
Failure to make progress on any one goal should not be seen as a failure of the program overall, but it should be viewed as an opportunity to improve. Over the course of a short trial period, it may be quite difficult to draw conclusions about the effect of shared mobility on long-term trends. But continuing to keep the goals in front of mind supports their eventual achievement.
Articulating Vision
A pilot evaluation is a useful way for cities to clearly articulate why the trial was conducted, and how the administration views the presence of shared mobility now and into the future. City staff preparing an evaluation should take the opportunity to provide their vision for a future with shared mobility, and help stakeholders and the general public see the bigger picture. Managing shared mobility is only one small part of a city’s plan for providing safe and reliable transport- how does it fit into the bigger picture? What constitutes a “good outcome”, not just for the trial, but for the future?
Putting it all together
As a trial period draws to a close, a City should publish a trial evaluation. Because the review needs to be objective and unbiased, it is often useful to have support by external experts to share best practices and clear analyses. Whether the entire report is produced internally, independent consultants or brought in, or research partners like universities and NGOs are included, the report should ideally be the work of the City, which authorised the trial.
It should not be taken for granted that the trial period will lead to a permanent, long-term extension of the program, or even any extension at all. The purpose of the trial is to evaluate the technology, the regulatory context, and progress toward objectives.
In addition to presenting the information about the program up to the point of the evaluation, City staff preparing the assessment could consider how the results would lead them to answer questions such as:
- Is the program the correct size for my city, with the correct number of operators and covering the correct geography?
- Is the governance structure (no permit, open permit, restricted permit, license, etc) the correct one?
- Do we need to impose additional regulations, and did the regulations that we did impose have the intended results?
- Is the city adequately resourced to manage the program in the long-term?
- What segments of the public or stakeholders were not involved in the trial but should be included in the discussions for a permanent program?
- Were we unable to answer questions in the assessment because of a lack of data, and do we have a plan to acquire that data in the future?
The answers to these questions should shape the format of the long-term program, if any. After a trial period ends, the periodic collection and review of data should continue in some fashion, and ideally cities can retain some authority to modify the conditions or features of their program as the need arises.
Managing shared mobility is a dynamic and iterative process. The industry is new and best practices are still forming. Cities who produce pilot assessments are doing a valuable service to themselves and their peers in adding to the collective knowledge of how to produce safer and more livable cities.
Pilot/Trial Evaluation Examples
The following are a range of Pilot and Trial Evaluations Vianova believes are good examples of the practice. Many of these evaluations have occurred first in the United States due to the earlier presence of many shared mobility systems. Vianova makes no claims on or about the results of these evaluations, merely that they represent good examples for cities to consider as they develop their own evaluations:
About Vianova
Vianova is a data platform that helps cities better integrate and manage shared, connected, electric and autonomous transport solutions in the urban space, enabling better use of city infrastructure, and promoting safer and more sustainable mobility. Vianova has offices in Paris, Zürich and London.
If you have any comment that you would like to share with Vianova, please send your comments to hello@vianova.io. If you would like to learn more about what it is like to work at Vianova, and join our talented team, visit our job-board or send directly your application to jobs@vianova.io.
