Better bus predictions (a lot better)

Why do bus predictions matter?

The very existence of bus predictions is a lifeline. Buses run in mixed traffic, so they almost never arrive exactly according to the schedule. Waiting for the bus inside is better than waiting outside. Knowing you have 3 extra minutes to play with your kids or finish an email or sit and stare while sipping your coffee rather than rushing to a bus stop is still amazing, 8 years later.

What are the components of a good prediction?

  1. Follows the goldilocks principle (but focused on “not too hot”).
    Buses have inherent variability. When they are close, the time it takes to arrive depends on things like the pattern of streetlights. They could make a series of green lights in a row, or catch all reds. Further out, it is the traffic and passenger demand that are the main sources of uncertainty. Think about when a bus arrives as a draw from a distribution. We may have 99% certainty that the bus will arrive between 2 and 10 minutes from now— but that’s too big a range. When we make a point estimate for customers, we want to draw from that distribution so the bus won’t arrive earlier than predicted and passengers miss it (e.g. the 5th percentile). For dispatching, we want the most likely time the bus will arrive (e.g. the median of the distribution of predicted times).
  2. Predicts well both in the short term and the medium term.
    This is hard. It could take 1 minute or 3 minutes to get through the 2 streetlights 3 blocks from my house. In the medium term, traffic, number of passengers, and weather all come into play. If the outcomes are widely distributed, because we are predicting at the 5th percentile, we tend to end up with predictions that are much earlier than the bus actually shows. That is better than the opposite — riders won’t miss their bus — but it’s not a great experience because they have to wait for their bus longer than expected. It is not a problem unique to the MBTA, and projects like bus lanes, TSP, and a better fare system which leads to all-door boarding will mean better bus service — and better predictions.
  3. Predicts for all vehicles that are running.
    Riddle me this: if we don’t tell you a vehicle is coming toward you, and you are relying on our predictions, is it really arriving?
  4. Differentiates scheduled and real predictions.
    Far in the future, we can’t make a better prediction than the schedule. If it is the first trip of the day for the driver, and they haven’t logged on to the bus yet so we know they are on their way, we don’t have a better prediction than the schedule. If the GPS unit on the bus isn’t sending information, the same. A good prediction makes these circumstances clear.
  5. Can handle interlining, other things that complicate predictions.
    Our buses often go from one route to the next to make their schedules more efficient. This means that the bus you take may be on a totally different route when you’re looking to see when it’ll arrive at your stop. A good prediction can handle that complexity.

How do you measure prediction accuracy?

First some numbers. On an average day our buses make 450,000+ stops. For this exercise we record predictions every 15 seconds, so long as they have changed since the last prediction. That makes for, give or take, over 30,000,000 predictions, or about 70 predictions for every stop a bus makes. That is a lot of data, and it gives us some confidence in the results that follow.

  • Short term (0–3 minutes away): Between 1 minute early and 1 minute late
  • Medium-near term (3–6 minutes away): 1.5 minutes early to 2 minutes late
  • Medium-far term (6–12 minutes away): 2.5 minutes early to 3.5 minutes late
  • Long term (12–30 minutes away): 4 minutes early to 6 minutes late²

How good are the predictions now?

They’ve served us well for the better part of a decade.

  • The spread across time periods is relatively consistent: 76% fall within 1 minute when the bus is 0–3 minutes away; 74% are accurate in the 3-6 minute bucket; 78% in the 6–12 and 12–30 minute ranges.
  • These basic numbers hold within a few percentage points across all types of routes, across weekdays and weekends.
  • Approximately 99.7% of stops that our regular bus service make have predictions.
  • They’re are also weighted to predicting that buses will come after the prediction than before the prediction. For example, for buses 12–30 minutes away, 25% of predictions were up to 4 minutes early; 52% were up to 6 minutes late, and 23% were outside both of those windows.

How much better are the new predictions?

  • Weekend predictions — when buses are less frequent, were 86% accurate.
  • Predictions in the short term were 82% accurate, in the medium term 83% and 85% accurate, and in the long term 82% accurate. The highest accuracy is for trips in the sweet spot of when you would be thinking about heading to your stop.
  • Predictions are weighted toward predicting buses will come earlier than they actually do, so you don’t miss your bus. For example, in the 12–30 minute bucket, 12% of weekday predictions were up to 4 minutes early, 71% were up to 6 minutes late, and 17% were outside both of those windows. That means fewer missed buses for our customers, across the board.
  • Predictions are also updated more frequently. The new system will make more than 50 million a day. That means if something goes wrong, you’ll know about it sooner.
  • We maintained that 99.7% of stops have predictions.

How do we know that our predictions are better?

We don’t make the predictions ourselves. We use companies that do this for lots of other customers, so we can get the benefit of their specialized expertise.

  1. We provided our bidders with a historical feed, so they could calibrate their systems.
  2. We then provided them with an actual real-time feed for 2 weeks.
  3. We compared their predicted arrival times to our own record of actual arrival times to see which predictions were more accurate.

What’s next?

  • More frequent real-time updates. This means being able to watch the bus and make your own judgement. But it also means better predictions. If we can tell if a bus is stopped at a light or stuck in traffic, we can be more accurate, especially in the short term. We wrote about it about it a few months back, and WE STARTED ROLLING IT OUT THIS WEEK. 151 buses have it as of publishing, and more each day.
  • Fewer phantom buses. They are the worst — when you get what look like real-time predictions for a bus, only for it to disappear from your app when it gets to your stop. That happens often because of scheduled predictions and interfaces that don’t do a good enough job differentiating. What we need is to have better data so that we know when the bus is running but we just don’t have real-time information, versus when the bus actually isn’t running. We’re working to combine various systems and update dispatch processes to deliver on this promise to our customers.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Block-Schachter

David Block-Schachter