Relearning the value of testing at the MTA

Josh Gee
Josh Gee
Oct 24, 2019 · 7 min read

Earlier this year, we pulled together a team at the MTA to build new real time arrival and information templates for the digital screens that are going up across our transit network. This effort built upon the back-end data improvements we made for MYmta, as well as some of the new design and development approaches.

Along the way, I relearned (again) the value of user testing. I’m sharing my experience to remind everyone (including myself) how important it is to show your product to the people you expect to use it. A three hour user testing session made our product approximately ten thousand times better.

Who am I?

I started at MTA New York City Transit about a year ago. My long, made-up-sounding title is the Director of Digital Customer Experience. My mandate is to improve all of our digital customer touch points. Lately, that’s expanded to some MTA-wide initiatives like MYmta and new.mta.info.

I’ve worked in political campaigns, corporate consulting, media, and government. Last month, some guy called me the “Godfather of Forms” and I just about melted.

The Prototype

The MTA is installing tens of thousands of digital screens across the network. If you live in New York, you’ve probably seen a few. This is an exciting new platform for communicating with riders. However, we haven’t been taking advantage of the screens as a truly digital medium. Under the direction of Sarah Meyer, we brought together a joint team of incredibly talented folks from our new Strategy & Customer Experience Digital Content Team and the MTA IT Applications group. Our goal was to build digital screen templates that could take advantage of the back end data improvements we made as part of the MYmta project and give our riders more accurate information. Our main goals were:

  • Give better information about not just when the next train is arriving, but where it is going. Especially during service changes or in unfamiliar parts of the city, riders struggle with this.
  • Give better information about service changes in a digital format, rather than in paper posters.

Most of our digital signs are in sets of two. After some initial prototyping and lots of pen and paper sketching, we designed a pair of screens that use real time data to provide solutions to the needs we identified above. The one on the left is called the “route map”. The one on the right is called the “station dashboard”.

Route Map — My colleague Joe crystallized the concept behind the route map when he called it a “contract with the customer”. If you get on this train, this is where it is going. The stops along the route dynamically update based on where that individual train is going. The transfers update dynamically based on how trains are actually running. All of this is generated from our realtime GTFS data.

Station Dashboard — This template contains everything you need to know about a station. It pulls in service changes currently affecting lines on this station, as well as real-time updates on elevators and escalators inside the station. We also stuck on the general service status overview so riders could get a quick preview of how the rest of the system is running.

The Testing Process

I was adamant we should do some user testing. However, I cannot overstate what a pain in the ass this was. I had to write a usability script. We had to get the templates working in a real station (we set up a whole testing session that was torpedoed by a mislabeled sign). I blew through a critical deadline, with a half-assed explanation for why the extra time was worth it. I was also dreading the session itself, as this user testing would involve standing on a subway platform, begging people to talk with me about digital signs as they were trying to catch their trains.

When the testing schedule was finally on the books, I was flooded with notes from coworkers about how they were going to stop by. A number of them also said they wanted to invite important external stakeholders to see the designs and give their thoughts. In many organizations, especially governments, this is what passes for testing. Don’t get me wrong, it’s important! But I knew I wanted to get feedback from people completely unrelated to the MTA. So, I may have bent the truth to my coworkers (sorry, pals!). I told everyone testing started at noon, and my team got there at 10 am.

Over the course of two hours we were able to fully interview 5 people. Three of them were regular transit riders while two were tourists. We also observed about a dozen more use the signs who refused to talk with us. Not the biggest sample size, but even one was worth it. FWIW, our goal was six. When my coworkers did arrive, they provided additional helpful and thoughtful feedback.

My assumptions:

  • Customers would use the heading to figure out when the next train is arriving, then use the strip to figure out what stops the train is going to make.
  • Customers would use the list of stops to make connections or plan the next leg of their trip.
  • Customers would use the station dashboard to get a sense of what issues are currently affecting that station.
  • Customers would be as annoyed by the 30 second template refresh as I am (This is a tough problem we haven’t been able to work out yet).

Coworker feedback:

  • Change title for both Service Alerts and Elevator and Escalator status to include “at this station”
  • Change order of service status so that they match up with our new priority list of service statuses.
  • This route map wouldn’t work at [specific platform where there are four trains on each side]

Rider feedback:

  • I thought the route line was for the route in general. I didn’t realize it was for the next train.
  • I didn’t realize the map contained any information about arrivals.
  • It’s not clear if this train is running north or south.
  • I don’t know where I am on this map. It should say “you are here”.
  • How do I get to Central Park?

The discerning reader will notice that, at least for the route map, I completely whiffed at what we were trying to do. We all missed the problems that real customers were struggling with.

An Updated Version + Next Steps

We incorporated all of the feedback into the next round of designs:

We made small changes, mostly to the route map, that make the intent of the map more clear. We think they help locate the user in space, as well as tie the core concept (When is the next train and where is it going) more clearly together. We also added the general concept of “how do I find Central Park” to our long-term road map.

We have test versions up in a few stations, and you should start seeing them more in the coming weeks. If you have feedback, feel free to leave a comment on this post or email me at joshua.gee@nyct.com.

Long term, we’re changing the station dashboard to match the card layout of the route map. We’re adding more kinds of information to route maps, including better accessibility information. We’re also working on other templates that work in commuter rail stations. We’re still working on the refresh issue.

I’m incredibly grateful for the team, who worked hard to make these digital templates a reality and are continuing to test and iterate.

The Point

Ultimately, it feels a bit trite to write a Medium post saying “testing digital consumer products is good”. But here’s the thing… it’s really good. No amount of meetings, experience, or expertise can substitute for putting a thing in front of people who have never seen it and asking if it works they way they think it should. It was worth standing in a subway station for three hours. It was worth getting in trouble for blowing a deadline. It was worth talking to a bunch of strangers. I’m much more confident in the product, and can’t wait for everyone to see it. If I’ve learned anything at the MTA, it’s that I’m sure we’ll get a lot more customer feedback.

In the end, telling a little white lie to my coworkers helped me realize the point of testing. I was directly confronted by my own assumptions, my coworker feedback, and real customer feedback — all in order! It was an illuminating contrast.

I realize why neither me, my team, nor my coworkers really understood the questions our riders had: we couldn’t. We are experts at the very thing we are trying to explain. Even when we imagine the kinds of questions people might have, we filter it through a set of base assumptions and core knowledge that we can’t unknow. It’s really hard for us to imagine people don’t know which way North is! Our minds instantly jump to edge cases, even though the average person doesn’t know anything about them. We place a product in the context of the million other initiatives and priorities going on at work, while the user is only thinking about what they need it to do for them in that moment.

It doesn’t matter if you are making something for transit, for health care, or for dogs. By the time you make a digital product, you’re an expert in whatever the field is. That very expertise is what skews the insights you might have.

Make time to show your product to people who don’t know where Central Park is. You’ll be glad you did.

Thanks to Jeff Greco

Josh Gee

Written by

Josh Gee

You can change the world, but first, lunch. Food writing at http://bit.ly/SnackCart. Marketing/Product at http://boston.gov.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade