How our driverless tram sees the real city

Hey Medium!

To keep it short, we have an experimental tram that we’ve been testing on one of the city routes in Moscow. We’re testing the self-driving functionality within a restricted area, and in the city, the autopilot is an active assistant for a human tram driver. The driver keeps his hands on the controls, but the test subject is an autonomous autopilot. The tram looks exactly like its conventional counterpart because we made a joint decision with the manufacturer to hide the instrument boxes deep beneath the dashboards and to use the available displays for our interfaces. The only small difference is a few cameras under the windscreen, a radar system hidden under the panelling, and a GPS sensor on the roof. We also sometimes add a lidar for troubleshooting.

In the course of testing, we learnt that driving regulations and the real situation on the roads are two very different concepts even for a tram.

In fact, the tram is an ideal sandbox for vehicle autopilot testing, and we have now made this a reality. Our “cheat codes”:

• We know the route and can be sure that our vehicle will follow it to the letter.
• We can follow this route in advance and map the locations of traffic lights and other objects to ensure easier recognition.
• Trams can’t change lanes. The main load on a car autopilot is related to steering in the right direction with thousands of scenarios, but we don’t need these on a tram.
• It brakes almost instantly and a little abruptly, so forecasting the trajectories of other vehicles on the road is less complicated.

The real challenge is dealing with passengers who want to be the first on board even if it means risking their lives.

Testing stages

The first two stages included autopilot tests within restricted areas (the plant, the tram depot, and the area in front of it). The tram can follow the route, make stops when necessary and open its doors, stop for pedestrians and other cars, and brake to avoid obstacles. In parallel, we were working on a driver assistant, which has already been tested in the city. The assistant includes two primary subsystems. The first one is speed limitation (SL), which makes sure the driver does not exceed the limits. For instance, tram operating rules prescribe a speed of under 20 km/h (12.4 mph) on bridges and 10 km/h in tunnels, while points must be passed over at a speed of 5 km/h, and so on. The other one is automatic braking (AT), which makes sure the driver doesn’t run anyone over by slowing down (up to a complete stop) in front of hazardous objects, red traffic lights, and closed points. The co-pilot controls the tram jointly with the human driver. This enabled us to gradually test various autopilot components in increasingly challenging conditions.

Now we’ve finally reached the stage of testing on an actual tram route. Under the assistance system testing programme, we’re following a real city route, but our tram doesn’t pick up passengers at stops and sometimes doesn’t even stop there. The “trial run” sign isn’t visible from afar, and people don’t always pay attention, especially in the areas where tram #17 is the only one available. Passengers often think we’re going to make a stop and start to boldly cross the street.

The pedestrians in question

When it comes to co-pilots for agricultural machinery and combines, our priorities are the highest possible harvesting efficiency and avoiding large obstacles such as tractors. With a tram, it’s different. Efficiency is important, of course, but the highest priority is to avoid hitting obstacles — and people. Apart from public safety considerations, there’s a financial motivation too: a road accident involving a tram means not only repair costs but also an expensive vehicle (lost profits) or even the entire route being out of action for a long period.

Luckily, we’ve already got a radar and three cameras to keep an eye on the surroundings, which means we can identify obstacles with high precision.

The problem is that a tram often crosses paths with people. In Moscow, people often stand at the very edge of the platform and seem to be swinging in the wind.

Our collision prevention model essentially does the following:

• Identifies obstacles and calculates the trajectories of their movement and possible behavior.
• Determines intersections with our trajectory and the timing.
• Reduces speed in advance and issues a warning or performs emergency braking.
• In the meantime, the radar is on the lookout for more obstacles ahead.

The model used for a pedestrian is an object that can walk, run, or jump at a speed of up to 5 mph. It is small, has almost no momentum, and is therefore capable of random movements. Running over it, however, is absolutely out of the question.

At the moment, we’re working on a more promising multi-hypothesis model. We realise that pedestrians at a given location can be walking up and down the platform, standing on it, or crossing the street. If none of the motion models apply, the pedestrian under analysis is classified as unpredictable and therefore capable of random movements, which means we must avoid them at all costs.

The first versions of the firmware couldn’t even approach the stop because unpredictable pedestrians were standing too close. The driver had to disable the co-pilot at every stop during that trip.

We were able to lower the sensitivity of the model in the vicinity of stops (since we know their exact location on a standard route) and increase it when on the move. It’s a working option to solve the issue of approaching the stop, but it’s not a good one if someone decides to jump in front of the tram.

Eventually, we had to build skeleton models of every single pedestrian (including those in the crowd, which is no mean feat) and monitor the positions of their joints. Now the tram understands that most pedestrians rarely walk backwards. We collected a more or less diverse training set of pedestrians crossing the street and trained the neural network to identify intent by posture. It worked. Kudos to modern technology for enabling us to stuff a small box with the combined computing capacity of the entire planet at the dawn of the space age.

The stops are at ground level (a feature of the route), and we’ve learnt to predict motion well enough. When working on the first prototype in 2018, we were unaware of this feature and requested emergency braking, thinking it was a regular hard stop. I was thrown off my feet and landed in the passenger compartment. “I warned you: hold tight,” said the driver. Passengers experience this very rarely; I’m sure that emergency braking would lead to all the passengers falling over.

Traffic lights

When a driver starts working on a new route, they make a few trial runs with an experienced driver who has been along the route hundreds of times. We follow the same procedure with our systems, mapping the route in advance to ensure the high precision of their work. An unpleasant feature of the route is that human drivers make the same mistakes as the autopilot during the first runs without mapping when trying to recognize road signs while driving. This inspires hope. Route mapping has ensured precision.

The exact location of traffic lights enabled us firstly to lower the detector accuracy requirements (if the detector fails where there are most certainly no traffic lights, it isn’t critical) and secondly to decide how to react to detected traffic lights. At every intersection, we know which particular set of traffic lights regulates our movement (and which doesn’t), where the stop line is, how long the green light is on for, and which signal means permission to go.

Without prior knowledge of the topology of the intersection, we may have passed a flashing green light and braked for a secondary red light at the exit from the intersection. Our system knows, however, that the further traffic light regulates the entrance to the intersection and not the exit from it.

When detecting a red light, the system needs to calculate the time of arrival at the signal and permissible speed.

External conditions are extremely important for emergency braking. If the rails are wet or covered with ice, an incorrectly performed emergency brake sends the tram into a skid, and the driver has to either use the track brake (a tram has three different brakes) or try to get out of the skid and perform regular braking. Unfortunately, the track brake may damage the rails. This is why our speed profile (and braking distance calculation) takes these conditions into account to prevent the tram from going off the rails or skidding.

The braking distance is calculated from the braking trajectory on a specific section of the route and the road surface condition. Braking is more difficult when it rains and easier in hot weather. The tram learns the weather by recognizing what’s going on in the street and assessing the results of the driver’s first careful actions at the beginning of the route and later on (that is, the co-pilot will know if it starts raining). The autopilot chooses from several driving profiles, including “pouring rain”, “snow”, “sun”, and “ice”, built on the driving patterns of human tram drivers. The driver can select one of the modes in advance from the interface and lock it in. By default, the tram assesses the conditions in real time.

If the driver doesn’t react to a red light, the tram will start slowing down and then brake softly.

We were in for another batch of surprises, however. Early versions of the co-pilot erred on the safe side, stopping the tram several yards before the stop line. The drivers shook their heads disapprovingly, saying that such an autopilot was doomed. As it turned out, the tram had to stop closer to avoid blocking the previous intersection with its rear. The current version of the robot requests several trial runs with an operator to take note of all the specifics and knows which traffic lights require the tram to stop closer to them.

Intersections

The next portion of disapproving head-shaking occurred at intersections during rush hours.

The autopilot, which had done very well at the same intersection in the morning, struggled.

As it turned out, car drivers’ biological neural networks prioritized fast driving over safety, and they were almost rubbing their vehicles against the side of the tram, which meant a high risk of collision.

The trickiest part was to adjust the settings to enable the tram to proceed in such complicated conditions without constantly braking. We needed to start moving so as to change the reaction of car drivers and then reassess the road situation accordingly. This was one of the most complex tasks.

Another risk we had to consider was reckless drivers making turns out of secondary roads at full speed without looking.

Since the real-life road situation during rush hours features frequent traffic violations on the part of all participants, the tram driver often wants to take control of the vehicle to stop the autopilot from being too cautious. We made a button called “10 Seconds of Freedom”, which temporarily disabled the system, lifting all the restrictions. The drivers use this button most often at blocked intersections. The frequency of its use is one of our KPIs for system performance assessment.

Aiming for a better future

Level 1 includes avoiding collisions, assisting the driver, and reacting faster than them to obvious dangers.

Level 2 includes driver control mode, switching points on the route, and speed control on particular sections. (For now, the driver may enter a curve at an unsafe speed or choose the wrong gear for icy rails, but the tram won’t exceed the speed limit set in the profile.)

As for higher levels, we can only test them within restricted areas. The movement of the tram is controlled by a human driver. Our co-pilot follows the route and monitors the road situation, doing a much better job of keeping the tram and other road users safe.

The better future we’re aiming for is a completely autonomous tram.

We have tested our solution on Russian tram models such as the Vityaz and Bogatyr (this one is not yet in operation). Their systems are managed through a CAN interface unit. There aren’t any non-native modules in the cab; only system interfaces. The only piece of equipment we added was a camera on the windscreen.

We are using industrial vision PoE cameras with a variety of lenses (because we need a long-focus lens for objects ahead and a wide-angle lens for side vision). Our radar system is more noteworthy: with a range of 150 meters and a beam width of 160 degrees, it works even in rainy weather. The hardware suite also includes a high-precision GNSS sensor for navigation and a unit to integrate it with the tram. The data is processed on the spot by a 65 TFLOPS computing unit (which looks like a big radiator with a few ports). As you can see, the vehicle in question is far more expensive and advanced than your average tractor, so there’s plenty of high-end equipment to choose from.

The share of light rail transport in the public transportation network in Moscow and other cities is growing. There are about 300 well-developed tram networks worldwide. The best ones are located in China, Russia, and Turkey, while Europe is following suit. China is experiencing a tram network boom because they are funded regionally, while underground networks require federal funding. Naturally, it’s easier and faster to secure regional, rather than federal, funding. We recently signed a contract with FiTSCO, one of China’s main urban rail transit signaling system providers, to launch our system in China, which will be our first tram project in this country.

Russia has 1,500 trams out of a total of 30,000 in the world. With the annual production of 2,500–3,000 new trams, we’ll soon move on from equipping the available trams with Level 2 and 3 autopilots to integrating with the manufacturers.

You already know about our co-pilots for combines and tractors, and now you’ve learnt about one for city trams. We also have a solution for shunting locomotives. The combined market volume for all these vehicles is comparable to one-third of the car market. No meagre fraction, but a third — and we are the only ones entering this market with such technologies. Watching the world change is fascinating, but it’s still more fascinating to change it.

Lastly, I’d like to point out that the radar system we’re using is our own creation. First, we had to gather low-level data to obtain more accurate models from the neural network, and then we gradually developed and assembled our prototype — and there you go. I think that our experience of putting together a radar warrants another story — if you’re interested.