How we developed the world’s coolest autopilot for a road switcher
To state an obvious point, we can say that our switch engine autopilot is the coolest because it’s the only one to have reached the stage of third-level autopilot operational testing. This has been possible only thanks to our expertise in the autopiloting of urban trams and other vehicles, which is critical for those who want to enter this market. Switch engines, or switchers, or shunting locomotives, are widely used, so the task of automating them is relevant and important for manufacturers, but not profitable on its own. We are aware of the progress made by NIIAS (the Russian Research & Design Institute for Information Technology, Signaling, and Telecommunications in Railway Transportation) and Siemens, but we haven’t heard of any real-life use cases of their technology for locomotives.
Given our fair amount of relevant experience and solutions for self-driving trams in Russia and China, we decided to run a few experiments at a major enterprise with a large pool of switchers delivering raw materials to shop floors.
The main challenge in creating an autopilot for a switcher is that its movement is regulated by a multitude of signals, the position of people and infrastructure and commands from the dispatcher. The driver of a switcher has to maintain a high level of concentration throughout their entire work shift (about 12 hours), including night. Unfortunately, their focus sometimes slips, which can result in damage to the switcher or injuries to pedestrians. This is why introducing autopilots for switchers is critical. Accidents lead to significant downtime of entire sections of infrastructure. An autopilot system can significantly decrease the workload of a human engine driver and help reduce the costs created by this downtime.
Our autopilot ensures operational safety under challenging climatic and weather conditions in any light.
Here is the setup:
A switcher needs three cameras because it can travel at a relatively fast speed, its braking distance is longer than that of a tram (which brakes abruptly in an emergency) and the freight in the rear is heavy. This is why the autopilot needs to be able to predict the situation the switcher will confront. It needs to read the signals very frequently in scenarios such as this:
Hence the three-camera setup:
Here is what they see:
And there’s a radar to boot.
The setup is rather expensive. It has a total cost of about $45,000. This system won’t work for the automotive market because its price is not proportional to cost of the vehicle. For locomotives, however, this hardware and software suite pays off, because it serves a different purpose. For example, at its latest conference, Google demonstrated the full range of the sensors, which, according to company representatives, is imperative for every driverless car. This system costs more than $100,000. Like Google, we offer convergent backup: if a subsystem fails, the remaining subsystems are enough to get the switcher to a depot or service station.
Why we started with Russia
Russia is among the world’s top three agricultural machinery users and top five railroad equipment users; it also features one of the best-developed street tram networks in the world. The global market for tractors, combine harvesters, trams and locomotives is about one-third of the global market for automobiles. One. Third. Russia is a perfect market for the development of related technologies because procurement was low in the 1990s, so the equipment in use is modern, with very rare examples of previous-generation models.
The world has a total of 300,000 locomotives, including around 39,000 in the U.S. In Russia, Russian Railways holds about 50,000 locomotives, of which 15,000 are switchers. In addition, 10,000 locomotives are in private ownership. Mostly, they belong to major industrial facilities, such as steel mills. The Novolipetsk Steel Plant, where we conducted some of our tests, for instance, owns 250 locomotives. They serve a specific purpose, ensuring the continuous delivery of raw materials to production facilities and delivery of the end product to the arterial railroad. Such tasks are relevant to many chemical, metallurgic, and coal and mining enterprises.
With fewer than 1,000 locomotives produced each year, we are dealing with an aftermarket, servicing the existing pool of trains. We offer complementary equipment, which is our primary area of expertise.
Accidents within closed territories occur more often than it may seem from the outside. There are hardly ever any victims, but these accidents result in material damage (split switches or derailment) and always disrupt the production flow due to the time needed for repairs.
Our task was to develop a Level 3 autonomous locomotive.
Solution
The technologies used in our tram solutions keep the switcher firmly on the track:
The technologies are also good at detecting obstacles:
Simply using the same code for a switcher as for a tram, however, won’t work. While a tram route is simple (a double track with few junctions), a railroad locomotive has to find its way around stations that feature a complex structure of densely intertwined tracks.
On the upside, switchers don’t travel around the country; they follow a closed route. The first thing we did was create track maps based on track plans using satellite images as an additional source of data (admittedly, satellite images made the process much faster and more reliable). We trained our neural network to detect tracks, but we adjusted the outcomes manually. Here are the schematics we developed:
The switcher is equipped with high-precision GPS + GLONASS, a visual orientation system and a radar. With this combination of inputs, we can use coordinates to identify the approximate section of the route and find the exact location by recognizing nearby objects.
A satellite map comes in handy.
The next thing we did was train the switchers to recognize stations. During each run, the switcher memorizes its surroundings and recognizes objects, classifying them as permanent or temporary (using a canister or a human as a landmark won’t work, but a combination of poles, lights and switches does just fine). Then, the switcher projects them onto the map. As one of the data sources, this considerably improves the accuracy of decisions made by the autopilot. The memorizing algorithm is similar to that of FaceID, with new successful data added to the training dataset.
Naturally, we had to map the datasets manually in the beginning. This was tedious and time-consuming, but essential. We had to build our datasets right at the facility because it didn’t look like anything we found in commercial data banks.
Stations are characterized by bright light and complicated recognition conditions. This is where radar is essential for the detection of pedestrians because video analytics aren’t enough:
Here is how we can enhance low-contrast images (to detect random obstacles on tracks and assess the distance to objects) with the help of stereo vision, which offers even more data for recognizing surroundings in low-light situations:
Since we’re using our own radar, we have an opportunity to collect L3 data from all the sensors and use the neural networks to find more correlations on the go than usual, including those in the source radar scan. This turned out to be an essential capability.
We also use human motion prediction, of course, but this is not a significant factor in these cases since there aren’t many pedestrians in a train yard.
Here is how the switch position is detected: we wait for the switch to appear in our navigation system, engage the low-range camera for a better view, highlight the segment with the switch in the frame and then submit it to the neural network to determine its position.
What makes a switcher different from a regular locomotive?
This is what a switcher looks like:
Here is a better shot:
Such a locomotive is generally used for moving sets of railcars around.
Its distinctive feature is that it can either push or pull. It moves both ways, which means the driver’s cab is at the front only half of the time.
Switching is regulated by plenty of additional rules and a dedicated set of light signals apart from the usual ones.
Whereas a regular running rail line features few switches, a station, where a switcher operates, has a whole network of them. The combination of GPS and RTK corrections is insufficient for stations because there is too much metal equipment around. Anti-aliasing with a Kalman filter cannot guarantee that the switcher stays on the track. There is another track about five feet to the side, and the switcher can position itself there, which leads to mistakes in the algorithm. We used a particle filter for L3 data, which improved positioning accuracy tenfold. A particle filter is demanding in terms of computing, but we can work around it and refrain from setting a specific movement model. On the bright side, we can apply it to data from all the sensors, including odometry, poles at the station, lights and switches; we can also bind it to the virtual map — and it works under bridges, too. The distribution of poles, switch points and lights at the station allows for a much more accurate position estimation than GPS. In the future, we plan to forgo the RTK corrections completely and use GPS only for initial position estimation. Further on, we will refer to the correlation of poles on the map and those seen by the camera or the radar.
Light signals are more complicated than those used by trams. A switcher needs to read the signals correctly. Main lines are equipped with continuous automatic train signaling functions, but these functions are not inducted into rails for moving cars around stations. We needed the night mode for appropriate exposure in a high-contrast setting as the light signals are too bright for the cameras. In daylight, however, they are sometimes too dim. To deal with this challenge, we came up with a system of filters. The autopilot has to recognize lights from a distance of 1,000 feet to be able to stop the train.
A properly mapped satellite map ensures a good result, but the autopilot can also train itself during five or six runs around a new station. This is why, in the absence of a map, we have to make a few trial runs in a new setting. Furthermore, if the map is outdated, our AI will find a way out. Importantly, this is a resource-hungry process because, in addition to recognizing objects, the autopilot analyzes their types. A block attendant can be treated as a permanent object because they are always in the same position, but if the AI classifies them as human, it won’t put them on the map.
An engine driver needs an assistant to have a 180-degree view in the direction of travel because the engine section, which is located at the front in a switcher, blocks the view to the left. The driver either has to work with a human assistant (normally, they aren’t allowed to work alone) or needs a surveillance and autopiloting system.
Once our system has completed the necessary tests and certification, it will be able to replace a human assistant during switching tasks. The system never falls asleep.
Here is one of the best illustrations of what can happen. In this story, a 130-ton heavy duty truck collided with a locomotive. The locomotive lost the fight, but the truck lost its suspension and had its crossmember broken.
Braking was tricky. In a tram, we have flexible control of the process and can perform partial braking. In a switcher, we have to bring the train to a complete halt to avoid a breakaway. A tram can lower its speed and manage the time to collision because of its braking system and constant weight. A train, however, has a different length every time — up to 100 cars at times — and the pneumatic brake takes several seconds to engage. There was another major challenge: if we were to control the brakes of all the cars, we needed to control pressure throughout the train, which went beyond the functionality of our autopilot.
What does the system do specifically?
At present, it initiates automatic braking at light signals — the prohibiting position of the trailing switch — if the switcher exceeds the speed limit when approaching a set of cars or if there is an obstacle on the track (including humans). It calculates the collision forecast, the trajectories of objects and the speed of the train. If someone is crossing the tracks in front of the train, the system issues a warning. If there is no time, it initiates braking automatically.
The general principle:
Our testing ground was in the central Russian city of Vologda, just under 300 miles northeast of Moscow. Safety matters are regarded with utmost seriousness there. A few years ago, an engine driver fell asleep and missed a prohibiting signal, colliding with a set of tank cars. As a result, the locomotive and many of the tank cars were derailed; luckily, the tank cars were empty. The photos of the accident are very impressive.
Had there been any fuel in the tank cars, there wouldn’t be any photos — only ashes from a huge explosion. Exhausted engine drivers miss prohibiting light signals more often than you might think. Alternatively, they could misunderstand the dispatcher’s command on the radio (blue can be a permissive signal if a dispatcher says so). Additionally, if you don’t check the position of the switch, you risk splitting it. In this case, your train will be fine, but the next one traveling in the other direction could be derailed.
The view from the driver’s cab:
As you may see, we’re on a steady path towards turning old Soviet-bloc ChME3 switchers into highly reliable driverless vehicles, or at least towards equipping them with a co-pilot. The latter functionality is already available and operational.
Afterword
In December 2019, we completed a joint project with Russian Railways, developing a hardware and software suite for assistance to engine drivers and presenting a prototype of a switcher with self-driving functionality. We demonstrated our first results to the management of the company last July and received praise. As of today, we have equipped 10 ChME3 switchers with the Cognitive Railway Pilot intelligent system under the existing agreement.
