A source of immense pride: Our robotic combines harvest their first crop

This is what our senior developers were up to last year.

You might remember us telling you how drastically you can improve the performance of a conventional agricultural combine harvester by using neural networks for the recognition of crops and obstacles, and a robot for driving.

Well, all of the above (except NVidia GPUs and some other hardware components) are the result of our own work. What makes us particularly happy is that this year’s harvesting season has come to a close in the southern regions of Russia and our combines demonstrated even better performance than we had anticipated. Long live robots!

This year, we’ve supplied several hundreds of powerful GPUs (for neural networks), video cameras, and hydraulic hoses or CAN modules for steering. Whereas last year our agricultural ADS was a pilot project, now we’re talking about mass production. And our products have been faring well so far.

In fact, they’ve been doing better than we expected, even though we left out some of the features in this release. Although we’ve basically offered only the core so far, it has produced tangible economic benefits nonetheless.

Naturally, there have been a couple of unexpected setbacks along the way. So let’s look at it in a bit more detail, with figures and use cases.

If you look carefully you can see a 2 MP video camera on the top of the cab. An NVIDIA TX2 with a huge radiator and a protective case is mounted under the driver’s cab, and the screen is installed in the cab.

Background information

An agricultural combine is not dissimilar to a church organ in terms of its operational complexity. When a driver works with an assistant, one of them steers (along the crop edge), while the other controls the reel, the fan, the threshing drum, and the harvesting process in general. A third operator can unload the harvested grain on the go, into a truck driving alongside the combine. A fourth operator keeps an eye on obstacles. In Soviet times, there were two operators in a combine crew, but now there is only one. This means choosing between safe driving and efficient harvesting. Since you can’t harvest grain without moving, driving becomes the top priority. Check out our first post to learn about the challenges of the job: how combines regularly harvest humans, run into tractors, and cross paths with utility poles.

Another challenge is the monotonous nature of both tasks: each requires constant vigilance, even with maximum focus. It’s like looking at the road ahead for 10 hours on end every day, with two small incidents occurring a day that require a quick reaction.

The third challenge is that combine operators often choose harvesting speed over efficiency (because they are paid per ton of the crop) instead of ensuring the highest possible crop yield.

Our mass-produced model is capable of autonomous driving along crop edges (the combine monitors the driving process and does the steering) and collision prevention (the combine closely monitors its surroundings and forecasts the trajectory of every object in sight, from people to tractors). This is where the expertise we acquired from the development of a self-driving tram, which we piloted in Moscow, came in handy. A field has much fewer obstacles. Here you can find out more about the video analytics.

A separate team trains neural networks (by taking pictures of situations and mapping the data) to identify specific crops, detect areas of lodged crops, and so on. Since there are no training datasets available, we go out to the fields and take our own photographs. This is important because a crop may look different depending on its variety or on the climatic conditions.

Another team is responsible for hardware design. We have developed an original radar unit for locomotives and trams, but we use only video cameras for combines because the price must be kept as low as possible. Dealing with driving protocols is another major challenge (they sometimes outlive their manufacturers, so we have to reverse-engineer them) and installing hydraulic units to use our signals for steering. Every combine features an autonomous computing unit.

What happened this year?

Instead of engineers and trained test drivers, our combines were operated by ordinary agricultural workers — our initial target audience. These farmers were harvesting real crops. And they gave our system a real road test.

Our greatest fear was that end users (combine drivers) would resist the innovation because they might perceive our system as a threat to their work. Luckily, that was not the case. They understand the limitations of the autopilot, realize they are needed in the cab, and see how the system makes their work easier and what they can do to improve the results. Their crop yield increases, which means they earn more during the harvesting season. This amounts to a considerable increase (about 10–15%), so they are willing to have our robot as an assistant. At one enterprise, they were fighting to get into a combine with a robot.

With our co-pilot, a driver’s workload is very low. A driver starts the system, lets go off the steering wheel and can choose between controlling the machinery and staring at their phone. Harvesting weeks are a real ordeal for combine drivers, who get no rest except for some sleep at night. In one month they need to earn enough for the upcoming six, so they are exhausted. However, the drivers who were using our solution realized they even had some energy left over for housework. They also paid more attention to the technical maintenance of the machinery because they didn’t feel like falling asleep immediately after the shift. Those who chose to work long hours said they could easily work two hours more than usual. They would have worked even longer, but harvesting at night is impossible because of dew.

When drivers from neighboring combines caught a glimpse of these photos, they were eager to see how the solution worked:

“Look at this devil, sitting there in his cab drinking tea! This way, combine drivers will be out of work in no time!” Then they realized the robot’s abilities were limited and the solution simply added a couple of new features to a conventional combine. And they felt reassured.

Here are the facts and figures:

1. Drivers can work longer shifts because they get less tired. Ten or 15 percent may sound like a negligible difference, but there’s a lot more to it than that. In fact, this means that a driver gains three extra days to harvest the crops. Consequently, if there are days of bad weather (showers that cause the grain to germinate or fall down), the probability of keeping the crop yield high is a lot greater.
2. Combine drivers’ workload is much lower. A driver can continuously oversee the combine’s functionality, such as header height and blockage. This work requires attention and skill, and previously drivers were unable to cope well because they had their head turned the other way, monitoring the crop edge. Now operators can make the most of their machinery, which lowers the cost of grain.
3. Drivers can even unload harvested grain on the go, which saves a trip to the edge of the field to empty the grain tank into a truck. The truck can follow the combine as the driver unloads the grain, which reduces downtime and mileage and improves overall performance.
4. Since the combine is controlling the driving mode, our robot safeguards drivers from mistakes. Agricultural enterprise owners say they can easily hire inexperienced operators for the job now. (Traditionally, a combine driver takes three harvesting seasons to get the hang of the job, “killing” an average of 1.6 combines in the process.)
5. The overlaps are narrower too: earlier, drivers preferred to err on the safe side because they tended to leave wide unharvested areas at the end of the shift due to fatigue. By contrast, our robot’s performance remains unchanged at any time of the shift.

Ultimately, both drivers and managers say unanimously that harvesting has become easier. Those who didn’t have grain unloaders are considering buying some. Different enterprises have different daily quotas, varying within the range of 20–25 hectares. We’ve seen management setting the quota at 30, and the drivers took it in stride. Some enterprises have jumped at the opportunity of reducing their pool of combines for the next year and will purchase two or three combines fewer. Strange as it may sound, two enterprises (out of a hundred or so) said this was exactly what they were going to do.

The harvesting season started with Sberbank CEO Herman Gref driving a combine at the Peschanokopskaya Agro Group (one of a large production batch) and saying he mastered it in just three minutes. If the head of Russia’s biggest bank can master it, then agricultural workers will certainly be able to cope.

If a few hundred civil servants run out of work in a certain region, they can quickly be retrained as combine operators. Just think about it. Rusagro has signed a contract for 240 combines. Many enterprises have extended their orders for next year to install the suites on their entire fleet.

A few unexpected setbacks

Indeed, the drivers quickly got the hang of the technology. An enterprise purchased four suites to pilot the solution. We installed them on the combines and performed start-up and commissioning. We couldn’t carry out odometric calibration because it requires daylight, so we decided to put it off till the following morning. In the morning, when the installation team arrived, they found that the drivers had already taken the combines out to the field using the default settings, had started the equipment, and were filming what the robot was doing. They might even have been broadcasting it to other drivers via Instagram. On the one hand, we wanted to complete the calibration; on the other hand, it was nice to see the drivers’ enthusiasm and make sure the default settings worked fine.

After the first few days, many drivers either started to trust in our robot as an almighty intelligence akin to the Terminator or decided to test it to death. At one of the enterprises, the drivers decided to try harvesting at night. Some get the misconception that robots think like humans. They were a little disappointed to see that ours underperform at night. The first operational release doesn’t allow for nocturnal harvesting, as it requires retraining and slightly different data processing algorithms. For now, we guarantee stable performance at night only if the combine’s lighting is wide enough (which is the case with almost any foreign combine three or four years old). In that instance, however, the drivers were operating a 16-year-old Russian combine with a narrow flickering cone of light ahead. Since very few enterprises harvest at night, we decided to postpone this feature until next year.

Another area where expectations were overly optimistic was performance in dusty conditions. When driving in file, combines raise a lot of dust. If there is any wind, the last combine suffers from the dust cloud the most. Since it uses a conventional video camera for orientation instead of an expensive radar unit, the robot cannot see what is ahead. Visibility inside a dust cloud can drop to six meters. In such instances, our system loses vision and signals to the operator to take over the driving. We were even told that our lidar unit had failed, but there is no lidar on combines. Operators would grumble: “How can it not see?” Eventually, this limitation reassured them. A human driver understands that the distance to the combine ahead is about 10 meters and that they are traveling at a constant speed. The dust cloud will blow away in a minute, and everything will be fine. No need to brake. Alex, the driver of the combine ahead, definitely won’t brake. Or will he? Since the system hasn’t spent years alongside Alex and cannot use life experience to predict his actions, it stops the combine and releases the controls. This is where human intelligence once again wins out over AI.

Automatic turns have not been included in this release either. This feature never failed to amaze combine drivers but turned out to be the most challenging during tests: the immense width of the header means that a huge number of hypotheses about objects beyond the line of sight need to be factored in. Each model has its specifics. In addition, it requires a complex driving system with pre-set or pre-programmed routes. We favor user-friendliness: you switch on the robot, drive to the field, and start harvesting. The robot says: “Let me do the driving, man.” You click the second switch, and it takes over. If you need to turn, go ahead — the co-pilot releases the controls and starts looking for a new crop edge. As soon as it finds one, it requests control again. Everything is simple and intuitive. Eventually, we entrusted turning between runs to human drivers. To automate this feature, we’re waiting for the completion of tests on rugged terrain. A run is normally up to five kilometers long, so the turns do not account for more than 1% of a driver’s workload.

We equip combines with only one video camera because keeping the price low is a top priority for us. A second camera is not expensive in itself but drives up the computational load, and the 4 Tflops CPU accounts for a major share of the hardware costs. The camera is positioned on the left, facing the header. There are a handful of exotic set-ups (rarely used in Russia) that require eyes both on the right and on the left. There are two principal methods of harvesting: concentric and in strips. The concentric method implies selecting an area of the field and harvesting it from the edges to the center in circles or rectangles. The combines move counter-clockwise. The second method is similar to printing: the combines enter the field and move back and forth in horizontal lines. Drivers have to make empty runs, but they can harvest an area of any configuration. There are more optimized methods available for fields of a complex shape, which require vision on both sides. We may include the option to switch between the cameras in a later release.

The so-called ‘Canadian’ method.

Now onto the installation process. At the height of summer, amidst the lockdown, everyone suddenly noticed our solution, which had been available since last year. Or rather, they noticed it last year but decided to order our suites right before the harvesting season (this may have to do with the system of short-term loans to agricultural enterprises). As a result, we had to travel around the country during the lockdown, which added an extra angle to the work of neural network experts. At some locations, the installation team had to spend 14 days in quarantine or to wear full biohazard protective gear, but we managed to equip a total of 50 enterprises (and 50 more to go).

We encountered technical challenges as well: some of the foreign combine models (and one Russian model) featured an extremely tight configuration of units. The difference from a conventional set-up was like the difference between the insides of an old Soviet car and a MacBook. Our engineers would send us the measurements from the field, and we hurried to order new bracket mounts or custom mount systems for a particular model.

The pandemic also forced us to change our hardware supplier. We used to order hydraulic units from Germany. In March, the manufacturer told us to come back in four months. This sent us into a panic because those components were crucial, but we found a local manufacturer that quickly reacted to the circumstances and offered us the hydraulic units we needed. They worked quickly but not always smoothly, and task-setting was a nerve-wracking business, of course. Before this year, however, we didn’t believe this could be possible in Russia at all.

What next?

Our robots were gathering grain — wheat, barley, oats, and rye — in southern Russia. The mass-produced solution has not yet been used on corn and sunflowers (the harvesting season for these crops comes later). We are also interested in rapeseed and soy. Rapeseed is grown in Central Russia, so we’re waiting for the harvest there. Soy is grown in Siberia, the Altai, southern Siberia and the Khabarovsk Territory, and will be harvested very soon.

Word of mouth spreads quickly. Over the last month and a half, around a dozen major holdings from the top 50 have approached us with custom requests. Some of them are already purchasing suites for piloting during this harvesting season. Others come with specific terms of reference, with very particular requirements for us to consider in the low season. We have been tasked with including crop yield monitoring (the amount of grain and the combine’s coordinates are included in the telemetry, which means we can monitor soil fertility with a precision of up to one meter) and combine performance monitoring (the transfer of telemetric data to the center). Some of the enterprises want to go fully digital; many need to indicate critical spots on the fields for crop rotation. It is important to know the average yearly crop yield to predict profits for each year. They also need analytics to figure out the approximate workload of tractors and other machinery and to extend or to reduce the park. In agriculture, entrepreneurs have to make various unpleasant advance payments, which include even fuel and lubricants. As the top-ranking manager of a major enterprise said, “We operate in the market. We don’t control it. The only way we can earn more is by cutting costs. If we don’t, our competitors will eat us alive.”

The official service life of a combine is 10–12 years (but we often see machines manufactured in 2005, and even equipped a combine built in 2001). We can upgrade any combine. As long as your horse is alive, you keep riding it. When the cost of repairing a combine eclipses the cost of buying a new one, the enterprise buys a new one. After that, the old combine is stripped for parts, which are used for other old combines. The parts may be worse for wear too, but they’ll do for another year or two. After that, they end up at the junkyard.

Interestingly, the use of our solution also means faster ROI for new combines. A combine may cost over $350,000 though the exact price depends on the manufacturer and model. The purchase of a Russian combine will pay off in five years, while foreign machines pay off in eight. Our solution cuts this period by about a year.

The plan is now to complete the harvesting season with our mass-produced models and a handful of experimental ones, do the math, and publish the results in international economic reviews. If we are successful, our modules will be supplied both in their current form — as additional suites for conventional combines — and as integrated devices in new combines. We’ve made it. It seems that the two and a half years that our team put into this project are gradually changing the world.