Why it is necessary to robotize agricultural combine harvesters, what the difficulties are, how we did it in 2 years
Combine harvester operators around the globe look forward to the start of the harvest season — a time when they can make several months wages in a few weeks. Usually there are just two weeks in which all harvesting is to be completed, the consummation of an annual cycle of investment in labor, fertilizer, fuel, etc. Work starts at 8 in the morning — it takes an hour to set up the machines, so harvesting proper begins at 9 — and lasts till nightfall, because the dew and the humid air at night are bad for the grain. The pace of work is relentless, so problems start cropping up as soon as the third or fourth day, with grain being chopped wrong and machines breaking down.
A layman in agriculture might think that the task facing the combine operator is simply to drive around the field in a serpentine pattern, mowing down all the wheat or whatever crop it is filled with. But reality is far different. There are hundreds of things an operator must watch even as he keeps his eyes fastened to the edge of the field to ensure that he moves alongside it with fine precision. Imagine driving for 12 or even 14 hours straight at 75 mph behind another vehicle that slows down abruptly every half hour, and you can get an idea what it feels like to be a combine operator; despite the mind-numbing monotony of such work, they must somehow remain constantly alert and prepared for a surprise.
Basically, a combiner must juggle two taxing jobs, controlling grain processing and steering the harvester. For one person, this can easily get overwhelming. Because diverting too much attention from steering would entail the risk of ingesting a stone — or, worse still, a human — it is the harvesting process that tends to suffer.
And yet part of the operator’s job is easily automated. Let me give you a brief tour of what we’ve done to make this a reality and how we install our robotics solutions even on fairly old machines.
Turning
So, our combine operator is in for two weeks of exhausting labor. The pay consists of two parts: their regular salary and a bonus for the volume of grain collected, which can add up to several times their regular wage.
At the same time, the agronomist oversees the combiners’ work to ensure the thoroughness of their harvest. Finishing off any areas of uncut stalks missed by the harvesters’ blades through poor turning would mean wasted costs in fuel, wear and tear on the combines, and machine- and labor-time. As a rule, such areas are finalized in the morning, and the combine’s efficiency is very low while doing so. The whole business efficiency of the farm depends very much on reducing the cost of harvesting: they cannot influence the market, but they can reduce their cost of grain. This has several implications:
1. The combine paths are designed in such a way as to ensure continuous movement for full coverage of the field.
2. Turning is the trickiest part of operating a combine, as it involves bringing around the 30-foot wide header at its front. The closest analogy is that it’s like driving with the wheels lined up to the lane markings. You can do it well for half an hour, but for the whole day — drifting is certain. And once you drift, you’ve left an uncut piece. That’s why the drivers pay close attention to the edges of the reaper and the field. Any uncultivated area is a cost: more wear and tear, more tractor runs, and most importantly — more time. But, if in the morning the size of uncut patches is from 10 to 20 inches, by evening it can get as large as 3 feet. We’ll go into the impact of that in a moment.
3. So, it is essential to ensure that the combine is steered with precision. We can already perform fine automated steering along the edge with our robot — this accounts for 90% of the combine operator’s time and the most unpleasant part of the driving — and are currently testing the turning capability.
As we noted above, due to fatigue at the end of the shift, the operator leaves up to 3 feet of the header length uncut. With a speed of about six mph, in four hours the combine harvester will miss about four hectares. To clean the remaining area during the next shift will take an additional three hours of work — or an extra day for each four days of work. The steering errors account for a 25% increase in the harvesting time. In addition to the time lost to rework, there’s also the wasted fuel. At a fuel consumption rate of 5 gallons per hectare, the direct loss in fuel (which will be required to reharvest the lost areas) will be: 4 ha x 5 gallons x $2.40 / gallon = $48 / per shift. Over a 20-day period, the wasted fuel adds up to $960 for just one harvester. A fleet of 10 combines brings that total up to $9,600.
This video is from a demonstration of our robot to a group of farmers. A combine operator films the machine’s performance as it drives. It is quite an experience to see the future suddenly materializing in your field:
Another thing worth mentioning is that agricultural mechanics are often hired as temporary labor. Neither the farm’s owner nor its agronomist knows them personally or has any record of their previous employment — they understand that the workers are likely to just zoom around the field to capture the largest bonus. The agronomist needs our system to safeguard the machines.
Collisions
According to our data, a typical farm averages one or two serious collision accidents per year. Several harvesters are engaged in a field at the same time, not just one. While driving a harvester through a field, you may collide with a another harvester, a tractor, a tree, an unexpected power pole, a human (perhaps a fellow combiner who has gone to the bathroom), a cow, or other obstacles.
In one incident we know of, a combine operator absorbed in watching the edge of his swath was startled to see something red flying out from his combine. A person had been in the field and was caught in the harvester’s teeth. As it happened, the operator was cleared; the police determined that the victim had already been dead, apparently murdered and dumped in the field. But, though cleared of charges, the unfortunate combiner had to seek psychiatric help. Another typical scenario: a combine driver pulls over to fuel up and runs over another driver whom he doesn’t know from Adam.
Clearly, if something is just hidden in the straw, collision is almost unavoidable. Which is why fields are kept free of boulders; they are removed before sowing, while smaller stones are filtered out by combines’ stone traps. The operator is perched fairly high, and they can easily spot most obstacles. It all comes down to which way they happen to be looking (e.g., at the edge of the row, at the control panel, or straight ahead) and how fast they can react if something happens.
In my experience, most of the damage is from chewing up trunks and breaking the cutting bar. Frequently, damage is caused by unsteady driving of the combine at the head of a group, e.g., in a wedge formation.
Our robot ensures that there is always a safe distance in front of the harvester.
Cognitive Pilot uses a video camera mounted inside the operator’s cabin to spot and recognize obstacles. The robot understands that trees and poles are stationary, while humans, cows, and tractors are not, so it tracks their speed and direction. When the operator’s involvement is required, an alert is produced. In a critical situation, the combine is brought to a halt immediately.
We can thus save a few days of downtime that would otherwise be lost to repairs, which is especially important during harvest time as it would prevent the loss of considerable quantities of grain as well as save the cost of spare parts.
Speed and other settings
What parameters does an operator control? One is the rotation speed of the pickup reel (a device like a screw that bends the straw into the cutter). That speed depends on the crop — the speed for corn is not the same as for wheat, for example — and its maturity.
The specifics of the combine harvester’s work make it a difficult to convert their tasks into algorithms with purely empirical assumptions. That being said, the combine has an almost complete log of its actions, and it’s possible to estimate the percentage of crop lost on account of using incorrect operating procedures. There is also a need for input data on humidity; crop type, variety, and maturity; and locality (the size and appearance of the same type of crop can vary widely depending on the soil and climate of the area where it grows). If you combine all this, you can get either a neural network trained to perfection over twenty years in the head of a combine operator, or in a robot able to perform the same job.
The skilled combine operator is a dying breed. Professional education has declined, and the young specialists joining the labor force aren’t up to the same standard. Though the same can be said of most manual trades, this effect creates a great demand for our robot.
So, if the pickup reel rotates too fast, grain will be knocked out and end up on the ground instead of being pulled into the harvester’s processing mechanism. The may cause losses from three to ten percent of the processed grain volume. On the other hand, if rotation is too slow stalks won’t be bent far enough, again causing some grain to spill to the ground.
If grain is not fully mature, if the kernels are already large but still sit firmly in the ear, the kernels can’t be separated from the ear. But, if it is just a tad overripe, hitting the stalk would make the grain tumble to the ground. The settings need to be adjusted for different parts of the field, depending on whether the crop is lodged or standing upright. The same applies for undulating ground, as depressions trap more moisture than hills, causing stalks to reach higher.
The driver must watch the edge while managing both the driving speed and the rotation of the pickup reel to the particular conditions of the patch one is passing through. Lodged crop normally requires speeds ranging from 0.5 to 1.5 mph, while normal crop can be gathered at a rate of 4–5 mph.
And, the combine operator gets paid by harvested volume.
There are several combine operators assigned to a field, and there is a limited volume of crop available for harvest. The operators aren’t there to save money for the farm, their goal is to collect the most grain to make the most money. Instead of managing their speed and the dozens of other relevant parameters for the specific crop they are gathering as best as they can, their performance is governed only by the threat of not being allowed to work the farm next season. In other words, they would try to work at top speed, limited only by the ire of the manager.
In practice, this means that the efficiency of grain harvesting is far from optimal. Even though an old farm hand may be able to show excellent results at the start of a shift, performance is apt to decline as the day progresses. For younger workers, it is all about making volumes, so their efficiency is low. Once harvest season is over, the first rain makes the outcome plain to see, with bands of fresh stalks sprouting from spilled grain: the evidence of the waste.
Processing of the stalks
The next type of settings relate to what takes place inside the combine. A conveyor system transports stalks from the header platform to the separating section. Though its design can vary, typically this consists of two drums with sieves underneath. The drums thresh grain out of the stalks by beating on them, making the grain drop into the sieves.
It is very important to get adjust the speed and threshing mode of the drum to the volume of the crop, the size and density of the ears and grain, and their maturity. Also, as a general rule, the riper the grain, the less threshing it needs.
In fact, this means that the settings should be changed before starting on each section of the field. In my experience, however, no one does that for fear of messing up the settings or because it is too much trouble.
Modern harvesters are fitted with sensor systems that display the processing status on a monitor. Old harvesters have no such convenience. The times when you had to twist something physical to change the machine’s operational modes are over. Now you can adjust the settings at any time. And yet in my twelve years of sales experience, it has been the rule rather than the exception that the dealer sets up the harvester right after sale, writes down the settings, and that’s it. Out of the 200 harvesters we have helped retire, there have been only three whose actual settings differed from the ones the dealer set up.
That means that there are losses because workers skip the settings adjustments. We’ve seen people ignore alerts that warned “slow down to lower loss” or “not enough threshing: adjust concave.” “That’s how our grandfathers worked, what’s the use of me monkeying with the settings? If I screw up, I could be idle for the entire season.” They end up just concentrating on driving.
The next stage is to collect the grain still attached to the straw. There are both rotary and key mechanisms for that, the latter being more popular. Straw is pulled through a key mechanism with a separator below. Cleaning and harvesting the ends with compressed air blowing the light pieces of straw and leaves to one side while the grain falls into the hopper. The force of the air needs to be tightly controlled based on the crop, but, fortunately, it can be set once in the morning before the shift starts. For rapeseed with its light seeds, the air must blow gently, but corn needs a much higher airflow.
in practice it looks like this: an old farm hand comes over, takes a stalk, rubs the grain in his hands, and then adjusts the air pressure to match the site. The rest of the operators come over, look at what the old-timer did, and copy the settings.
Working with a tractor
Harvesting can be done either by dumping grain into a truck that drives next to the combine in an already mown area or by calling a tractor from time to time to unload the harvested crop from the combine.
When operating in tandem with a grain truck, the operator must make sure that the grain ends up in its truck’s tank. This means the driver has to pay attention to two directions now: eyes on the control panel as well as forward.
When calling for a tractor, though, it gets more difficult. The operator must monitor the level of grain in the combine’s tank and also figure out the best way for the tractor to approach, which is a challenging optimization problem. It is known that people tackle such problems using imperfect greedy algorithms, which in this case results in wasted fuel. Fortunately, this is the least of the problems on the field, so we ignore this for now.
Things get more interesting when combines work together in a wedge rather than in their own areas. Each driver has to watch out for the other combines as well.
Points of refinement
What a robot can do
One of the hardest parts of the combine operator’s work is keeping the combine moving in the right direction — steering. The previous generation of automation systems used GPS to automate steering. Here’s why that solution is unsatisfactory. Remember, the steering tolerance is very tight, less than 10 inches. This can be tough, because without dynamic guidance via the Internet one needs a ground station with a 10-mile radius of coverage. A station like this is expensive and still needs subscription-based signal bundles to make it operational. When it comes to precision movements, GPS-guided harvesters leave much to be desired. Under ideal conditions, they can clear a field themselves, but they do not monitor the surrounding situation and can’t make difficult turns. If the GPS signal is disrupted, they can leave uncut patches behind.
We first started thinking about combining GPS with visual data analysis, but it didn’t take us long to realize that visual analytics alone is enough.
As of today, our automated combine can follow a planned route around a field. It also recognizes obstacles in its course and stops before them, and it coordinates its movement with a grain truck and with other combines when it is part of a wedge.
Using GPS requires a lot of coding to create a navigational map. It’s a complex approach, and complex approaches don’t work well in a field. We made it simpler. Our autopilot thinks, “Look, it’s a field!” and asks for permission to take over the operation. That’s it.
With the robot steering, the operator can focus on the harvest and adjusting the process to the specific features of the crop.
Operating regimes
In additional to the combine, our robotic co-pilot can be fitted on a tractor, a mower, or a sprayer.
Now you have a rough idea of what it takes to get bread, corn, spaghetti, etc. to your table, and the kinds of kinks that have to be straightened out in the process. I’ll stop here for now because my specialty is working with agricultural equipment, not with neural networks. In the next post, my colleague from the development side will tell you exactly how it all works and what difficulties we hit while implementing our robot combine driver over the last two years.
The bottom line is that our robot can shave 2.5–5 percent off the cost of grain. This is an estimate, not a proper A/B test (which is rather tricky to perform, because no two parcels of land are identical, any more than any two combine operators are, and two years is not enough for that kind of analysis), but if you are interested, We can share how we calculated it..
P.S. If your agronomist is interested, but he does not read Medium, you can contact us through cognitivepilot.com and we can discuss in detail your kind of combine, what type of equipment you need, how much it costs, and how you can quickly see and test it.
