Cognitive Agro Data Factory: The world’s largest ‘neural university’
I’ll start with a revolutionary statement: by inserting the C-Pilot artificial brain into agricultural machinery, we become a creator, something akin to God. We turn an object into a thinking and reasoning being; that is, a combine harvester equipped with Cognitive Agro Pilot starts seeing and understanding its surroundings and making decisions about further actions according to the operational task at hand. In a way, we are creating a new social stratum of rural workers — C-Pilot AI-based agrobots, which analyze and complete agricultural tasks set by humans.
In fact, we are witnessing the emergence of a new community that requires massive, consistent education. Humanity had thousands of years to develop an evolutionary layer of consciousness, while robots only need months. What they also need, however, is a suitable environment: a major factory for artificial brain training and data preparation. In this publication, we will offer you a sneak peek of Cognitive Data Factory, a combine that harvests and processes data for the agricultural industry.
Your kid’s textbooks and teachers have a definitive influence on their development and future career. The same goes for the automotive industry, where quality data and its correct mapping are of primary importance for developers of AI for driverless vehicles and other highly automated control systems. Cognitive Pilot solutions are training at our unique Data Factory. So how does it work?
Video data is the “food for thought” necessary for the growth and training of neural networks, which form the core of any artificial brain. When Elon Musk says that Tesla’s brains benefit from a daily inflow of data collected from 35,000 cars in the cloud, it’s hardly an exaggeration; this is Tesla’s main differentiator, and sets the manufacturer apart from Mercedes, for example. When Zoox announces it needs another half a billion for thousands of cars to drive around gathering data, investors take it very seriously. At Cognitive, we started preparing for all of the above in 2012. Today we proudly present one of the world’s best data factories.
An in-house team is better than outsourcers
Data classification and mapping processes are almost identical for our agricultural and rail transport projects, or for our self-driving tram. We decided against setting up structures with duplicate functionality, so these functions are performed by a single Cognitive Pilot department, which supports internal service for all of our projects. A team of qualified professionals is more efficient than disparate specialists whose resources are spread too thinly across the working schedule of multiple departments. We’ve also set up a common repository, where every company employee can gain access to data collected by adjacent departments.
Data wrangling is often outsourced, but we’re against hiring third-party contractors for such a crucial task. A close-knit team allows for fast onboarding of new members, who can always seek the advice of their more experienced peers. Core team members are selected more rigorously and stay at the company for many years. Engaging outsourcers results in high personnel turnover and higher training costs, and whenever a freelancer leaves, we lose their expertise irrevocably. Furthermore, the company’s office is equipped with hardware that meets corporate requirements, while there’s no telling what a freelancer might use. In our experience, the same specialist performs much better at the office. We should also bear in mind the risk of data leakages, since data is mapped using proprietary tools and is commercially sensitive (especially in its processed form).
We’ve mastered the combine, now for the tractor
It’s best to explain the principles of the Data Factory using the example of our agricultural project. We’ve released an ADS for combine harvesters that uses video cameras, a geopositioning device, and travel speed and steering angle sensors. Now it’s the turn of the tractor: our engineers are gathering data in several regions of Russia, in fields belonging to those of our agricultural partners who are interested in making use of modern solutions. Filming began as early as last year because the machinery is engaged in seasonal work: cultivation, sowing, fertilizer application, weeding, spraying, and so on. We’ve had to install several video cameras on each vehicle for multi-angle filming, but this doesn’t affect the operators, whose routine has remained unchanged.
Our current objective is to gather as much data as possible, in order to gain insights into the specifics of agricultural tasks performed by manually controlled machinery. The same tractor is used with various trailers in fields with different crops. Its course depends on the operation performed, so all videos are different. We have to keep moving the cameras and changing the angle; since it makes no sense to waste an entire day filming one type of work, we try to include as many options as possible.
The main challenge is to distinguish between processed and unprocessed areas in order to identify the crop edge and subsequently train the ADS to detect it. The task is complicated, as the differences aren’t always apparent even to the human eye if the tractor is applying fertilizer or sowing seeds. It’s also crucial to correctly identify obstacles, such as people, other vehicles, transmission towers, or trees. We often have to mount cameras not only on the tractor but also on trailers. Cultivation is one of the most challenging types of work. It is normally done in dry weather, with the tractor raising a cloud of dust. Two runs across the field make the lens too dirty to see anything, so we have to install the camera in the cab. We encounter countless problems when filming, but they are all manageable. We’ve mastered the combine, we’ll master the tractor.
Storage and processing
We accumulate the data gathered in the field in a dedicated computer room — the repository. Backup is a must because the data is far too valuable to lose due to a system failure. Incoming data is viewed by human operators who select useful fragments. The volume of data uploaded to the Data Factory from our agricultural projects is large, so our field employees help with the selection too. We often have to discard irrelevant fragments, such as footage of breaks. For easier navigation, we rename selected files according to our internal rules and categorize them by placing them on so-called stands. We can also see which data hasn’t been cut yet if there are video streams we haven’t yet processed. The Data Factory enables us to take stock of the objects we have already mapped and gather mapping statistics, such as average speed, on a stand or as an episode. We also store other types of internal data selected against a range of criteria. Looking at the map, we can see where the video was shot, which sensors were used in the recording, etc.
We use completed stands to create training and test datasets. Depending on the task, we may not use all the frames, since agricultural machinery driving automation often requires only one frame per second, which makes processing much easier. Apart from our agricultural project, the Data Factory also services our other projects, each with its own tasks and requirements.
Field theory
The selected videos are mapped according to the task at hand. For instance, we may need to classify all the pixels of an image against preset categories or to identify the line of the crop edge. If we add a new type of task or a new crop, we don’t have neural networks trained specifically for the processing of such videos, so we have to perform the initial frame mapping manually. Subsequent stages include automated preprocessing, which makes the task much easier. We use original in-house tools for mapping and improve them continuously. These tools are also developed by the Data Factory team, which includes both mappers and developers; though in reality there is no rigid distribution of functions at Cognitive Pilot.
When making training datasets, we focus on including the widest possible variety of scenarios because quality here is more important than quantity. Prepared datasets are submitted for automated postprocessing, which is overseen by specific project teams and not Data Factory engineers. In agriculture, for instance, our specialists often have to tackle the task of real-time conceptual image segmentation. To that end, they often use artificial neural networks built with the help of deep learning mechanisms. Each pixel of an image is associated with a finite set of possible states: reaped, unreaped, combine header, windrow, crop rows, and other classes. The network architecture maintains the right balance between the speed and productivity of segmentation in datasets. It suggests methods of data processing for various volumes of data with a subsequent combination of the outputs of ResNet (Residual Network) and the spatial resolution branch.
How we train combines
Once we have a segmentation map, we need to determine the path for the combine to follow (for tractors we are yet to reach this stage). There are multiple ways of tackling this task depending on work mode and crop type. We differentiate between three modes: harvesting along the crop edge, a windrow, and a crop row.
Harvesting along crop edges
In this mode, a combine moves across the field, maintaining the necessary cutting width. The results produced by semantic segmentation enable us to detect the border between “reaped areas” and “unharvested crops”. Using its system of coordinates, the robot plans a course, taking into account the combine’s header width, crop height, sensor data, previous results, and other criteria.
Harvesting along windrows
A windrow is a row of cut grain crop lying on the field. In this mode, a combine with a mounted windrow pickup has to move along windrows. The task is similar to the previous one, but instead of moving along the border between two classes, the combine has to stay in the middle of the windrow segment.
Harvesting along crop rows
Corn is harvested with special headers for row cultures. To that end, the robot has a mode in which the combine attempts to keep the header in the middle between rows. With the frame segmentation map and data on the header position, we can identify the so-called “vanishing point” and calculate the deviation of the header tooth from the necessary position.
How many fields, crops, and weather types do we store?
Our ambitious agricultural odyssey was launched personally by Vladimir Yalovenko (whom we cannot thank enough), who offered us the use of his agricultural enterprise late last year as an industrial testing field for the scaling of our Data Factory and the innovative expansion of our work to include tractors. The subsequent contract with Rusagro for the robot automation of 242 combines across the entire country boosted the volume of data gathered in the field to Zoox-like scope and ambition. We started getting inquiries and offers from Western partners because this database is truly unique — there is nothing like it in the entire world. Our operation reached a truly global scale when we concluded a service agreement with EcoNiva for the innovative servicing and installation of 10,000 devices over three years in regions all over Russia, from the Smolensk Region in the west to the Irkutsk Region in the east, from the Krasnodar Territory in the south to the Komi Republic in the north. How about that, Elon Musk?
This is how it works in the field.
In the fields belonging to the Peschanokopskaya Agro Group, grain harvesting begins with two mowing runs along the perimeter (with the regular cutting width of nine meters) for fire safety. Then the combines mow an area for machinery and equipment. The remaining field is divided into so-called “runs”: areas that are reaped by combines in circular motions.
A computer can divide the field into runs in a way that best fits the parameters of the header, so that combines don’t have to harvest narrow strips. Besides, it’s hard for an operator to follow a crop edge manually while controlling the speed and other machinery settings. Work like this requires constant focus and numerous adjustments, which leads to fatigue and mistakes. Harvesting is a round-the-clock process, and driving a combine in the evening or at night is particularly hard, so operators grow tired even faster. The ADS takes over a multitude of functions, making an operator’s job much easier and boosting the efficiency of the advanced machinery.
We’ve had to delay implementation a little due to a drought in the south of Russia and the cold spell that followed. Now, after a rainy period, the wheat that’s still alive is growing quickly. Harvesting will most likely be postponed, and we’ll be ready to discuss the results of our project closer to the fall. It’s worth mentioning that, with the client’s consent, we can use the mobile internet connection to upload data from an operating ADS to the Data Factory. At the moment, it’s receiving images from eight combines owned by a client we supplied with our system last year. We’ll shortly start receiving data from the combines of the Peschanokopskaya Agro Group, which is necessary for our tech support to be able to respond quickly to user requests about ADS malfunctioning. For security reasons, only Cognitive Pilot employees can access the data.
Feedback
Data Factory is not just a repository of raw data or stands of categorized information; it also holds ready-to-use (mapped) datasets. As we’ve mentioned, apart from training datasets there are test sets. If engineers in a new testing field realize that the system is not functioning as planned, they can use these sets to retrain it. Sometimes Data Factory engineers have to remap training datasets according to test results — by adding new classes, for instance. We don’t change the existing mapping, however, provided it’s free from any critical errors. We also monitor the availability of open datasets relevant to our tasks, and if we find any, we make sure to use them in our work.
Cognitive Agro Dataset
Until recently, there were open datasets available for automotive projects, while the agricultural industry could only use standalone dataset combinations for specific tasks like spraying fields with herbicides. We decided to remedy the situation by sharing our resources with the community. In Q1 2020, we launched the preparation of Cognitive Agro Dataset, a comprehensive database gathered in real-world conditions to train neural networks for agricultural machinery automation. Cognitive Agro Dataset will include images from video cameras and data from odometric and inertial sensors installed on self-driving vehicles. The data is gathered in a variety of settings: different test field geometries, crops, and types of work.
We’re currently in the process of mapping the main objects of the field scenario, such as machinery and equipment, processed and unprocessed areas, crop rows, windrows, humans, and objects of other classes. Data Factory engineers are directly involved in the project, with completion scheduled for the beginning of next year. We plan to extend the database by adding data on more crops, videos obtained with the help of stereo cameras, and data recorded with IMUs (Inertial Measurement Units).
A common cause
Some of the datasets will be made available to developers free of charge, hopefully giving a powerful boost to the development of autonomous agricultural machinery.
