Finding 5 billion chickens with human-in-the-loop AI model tuning via Earth Index
Almost all the chickens consumed in the United States come from factory farms. Every year, nine billion of them. That equates to 27 whole birds consumed for every adult, child, and baby in the land. You’d think the country was beset by a plague of chickens. How do you even step foot outside your home without stirring up angry clucking and a cloud of feathers? Where the heck are we hiding these billions of birds?
The short answer is CAFOs: Concentrated Animal Feeding Operations. Massive enclosed chicken sheds.
As to the whereabouts of the CAFOs, public data are lacking. For our food systems and climate emissions research at Earth Genome, we need to know where these poultry operations are located. We are building an environmental search tool, Earth Index, which provides for interactive labeling and analysis of satellite imagery using the latest AI foundation models. We applied Earth Index to the task of locating CAFOs and came back with the most precise available public dataset for poultry CAFOs in the industrial poultry farming belt in the American southeast: 16,372 detections in six states, covering over 5 billion birds raised and slaughtered each year.
The results are being published today on the Plotline. The location data set is available on GitHub. This is an account of the work.
Machine learning like a sauna under the northern lights
I have been working in machine learning for over a decade, mostly on remote sensing of the environment, these days from my home in the northern latitudes of Norway. In that time, object detection has become remarkably more effective, and the systems have become correspondingly more complex to design and train. With power comes headaches: model architectures, data augmentations, hyperparameter permutations, and GPUs. Working to detect CAFOs with Earth Index reminded me of the time before the advent of deep learning, when we compared random forests against support vector machines and counted neurons in the dozens. Except now, building on the hard work of many folks, we can apply simple classifiers atop a vision transformer model pre-trained on a half terabyte of satellite imagery, with all the expressiveness implied.
For this project, I got to focus on model evaluation and tuning, using a custom human-in-the-loop labeling and training interface. It felt easy and light.
Imagine you’ve been working hard all week, but there’s skiing in the evening, 7.5 km on the track outside town, looped twice, and your friend has stoked the fire, and you lean back against the hot boards of the sauna. You feel the heat come and your cares melt out. It felt like that.
The data problem
From satellite, CAFOs look like long, metal-roofed sheds, often longer than a football field, lined up on a farm several abreast. The National Chicken Council estimates that about 20,000 birds fit inside a typical 1500 m² shed. This is the object we want to detect.
Even if someone in a state regulatory body, company, or group of farm workers and neighbors knows the location of any given CAFO, there is a strong public interest in having better aggregated and transparent data. There is so much to say here: Air and water quality. Climate crisis. The bare fact that these animals live crowded together, inside for most of their lives, only to be harvested as adolescents for their meat. Poop. The fate of the feces of hundreds of thousands of animals living in sheds. The life-changing stench enveloping neighbors downwind. The public has a right to know these things. For those of us who ever eat meat, we need to know these things.
Of course CAFOs are subject to regulations to try to manage these harms. And on the other hand, there are hurricanes. I remember watching after Hurricane Florence flooded North Carolina in 2018, as rivers of liquid pig shit washed from the holding ponds and flowed across the landscape, into actual rivers, and into the sea.
Pick a state in the American southeast. Any idea how many poultry CAFOs you will find? A few dozen? A hundred? Some states make data on operating permits readily available. For other states, public data are hard to find. The image above shows the chicken CAFOs in Arkansas mapped on CAFOmaps.org, a project from the University of Iowa to collect publicly available data. We can count about a dozen.
(I am setting up a straw man but mean no slight to the effort that went into CAFOmaps. It has been an important resource.)
A few dozen CAFOs? Let’s see what’s out there.
Cracking open Earth Index
In its core interface, Earth Index invites a user, perhaps a journalist or researcher without programming expertise, to click a few examples of interest on a satellite map, and then to kick off a search for similar objects. If you wish, with another click, you can train and run a random forest classifier to refine the results. Behind the curtain, the search runs over vector embeddings of Sentinel-2 satellite imagery, pre-computed on 32 x 32 pixel patches using an open-source ViT-DINO (vision transformer) model.
I wanted more control over the training and evaluation of the classification heads. Together with my colleague at Earth Genome, Ben Strong, and with an assist from ChatGPT, we shuffled the core components of Earth Index into a new interface built to my specs. We took the embedding vectors, the map, the on-the-fly labeling and training interface, and deployed them in an iPyLeaflet map in a Jupyter notebook.
We had access to an older, but mostly on-point, labeled dataset of CAFOs in North Carolina, from the RegLab at Stanford University. My plan was to use the labeled data to train a small neural net (a multilayer perceptron) on top of the Sentinel-2 embeddings, sample new labeled data against the model errors, retune the model, extrapolate to the new geographies, and repeat. It amounted to about a week of work.
The first model loved solar farms. They are long, linear, row-like structures. Of course. The RegLab built their dataset on 2014–2016 aerial imagery from the National Agriculture Imagery Program (NAIP), when solar farms were a glimmer in a child’s eye. I needed to add samples of the new solar installations to the negative class. Other false positives modes included isolated box stores, sections of interstate freeways, and certain swine CAFOs, when the patch boundary separated the buildings from the tell-tale pig feces lagoon.
In the past, I would have exported model predictions, loaded to QGIS, marked new data points, and pulled relevant image patches back in the training environment. Here, training and labeling were integrated in the Jupyter notebook. I clicked new positive and negative points on the map, merged the new and old vector datasets, and clicked to train the model for another 10 epochs. By another button, the new detections appear on the map.
Through a couple of rounds of labeling and retraining for North Carolina, I landed on a model with accuracy 0.98 and area under the Receiver-operator-characteristic (ROC) curve of 0.99, evaluated on a reserved validation set. With 242 new points added to the original 15,655 samples, the model learned to classify the fail modes while maintaining its baseline performance. The RegLab labels don’t align with our image tiling scheme, and given that our goal is to find CAFOs, we can count a success when the model identifies a CAFO in the patch with nearest centroid to the label or in any neighboring patch. With that criterion, 94 out of every 100 detections are good detections. 82 out of 100 labeled CAFOs are captured by the model. I was very pleased with this performance for a real-world application. At the expense of missing a few buildings, we can be confident that the vast majority of affirmative detections we see are actually CAFOs.
For comparison, in their study, the RegLab used 1-meter resolution, 4-band NAIP aerial imagery. We are using 10-meter resolution, 12-band Sentinel-2 satellite imagery, in which a CAFO is only a couple of pixels wide. As we have seen in other contexts, the spectral breadth of the Sentinel-2 sensor gives it surprising robust detection ability at the limit of its spatial resolution. Although it would require some adjustments to make a one-to-one comparison, our model performance in North Carolina looks competitive with RegLab’s custom convolutional neural network on the higher resolution data.
To extrapolate to the other states, I had to account for new types of terrain, like the Mississippi River valley in Arkansas, with its large row-crop farms, and the southern lowlands of Georgia, with larger, drier parcels and center-pivot irrigation. These lit up with false positives when applying the North Carolina model. I repeated the tuning process and watched the false positives melt away with a few hundred fresh data samples and tens of epochs of retraining.
Putting CAFOs on the map
There are more than a few dozen. Our models make 16,372 CAFO detections across the six states of this study. Whole farming regions in these states have been given over to industrial animal production. It’s worth scrolling around the live map to see how far CAFOs have come to dominate the landscape.
Because large facilities often straddle multiple neighboring image patches, I merged adjacent detections. In regions dense with CAFOs, that sometimes means counting multiple CAFOs as one. Also there are a handful of false positives. The number of detections doesn’t equal the number of CAFO operations. But the number of detections are indicative of the scale of CAFO farming: thousands of operations in each of the states. From the USDA, we know that in 2022 there were on the order of a billion broiler chickens produced in each of these states. The numbers are staggering.
With the public data from CAFOmaps next to a map of our detections, we readily see the role direct sensing could play inpublic-interest efforts to understand this growing and sometimes secretive industry.
A remote sensing project can flounder in the transition from local proof-of-concept to a broad-scale monitoring effort. Take as case in point the ubiquitous false positives generated by another CAFO detection model when extrapolated directly from the Delmarva Peninsula to the whole of the United States. I don’t know of any shortcut for sampling new geographies. It can be tedious work. The human-in-the-loop approach used here feels like a step in the right direction. By iteratively training and viewing model outputs, and by adjusting new samples accordingly, I found that it only required hundreds of data points to tune a state model to work well on a neighboring state. For the multilayer perceptron classification heads, retraining is effectively instantaneous.
Do you need to find things on the surface of our planet, whether a component of our food system or any other environmental impact? We’d love to hear from you about your requirements. You can also sign up for the Earth Index waitlist to access the tool as we release more widely.
