FireMind: Lessons Learned

FireMind succeeded, in that I successfully created a neural network and trained it on real world fire data from complex and different fire sources, and did it from scratch in the space of three weeks. Yet the most important test, whether fires would behave in a plausible way, failed. The probability blooms of the simulated fire front did not trace canyons or move as pushed by the wind.

I’ve come up for three categories of reasons why this project didn’t match a full fledged simulation.

Machine learning architecture is not easy

1. Hidden classes imbalance issues: A touchstone of categorization models is that classes should be balanced, otherwise the model will train to the most common class. This is especially important in cases where positive results are rare, and false negatives carry a high burden, such as cancer screening. While I had a relatively good balance of ‘on-fire’ and ‘not-on-fire’ pixels in the data, most of the 71,000 fire progress tensors had relatively static fire progression. The most important case, and one not well represented in the tensors, is when the fire advances rapidly. My model trained well on the 90% of situations where a fire perimeter remains relatively static.
2. Vanishing gradients and deep neural networks: Deep neural networks train on backpropogation, where the new weights are proportional to the error as filtered through the derivative chain rule on each intermediate layer. I doubt the CNN and LSTM were getting much usable feedback out of the perceptron. With more time, I would have experimented with alternative architectures.
3. Poorly specified manifold space: The model trained on fire perimeters, a binary array of size 256 * 256. There are approximately 2*10¹⁹⁷³⁰ possible arrays, a ludicrously large number. Yet actual fire perimeters are not random noise, they are discrete areas. If I’d read Goodfellow et al’s Deep Learning before doing this project, I’d have through more about what a plausible output space for the model is.

A plane drops fire retardant behind homes along McVicker Canyon Park Road in Lake Elsinore as fire burns near homes. Carr Fire. AP Photos.

Data quality and access was not as good as it could be

While I have to thank the hardworking scientists and administrators responsible for the data sources I used, these data sources were created for programs with goals that did not include machine learning. Detailed data, on a minute and meter resolution, collected from a unified sensor suite rather than assembled after the fact, would improve this project.

1. I couldn’t find public data on fire fighter activity, which meant that crews, trucks, and airdrops weren’t included in the model. There may be situations where the model sees the fire halt for no apparent reason, but in the real world there are thousands of brave men and women making sure the fire doesn’t breach a line.
2. Time and spatial resolution were larger than I would have liked. 30m x 30m is a big chunk of land. The 24 hours between perimeters meant that I couldn’t capture rapid advances in the timescales in which they happened. Worse, it turns out that perimeter timestamps are accurate to no more than the day, so my weather data could be off from the perimeters by hours in either direction. And while final fire perimeters are very accurate, interim fire perimeters may represent an approximation of what is actually burning.
3. Structures were not included. Gregory Simon, in Flame and Fortune in the American West, describes the modern home as a 2500 square foot duraflame log of plastics and wood chips. Structures burn hotter and faster than trees, and present challenges for firefighters in the wildland-urban interface. For a real world use case, we’re much more concerned with the flammability of densely inhabited regions than remote wilderness without human presence. Structures must be included in the model.

The problem might be infeasible

A fire tornado veers close to homes during the Freeway Complex fire on Nov. 15, 2008, in Yorba Linda, Calif. Strong Santa Ana winds helped the fire to destroy more than 30,000 acres and 314 residences. DAVID MCNEW/GETTY IMAGES

The essence of wildfire is its wildness. While the Ten Standard Firefighting Orders and 18 Watch Out Situations are rules that keep firefighters alive, they do not tame fire. A flame front can burn hot enough to melt metal, while leaving grass patches a few dozen feet away untouched. Wildfire is unpredictable, complex and chaotic in both the mathematical and ordinary senses of those words.

I started this project with the belief that emergent properties of fire could be captured by machine learning. I still believe that to be true, even if this specific approach did not capture those emergent properties. But respect for the flames means that we must always consider the alternative, that this phenomenon can’t be meaningfully approximated.

Of course, one person and one project done in three weeks should never be considered a formal verdict. As the New York Times summarized in an article, Can ‘Big Data’ Help Fight Big Fires? Firefighters Are Betting on It, the WIFIRE lab at UC San Diego is having success with an approach that seems broadly similar to FireMind. I wish them all the best!