When adaptability breaks machine learning models
What Facebook could learn from New Zealand mud snails
I lied in my previous post — or rather oversimplified. You don’t really need creativity to break a machine learning model. In fact, any adaptive system is capable of breaking a model of its behavior. I will illustrate this point with the example of cancer evolving to escape treatments.
I also previously said that creative model-breaking behavior was most likely under certain constellations of incentives. Also a bit of a lie. In the case of adaptive systems, fast evolutionary change is most likely under certain constellations of selective pressures, such as parasite-host relationships. New Zealand mud snails demonstrate this principle by evolving faster when parasites are present.
The lesson for machine learning stakeholders: when a model’s output exerts selective pressure on an adaptive system, the model’s success will be short-lived. To keep up, model owners will need to change the model — and change it again and again.
Doctors chase the evolution of cancer
Let me start with cancer. Cancer is different from most other maladies, like diabetes or heart failure. Cancer can evolve to avoid treatment.
An example can clarify. In chronic myeloid leukemia (CML), patients develop a genetic mutation called the Philadelphia mutation. Researchers sought a drug that would inhibit a protein in the Philadelphia mutation and found imatinib, later branded Gleevec in the US and Glivec internationally [1]. This drug was so effective in blocking proliferation of the Philadelphia mutation that oncologist Brian Druker gushed, “These once-dying patients were getting out of bed, dancing, going hiking, doing yoga. The drug was amazing” [2].
Unfortunately, over time some patients with the Philadelphia mutation can develop various secondary mutations that lack the cleft where Gleevec binds but still spur cancerous growth [3]. Thus, the cancer relapses.
Gleevec was discovered the traditional way, but even if it had been discovered with machine learning [4], a cancer can evolve to avoid its effect. Before Gleevec, patients almost always died before secondary mutations could happen, so there was no data on these secondary mutations for a machine learning model to train on. The use of Gleevec created selection pressure that made secondary mutations more common. The cancer cells that had the secondary mutation could proliferate while those with just the Philadelphia mutation were inhibited by Gleevec.
But oncologists do not give up. Instead, they are ready to change treatment to keep up. The approach, as described by Siddhartha Mukherjee, is “switching drugs when the tumor relapses, and switching again when the tumor relapses again….” This might sound like failure since doctors did not permanently cure the cancer. But the approach is effective: “The cat-and-mouse game with cancer has extended the survival of myeloma patients — strikingly in some cases” [3].
What’s the implication for machine learning methods of drug discovery? When you discover drugs to treat diabetes or heart failure, you can expect a stable success. But if you discover a drug to treat cancer, be ready to follow up to discover the next drug and the next one. Be ready to keep retraining the algorithm to keep people healthy.
More generally, machine learning stakeholders should ask themselves, Could the outputs of this model exert selective pressure on the domain it is modeling? For example, can you see any risk of accidental selective pressure in models to identify agricultural pests [5]? I would argue that if models were used to determine pesticide application, then the pests would eventually evolve in response. For example, if the model identified aphids partly by their color, and pesticides were used only in fields where the model identified aphids, then a mutant aphid with a different color would proliferate. Eventually, the pest-identification model would be useless — at least until a new model could be trained with data on the mutants.
Of course, not all models of adaptive systems will need to change. After all, just because an adaptive system can change does not mean it will. To understand conditions that make change likely, let me use a different example.
Coevolutionary hotspots spur faster evolution
Coevolutionary relationships can teach us about natural spurs to change. Specifically, a locale where two species exert reciprocal selection on each other is called an “coevolutionary hotspot,” and it leads to rapid evolution in both species [6]. For example, a parasite can spur a host to evolve faster, which in turn spurs the parasite to evolve faster.
An instructive example is the New Zealand mud snail (Potamopyrgus antipodarum) because it has two forms of reproduction. This snail normally reproduces asexually, which results in slow evolution because each offspring is the same as its parent — no genetic change. However, where a parasitic fluke is present, near the shallow shores of ponds of New Zealand, the snail switches to sexual reproduction. Why? Sexual reproduction produces new combinations of genes every generation, resulting in fast evolution, potentially helping the snail’s immune system outwit the fluke [7]. Whenever the population of snails evolve stronger immunity, they (accidentally) create selection pressure on the parasite flukes to evolve a better attack. And when the parasites do evolve a better attack, they (accidentally) create selection pressure on the snails to improve their immune defense.
In a coevolutionary hotspot, the New Zealand mud snail has to evolve quickly with sexual reproduction. In all the other locales, it is free to evolve slower with asexual reproduction (a.k.a. cloning).
What is the lesson for model owners? Just because your model has been stable for a while does not mean it can stay stable forever. If a coevolutionary hotspot develops, you will need to change your model.
For example, consider the varying context of Facebook over time. Facebook’s newsfeed algorithm decides which posts a user sees, and the algorithm could remain stable when Facebook was a site for college students to share photos. But when Facebook became widely used by many people who also vote in United States elections, parasites started spreading fake news and misinformation. In that new context, Facebook needed to switch to the mode of a fast-evolving New Zealand mud snail — or a cancer researcher who keeps fighting every new mutant.
Creativity and adaptability
So if I don’t lie — or oversimplify — what’s the generalized take-away message? Any domain with variation and differential success from the model’s use could end up breaking the model. In the case of humans, variation can be deliberate creative change and is most likely to be spurred when incentives are big. In the case of biological organisms, variation from mutations can be spread by accidental selection pressures from the model. In both cases, model owners need to be ready to change their model quickly to keep up.
Stepping back and reflecting, I feel awe. We humans are a product of evolution’s amazing ability to generate complex, adaptive, new forms. If evolution can bring about model-builders, why should it not also bring about model-breakers?
Next post: Types of model-breaking change.
Bibliography
[1] “How Gleevec Transformed Leukemia Treatment — National Cancer Institute,” Feb. 02, 2015. https://www.cancer.gov/research/progress/discovery/gleevec (accessed Oct. 03, 2021).
[2] C. Dreifus, “Researcher Behind the Drug Gleevec,” The New York Times, Nov. 02, 2009. Accessed: Oct. 03, 2021. [Online]. Available: https://www.nytimes.com/2009/11/03/science/03conv.html
[3] S. Mukherjee, The Emperor of All Maladies. London, England: Fourth Estate, 2011. Accessed: Oct. 03, 2021. [Online]. Available: https://www.simonandschuster.com/books/The-Emperor-of-All-Maladies/Siddhartha-Mukherjee/9781439170915
[4] J. Vamathevan et al., “Applications of machine learning in drug discovery and development,” Nat. Rev. Drug Discov., vol. 18, no. 6, pp. 463–477, Jun. 2019, doi: 10.1038/s41573–019–0024–5.
[5] J. Liu and X. Wang, “Plant diseases and pests detection based on deep learning: a review,” Plant Methods, vol. 17, Feb. 2021, doi: 10.1186/s13007–021–00722–9.
[6] T. P. Craig, “Geographic Mosaic of Coevolution,” in Encyclopedia of Evolutionary Biology, R. M. Kliman, Ed. Oxford: Academic Press, 2016, pp. 201–207. doi: 10.1016/B978–0–12–800049–6.00194–3.
[7] K. C. King, L. F. Delph, J. Jokela, and C. M. Lively, “The geographic mosaic of sex and the Red Queen,” Curr. Biol. CB, vol. 19, no. 17, pp. 1438–1441, Sep. 2009, doi: 10.1016/j.cub.2009.06.062.
