Machine Learning Intern Journal — Deep Mind’s AlphaFold
As the title indicates, this is the journal of a Machine Learning (ML) intern at the impactIA Foundation. I’ll be attempting to keep a weekly journal of my activities in the Foundation to keep track of my progress and leave a roadmap for the interns who come after me.
Last week, a new milestone in Artificial Intelligence was attained. It was all over the news, everyone was talking about it. But what was all the hype about? And was it justified? This week, we’ll discuss DeepMind’s AlphaFold system, and what its recent performance on the CASP challenge means for the future of humanity (Spoiler: it’s quite a big deal).
Who is DeepMind?
DeepMind was founded in 2010 by Demis Hassabis, Shane Legg and Mustafa Suleyman. To accelerate the development of artificial intelligence systems, DeepMind took “an interdisciplinary approach, bringing together new ideas and advances in machine learning, neuroscience, engineering, mathematics, simulation and computing infrastructure, along with new ways of organising scientific endeavour.” In 2014, it was acquired by Google. DeepMind puts a heavy emphasis on biological intelligence as more than just inspiration for artificial systems, but as the key to creating Artificial General Intelligence. They talk about a ‘virtuous cycle’ between Neuroscience and AI, where the prior enables advances in the latter, which then enables advances in the prior, and so on. Notable breakthroughs include creating a system capable of achieving human-level performance on Atari video games (surpassing humans in several of them) — based solely on games’ visuals, developing AlphaGo — the first AI system to beat a human professional at the game of Go (I highly recommend watching the documentary), and more recently finding a solution to the protein folding problem — which we’ll explore more deeply in this blogpost.
What is the Protein Folding Problem?
Amino acids are the basic building blocks of life. They form chains to create proteins, the workhorses of living organisms (carry oxygen in your blood, let your eyes detect light or make your muscles move). What a protein does largely depends on its unique 3D structure (for more about proteins, I recommend this video). Many diseases are caused by protein misfolding, including Alzheimer’s, Parkinson’s and diabetes. Unfortunately, we have not found instructions in the chains of amino acids for how they should fold. Figuring out what shapes proteins fold into is known as the “protein folding problem”, and has stood as a grand challenge in biology for the past 50 years. Why hasn’t it been solved yet? Because the estimated time to sequentially sample all of the possible conformations of an amino acid chain is greater than the age of the universe. However, small proteins are known to fold spontaneously in milliseconds or even microseconds. This is referred to as Levinthal’s paradox.
In 1994, Professor John Moult and Professor Krzysztof Fidelis founded CASP as a biennial blind assessment to promote research, monitor progress, and establish the state of the art in protein structure prediction. Participants must blindly predict the structure of the proteins, which are then compared to the ground truth experimental data. The main metric used by CASP to measure the accuracy of predictions is the Global Distance Test (GDT) which ranges from 0–100. In simple terms, GDT can be approximately thought of as the percentage of amino acid residues within a threshold distance from the correct position. According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods.
Enter: AlphaFold
DeepMind first entered CASP13 in 2018 with their initial version of AlphaFold, which already achieved the highest performance among participants. This year, they entered their new version (which we’ll discuss below) and it smashed the ball out of the park. Below are the results.
DeepMind greatly increased their model’s GDT to reach a score of 92.4 GDT overall across all targets — hence they have solved the protein folding problem as established by the founders of CASP.
How Does AlphaFold Work?
Although DeepMind has only released a blog post so far (they are working on a paper), we can look at the paper they published for their first version of AlphaFold and use clues from the blogpost to speculate about the changes and improvements. This will get a bit technical, you can skip to ‘The Future’ if you are not particularly interested in understanding what’s going on ‘under the hood’.
The first part of AlphaFold is a Convolutional Neural Network (CNN) that takes as input a sequence of amino acid residues plus many features that the paper describes (we won’t get into them in this blog). The output of the network is a distance matrix with the rows and columns being amino acid residues that give a confidence distribution of the distance between the pair of amino acids in the final geometric structure. Finally, a non-learning based gradient descent optimisation of folding this 3D structure to figure out how you can most accurately match the distance matrix.
The first notable difference is that DeepMind says “we created an attention-based neural network system, trained end-to-end, that attempts to interpret the structure of this graph, while reasoning over the implicit graph that it’s building”. The key word there is ‘attention-based’, which hints at Transformers. So it seems they have replaced the CNN from AlphaFold 1 with a Transformer-based unit. Transformers, specifically attention-based mechanisms (we might explore this further in a future blog) are all the rage at the moment, having led to huge advances in Natural Language Processing (BERT, GPT-3, check out this article for a summary of GPT-3’s power) and seeing its use extended beyond NLP seems to position it firmly as a key player in Machine Learning advances in the next few years. A second notable difference is that the evolutionary related sequences and multiple sequence alignment are now part of the learning process rather than simply feature engineering. We’ll have to wait for the full paper to fully understand how exactly AlphaFold 2 works.
The Future
So, why is this breakthrough so significant? As explained above, many diseases are caused by protein misfolding. Understanding how proteins fold could advance our understanding of these diseases and pave the way for cures. Furthermore, understanding the structure of these proteins could help us understand their function as well as that of genes in our DNA. There are a plethora of applications, and I’ll list a few below.
- Enzymes to break down plastic waste
- Enzymes to capture carbon from the atmosphere
- Insecticidal proteins
- Frost protective coating
- Tissue regeneration through self-assembling proteins
And so many more! In the long-term future, the possibility of predicting multi-protein interactions and complex protein formations, as well as accurate physics-based simulations of biological systems promises a significant leap in scientific knowledge.
Thanks to Lex Fridman’s great summary video.
