Part 3: Biological Operators to Math Operators ~ Mixture of Operators for Modeling Genomic Aberrations

Nature is modular and multi-scale. While natural systems exhibit chaos and complexity in the codomain with high variability, the natural phenomena itself can be captured by countably finite foundational operators. Getting lost in the variability and chaos of the range and codomain is not much helpful when capturing governing functions.

In previous parts, we explored the foundational biological operators governing gene expression and the perturbations. Now, we delve into a mathematical framework for modeling these complex biological systems using some interesting computational techniques. One such promising approach is the Fourier Neural Operator (FNO). This part will discuss two potential strategies for employing FNOs: a single uber operator that learns all biological operators and multiple operators combined into a Mixture of Operators (MoO) model.

Series

• Part 1: Biological Operators ~ Genomic-Level Perturbations and Their Downstream Effects in Transcription
• Part 2: Biological Operators ~ Capturing the Effects of Synonymous SNPs Using Genomic Assays
• Part 3: Biological Operators to Math Operators~ Mixture of Operators Genomic Aberrations

I provide a framework (among many) for foundational AI models to capture the biological operators (one way of thinking about modeling). The article is kept intentionally at a higher level and will progress deeper into math and AI modeling in further parts.

Can an Operator Capture Chaos? Really?

An operator can indeed capture chaotic behavior. Chaos refers to the sensitive dependence on initial conditions and the complex, seemingly random behavior that can arise in deterministic systems. For example, here’s a deeper look into how operators can capture chaos

• Logistic Map: The logistic map, where T(x)=rx(1−x), is a simple yet profound example of how an operator can capture chaos. For r values between approximately 3.57 and 4, the logistic map exhibits chaotic behavior.
• Lorenz Attractor: The Lorenz system is an example where a set of differential equations with differential operators exhibits chaotic trajectories. The solutions to these equations, for certain parameters, do not settle into a fixed point or periodic orbit but instead exhibit a chaotic attractor.
• Baker’s Map: The baker’s map is another example of a chaotic operator in the context of symbolic dynamics. It is defined on the unit square and shows how a simple piecewise linear transformation can lead to chaotic dynamics.

How about the Hamiltonian operator in the Schrödinger equation? the Hamiltonian operator captures the total energy of a quantum system, including both kinetic and potential energies. It plays a crucial role in determining the system’s behavior, both in terms of its stationary states (solutions to the time-independent Schrödinger equation) and its time evolution (described by the time-dependent Schrödinger equation).

But Biology is Different!
Nope, we have built up first two parts of the series to warm up towards this idea. The rest of this part unravels a bit of that.

Two Approaches to Capture an Operator

There are two approaches to design an operator.

In the first approach, you develop a rigorous mathematical framework to elucidate the governing functions that can accurately describe the underlying dynamics of the biological state. During this phase, the objective is to formulate a Hamiltonian that can effectively model biological systems. This process demands a mental model grounded in advanced mathematics to derive a suitable equation. Once such a equation is designed, you can use them along it’s initial and boundary conditions to constrain a neural network to converge quickly on minimal data. The necessary components and methodology for this approach are detailed in a six-part series post available here:

• A Rigorous Mathematical Exposition on N-Dimensional Genomic Grammar vs One-Dimensional Linguistic Grammar

In the second part, you train a neural network to identify the fundamental operators without needing to design them manually. This approach allows Partial Differential Equations (PDEs) to be “learned” based on a given data distribution, assuming the underlying data sufficiently captures the necessary variance to achieve threshold potential for your targets. In this post, we will provide an overview of this second part.

Fourier Neural Operator (FNO) in Genomics

It is important to remember that the Fourier Neural Operator (FNO) is one among many approaches and is notably versatile and flexible, currently representing the state of the art (SOTA). The aim is not to become exclusively committed to the FNO model, but rather to recognize and appreciate the opportunities this concept provides for capturing operators.

The Fourier Neural Operator is an innovative deep learning model that extends neural networks to learn operators mapping between function spaces. Unlike traditional neural networks, which learn mappings from finite-dimensional input vectors to finite-dimensional output vectors, FNOs are designed to handle infinite-dimensional spaces. This capability makes FNOs particularly suitable for modeling continuous functions and dynamical systems, such as those encountered in genomics.

Mathematical Framework:

The core idea behind the Fourier Neural Operator is leveraging the Fourier transform to efficiently represent and learn operators acting on function spaces. Here’s a deeper look into the mathematical underpinnings of FNO:

Function Representation: Consider a function u(x) defined over a domain Ω. The Fourier transform u^(k) of u(x) is given by:

where k is the frequency domain variable.

Operator Learning: The goal is to learn an operator G such that G(u)≈v, where u and v are functions in a given space. FNO decomposes this operator into a series of transformations in the Fourier space, leveraging the convolution theorem to facilitate efficient learning.

Neural Network Architecture: The architecture of FNO consists of multiple layers, each performing the following operations:

• Fourier Transform: Convert the input function u(x) to the frequency domain.
• Linear Transformation: Apply a linear transformation in the Fourier space. This is where the operator learning occurs, as the linear transformation captures the essential characteristics of the target operator.
• Nonlinear Activation: Introduce nonlinearity through activation functions, enabling the model to learn complex mappings.
• Inverse Fourier Transform: Transform the modified function back to the spatial domain, providing the final output.

Mathematically, this can be expressed as:

where F^-1 denotes the inverse Fourier transform, W represents the linear transformation matrix, and σ is the activation function.

Here is a video to learn about FNO

Learning the Operators

Strategy 1: Single Uber Operator

In this approach, a single uber operator is trained to learn all foundational biological operators simultaneously. This strategy leverages the power of FNO to model complex and interdependent processes within a unified framework.

Advantages:

• Holistic Modeling: A single model captures the entire spectrum of biological processes, providing a comprehensive understanding of gene regulation.
• Efficiency: Reduces the computational overhead associated with training multiple models, streamlining the integration of different data types.

Challenges:

• Complexity: Training a single model to accurately learn all operators can be complex and computationally intensive.
• Generalization: Ensuring that the uber operator generalizes well across different biological contexts and conditions may be challenging, necessitating the use of multi-scale, multi-fidelity training data, including synthetic data to capture the full range of biological variability.

Application: A single FNO could be designed to take as input a multidimensional representation of genomic data (e.g., DNA sequences, chromatin states, transcription factor binding sites) and output the predicted gene expression levels, splicing variants, mRNA stability, and other regulatory elements. This model would need to be trained on a combination of high-resolution empirical data and synthetic data to accurately capture the complexity of biological interactions.

Strategy 2: Mixture of Operators Model

In this approach, multiple operators are trained separately, each specializing in a specific biological process. These specialized operators are then combined into a Mixture of Operators model to provide a comprehensive view.

Advantages:

• Modularity: Each operator can be optimized independently, simplifying the training process and allowing for more focused improvements. This modularity also facilitates the incorporation of new data types and experimental results without necessitating a complete retraining of the model.
• Flexibility: New operators can be added or existing ones modified without retraining the entire model, facilitating continuous refinement. This flexibility is particularly advantageous in rapidly evolving fields like genomics, where new discoveries frequently necessitate model updates.

Challenges:

• Integration: Combining the outputs of multiple operators into a cohesive model requires sophisticated integration techniques to ensure consistency and accuracy. Techniques such as ensemble learning, Bayesian inference, or hierarchical modeling can be employed to integrate these operators effectively.
• Data Requirements: Each operator requires multi-scale, multi-fidelity training data, which can be demanding in terms of data collection and processing. Incorporating synthetic data generated through simulations or computational models can help bridge the gaps in empirical data and ensure comprehensive training.

Application: A Mixture of Operators model could involve separate FNOs for different aspects of gene regulation:

• FNO for Gene Expression: Models transcription dynamics, predicting gene expression levels based on promoter activity, transcription factor binding, and chromatin accessibility. This operator can be trained using RNA-Seq data, ChIP-Seq data for transcription factors and histone modifications, and synthetic data modeling various transcriptional scenarios.
• FNO for RNA Splicing: Focuses on alternative splicing events, predicting splicing variants based on splice site mutations and regulatory elements. RNA-Seq and CLIP-Seq data, along with synthetic datasets simulating splicing variations, can be utilized to train this operator.
• FNO for mRNA Stability: Captures the impact of synonymous SNPs on mRNA decay rates and secondary structures, predicting mRNA half-life and stability. This operator can leverage RNA decay assays, RNA structure probing data, and synthetic data reflecting different mRNA stability conditions.
• FNO for Translation Efficiency: Models the translation process, predicting ribosome occupancy and translation kinetics based on codon usage and mRNA structure. Ribosome profiling data, codon usage tables, and synthetic data representing various translational efficiencies can be used for training.

These specialized operators can be integrated to provide a comprehensive understanding of how genomic perturbations affect the entire gene regulatory network. For example, Bayesian inference could be used to combine the predictions from individual operators, providing a probabilistic framework to capture the uncertainties and interdependencies among different biological processes.

Input and Output Domains

In the context of FNOs applied to genomics, the input domain is the whole genome sequence, encompassing all relevant genomic features such as DNA sequences, chromatin states, and epigenetic modifications. The output domain consists of the results of various bioassays, such as gene expression levels, splicing patterns, mRNA stability, and translation efficiency.

• Input Domain: Whole genome sequence including promoter regions, enhancers, introns, exons, untranslated regions (UTRs), and epigenetic marks.
• Output Domain: Results of bioassays including RNA-Seq (gene expression and splicing patterns), CAGE (TSS usage), ChIP-Seq (TF binding and histone modifications), ATAC-Seq/DNase-Seq (chromatin accessibility), and ribosome profiling (translation efficiency).

Hmmm Give me Another Example in Biology!

Okay, Let’s take Transcription Factors.

Transcription factors are proteins that bind to specific DNA sequences, thereby controlling the transcription of genetic information from DNA to messenger RNA (mRNA). They play a crucial role in regulating gene expression.

Domain, Codomain, and Range in the Context of Transcription Factors

1. Domain (Target DNA Sequences): The domain for transcription factors consists of specific DNA sequences, usually found in the promoter or enhancer regions of genes, where transcription factors can bind.
2. Codomain (Potential Regulatory Effects): The codomain includes all possible effects on gene transcription, such as activation or repression of gene expression.
3. Range (Actual Regulatory Effect): The range refers to the actual effect the transcription factor has on gene expression, which can be the upregulation or downregulation of specific genes.

How Transcription Factors Capture Gene Expression Processes

• Specific Binding: Transcription factors recognize and bind to specific DNA sequences, which determines which genes they will regulate.
• Regulatory Function: Once bound, they can either promote or inhibit the recruitment of RNA polymerase, the enzyme responsible for transcribing DNA into RNA.
• Modulation: Transcription factors can work alone or with other proteins in a complex to finely tune the level of gene expression.

Importance of Transcription Factors

• Cellular Differentiation: They are essential for cellular differentiation and development, as different transcription factors are activated in different cell types.
• Response to Signals: They allow cells to respond to various internal and external signals, adjusting gene expression as needed.
• Disease Implications: Dysregulation of transcription factors can lead to diseases such as cancer, where they may cause the inappropriate activation or repression of genes.

Visualization of Transcription Factor Activity

1. DNA Binding: The transcription factor binds to a specific DNA sequence in the promoter or enhancer region of a gene.
2. Recruitment of RNA Polymerase: Depending on its function, the transcription factor either recruits or inhibits RNA polymerase binding and activity.
3. Transcription Modulation: The level of mRNA produced from the gene is increased (activation) or decreased (repression) based on the action of the transcription factor.

Example: The p53 Transcription Factor

p53 (my favorite) is a well-known transcription factor involved in regulating the cell cycle and apoptosis. It is often referred to as the “guardian of the genome” because of its role in preventing genome mutation.

1. Domain: The target DNA sequences in the promoter regions of genes involved in cell cycle regulation and apoptosis.
2. Codomain: Potential regulatory effects include activation of genes that halt the cell cycle or initiate apoptosis.
3. Range: Actual effects can include stopping cell division to allow DNA repair or triggering programmed cell death if the damage is irreparable.

Future Thoughts

The future of research into genomic transcription and isoforms lies in advancing our understanding of biological operators and capturing them as governing functions through mathematical operators. Nature is modular and multi-scale; although natural systems exhibit chaos and complexity in the codomain with high variability, the underlying phenomena can be effectively captured by a countably finite set of foundational operators. The challenge is to identify the smallest number of assay types necessary to capture the broadest effects caused by genomic aberrations.

Hybrid machine learning models based on Fourier Neural Operators (FNO), Diffusion Models, and Physics-Informed Neural Operators (PINO) provide a powerful framework for this endeavor. By modeling biological operators with these advanced computational techniques and integrating multi-scale, multi-fidelity training data, including synthetic data, we can accurately simulate the complex dynamics of gene regulation. These insights will drive the development of targeted therapeutic strategies and advance precision medicine, transforming our understanding of genetic regulation and its implications for human health.