Unveiling the Unpredictable: How Catastrophic Forgetting Affects AI Performance
A Stanford University study compared ChatGPT, across tasks like math problem-solving, answering sensitive questions, code generation, and visual reasoning over several months. The study discovered significant performance fluctuations (termed “drift”) between the March version (GPT-3.5) and the June version (GPT-4). For instance, GPT-4’s accuracy in identifying 17077 as a prime number dropped from 97.6% to 2.4%, while GPT-3.5’s accuracy increased from 7.4% to 86.8%. Such variations were surprising, considering the chatbot’s sophistication, and the study highlights that improving performance in one area can adversely affect others. Additionally, ChatGPT stopped providing step-by-step reasoning, and transparency decreased in sensitive question responses. Researchers stress the need to monitor large language models continuously, but limited visibility into model changes remains an issue.
One possible reason for this phenomena is catastrophic forgetting. Artificial Neural Networks, the building blocks of AIs like ChatGPT, can only retain a certain amount of information. Still, unlike structured databases, people are determining precisely how much data, or inference, a neural network of a specific size can hold. Catastrophic forgetting is when you overtrain a neural network on new tasks that it forgets the old ones, perhaps more basic functions.
What is Catastrophic forgetting:
Catastrophic forgetting refers to the phenomenon where neural networks lose the ability to complete previously learned tasks after training on new ones. It occurs due to the instability of neural networks’ internal representations.
When a neural network learns a new task, it must update its weight parameters to accommodate the new input-output mappings. This involves changing the distribution of weights across the network. Neural networks rely on specific weight configurations to encode particular representations and patterns. Altering the weights to learn new tasks interferes with previously stored representations.
During sequential training, the network is initialized, trained on task A to optimization, then trained on task B. Training on task B requires modifying weights to minimize loss for task B examples. But this disrupts weights encoding representations for task A since they remain fixed after initial training.
Mathematically, given a model f(x;θ) with parameters θ, learning task A involves finding θA that minimizes LA(f(x;θA)). Learning task B then requires finding θB minimizing LB(f(x;θB)). But θA ≠ θB in general. So f(x;θB) will have degraded performance on task A, exemplifying catastrophic forgetting.
The instability of neural network representations manifests as an abrupt, drastic performance loss on earlier tasks. Minor changes in weight parameters lead to a complete inability to reconstruct previously learned input-output mappings. This illustrates the degeneracy problem where neural networks can map distinct weight configurations to similar representations.
Catastrophic forgetting is especially problematic for continual, sequential, or lifelong learning settings. Single-task training avoids it but lacks flexibility. Overcoming catastrophic interference between competing tasks remains an active research problem in building adaptable, cumulative learning AI systems.
What causes Catastrophic forgetting:
The core cause of catastrophic forgetting is the instability and interference of neural network representations during sequential training. When training on a new task, the network weights must be updated to minimize loss on that task’s data examples. However, this disrupts the existing weight parameters that encode representations relevant to previously learned tasks.
Specifically, artificial neural networks rely on the weight matrix values to encode learned pattern mappings in the connections between neurons. The knowledge about a training dataset is captured in the network’s parametrization. Updating the weights to learn a new task destroys the previously stored mapping in the weights.
Mathematically, we can analyze this via the network function f(x;θ), where θ refers to the weight parameters. Learning a task involves optimizing θ to minimize the loss for that task. When trained on task A, the network learns weights θA that encode task A patterns. Learning task B requires computing new weights θB fitting task B data.
But θA ≠ θB in the general case. Therefore, the new weights θB lead to degradation in the network’s ability to complete task A, exemplifying catastrophic forgetting. The weights contain no residual information about the earlier mapping after updating.
This stems from the fact that neural networks rely on static weights after initial training. The weights are not designed to retain information cumulatively. Updating them to learn sequentially thus intrinsically interferes with prior representations.
Additionally, neural networks exhibit representation degeneracy — different weight configurations can produce similar outputs. Therefore, small weight changes can completely deteriorate prior representations adding redundancy to the reparametrization.
In summary, sequential training inherently conflicts with static weight encoding in neural networks. Learning new tasks opposes retention of old tasks under fixed parametrizations leading to abrupt loss of earlier knowledge. Static weights, interference, and degeneracy are the central factors underlying catastrophic forgetting.
What are the solutions to Catastrophic forgetting:
A variety of techniques have been proposed to mitigate catastrophic forgetting in neural networks during sequential learning:
Regularization Methods: These involve adding constraint terms to the loss function when training on new tasks to preserve weights important for old tasks. For example, elastic weight consolidation (EWC) adds a penalty scaled by the Fisher information to limit updates to weights crucial for previous tasks.
Rehearsal Methods: Here, data from previous tasks is stored and interleaved with new task data during training. This directly reminds the model of prior tasks — simple rehearsal stores subsets of old data. Pseudorehearsal uses generative models to synthesize artificial data maintaining old task statistics.
Dynamic Architectures: The network architecture adapts during sequential learning to accommodate new tasks alongside old ones. Progressive networks add new sub-networks and freeze old task weights. Expandable networks learn which weights to retrain vs freeze.
Dual Memory Systems: These maintain separate memory stores for prior tasks and new learning. The stores interact and consolidate information. Complementary Learning Systems leverage synaptic consolidation and neural replay to integrate new knowledge into existing representations.
Multi-Task Learning: The model learns tasks simultaneously rather than sequentially. Task-specific adaptive weights allow tuning parameters for each task independently. Gradient episodic memory modifies this for continual learning settings.
Optimization: New loss functions and training procedures can make learning more sequential. Meta-learning aims to optimize model parameters to facilitate quick sequential adaptation. Transduction methods limit plasticity temporally or spatially during training.
In summary, a broad array of techniques in regularization, rehearsal, dynamic architectures, dual memories, multi-task learning, and optimization exist to enable neural networks to learn sequentially without catastrophically forgetting previously acquired knowledge.
The fluctuations in ChatGPT’s performance over time (GPT-3.5 to GPT-4), possibly due to catastrophic forgetting, underscores the challenges of fine-tuning large language models. Both developers and investors in this space must understand the implications and continuously monitor model performance. Mitigation techniques exist to address this issue, fostering the development of more robust AI technologies. Ongoing research aims to improve model stability and transparency, making AI-driven innovations more reliable and attractive for investment opportunities.
