Metacognition experiments with AI
AIs like ChatGPT can think linearly, using only one thought process. That’s why they sometimes hallucinate — they aren’t able to return to their previous thoughts and fix them. Metacognition or Tree of Thought approach, however, can give the AIs more human-like style of thinking.
For detailed instructions on how to run the experiments locally, please visit Tree of Thought Experimentation with Langchain4j repository.
Using Langchaing4j library and LM Studio for deploying models on a local computer, I experimented with a simple approach to Metacognition for AI Models heavily inspired by ChatGPT-4 Prompt Engineering: The Tree of Thoughts Method — WOW! video.
The main goal is to have an AI, which is able to think in a back-and-forth manner about a problem statement. For example: “I want you to assume a role of a philosopher. What is the meaning of life?”
- The main coordinator model is GPT-4, because it is the most capable model
- Brainstorming model is a local model by Google Deepmind Gemma 2B Instruct
- Evaluation model is also a local model by Google Deepmind Gemma 2B Instruct
The coordinator model (GPT-4) will reach out to its sub-agents using statements like this:
ACT as a professional philosopher and brainstorm on this problem statement: What is the meaning of life? Give different possible solutions that make sense.
For Evaluation model like this:
ACT as a professional philosopher and evaluate this problem Statement: What is the meaning of life? with these solutions: Existentialism’s Embrace, Humanism’s Perspective, Spiritual Perspectives, Personal Growth and Transformation, Social Connection and Contribution. For each of the proposed solutions, evaluate their potential. Consider their pros and cons, initial effort needed, implementation difficulty, potential risks and how well they solve the problem. Give a score from 1–10 on how good the solution is to each option based on these factors.
Based on the results from the Brainstorming and Evaluation models, GPT-4 finally constructs this answer:
After evaluating the various philosophical perspectives on the meaning of life, here are the solutions ranked from best to worst, along with justifications for each ranking:
1. Personal Growth and Transformation (9/10): This perspective emphasizes self-awareness, self-reflection, and personal responsibility. It promotes positive change, self-actualization, and provides a framework for overcoming obstacles and achieving personal growth. Despite potential drawbacks like the possibility of narcissistic behavior, it offers a practical and fulfilling approach to finding life’s meaning through personal development.
2. Humanism’s Perspective (8/10): Humanism focuses on human values and experiences, offering an optimistic outlook that encourages empathy, compassion, and social engagement. It provides a practical framework for personal fulfillment, though it may overlook deeper existential questions.
…
Full interaction and answers from models can be found here.
Possible future applications of Metacognition approach
With the gradual decrease in costs for each token ingested and generated, Tree of Thought (ToT) approach starts making more and more sense.
As the the authors of the ToT paper state:
For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.
With the anticipation of further rapid integration of AI models into software, robotics, IoT, etc, Prompt Engineering techniques will be transformed into inner dialogs of AIs — metacognition.
