Creating a Continual Learning Process in Autonomous Agents

Dion

5 min readJun 29, 2023

Prompt: Wise man that’s constantly learning

Introduction

In this article, we’ll be going through the why and how to create a continual learning process in autonomous agents.

Why continuous learning is essential
Forms of learning in autonomous agents
Existing prompt-based solutions that mimic learning
Building a continuous learning mechanism with EntityMemory
Problems with EntityMemory

Why it’s essential

Here are three reasons why continuous learning is essential

First-class personalised experience

By updating themselves to keep up with their ever-changing needs, they can provide first-class experiences to your users.

Enables iterative discussions

To be able to brainstorm ideas, an agent needs to be able to build up their knowledge about a matter, and facilitate a meaningful exchange of new ideas in the discussion.

Strengthen Decision-Making Capabilities

By constantly learning through user interactions, autonomous agents can ensure that the decisions remains relevant to the latest information available.

Forms of learning in autonomous agents

Foundational Learning

Involves updating the actual weights and parameters within the model itself. It won’t be the part of our discussion in this article.

Prompt-Based learning

Essentially prepending the prompt with learnt information that the agent might need. It will be the topic of discussion in this article.

Examining Current Prompt-based Solutions

These solutions involve keeping a memory storage of past conversations to mimic the learning process of a human.

BufferMemory

BufferMemory is the simplest type of memory described in langchain. It just stores and loads the previous interactions in the conversation.

EntityMemory

EntityMemory leverages LLMs to track information about specific entities within a conversation. This refers to any relevant people, places, concepts, or details introduced during a conversation.

ConversationSummaryMemory

ConversationSummaryMemory leverages LLMs to keep track of the conversational history and generate summarised versions of it.

Building a Cycle of Continuous Learning with Entity-Based Memory

Why Entity-Based Memory Might be a Good Idea

Just like how humans build up their internal models of the world around them, entity based memory constantly update their own internal models of the world based on the interactions they have with the users so far.

If you’re interested in the technical details of how EntityMemory works in langchain, check out this article:

Tutorial: The mechanics behind EntityMemory

Integrating Entity-Based Memory in Current Systems

You can use a popular LLM framework called langchain to do this.

Install the required dependencies

npm install -S langchain

2. Initialise an instance of the EntityMemory class

In langchain, there’s a concept known as chains. It’s just an abstraction that allows you to execute and propagate data sequentially.

Here we have a very simple chain that connects the EntityMemory to a LLM, so that the LLM can have access to the memory store.

import { OpenAI } from "langchain/llms/openai";
import {
  EntityMemory,
  ENTITY_MEMORY_CONVERSATION_TEMPLATE,
} from "langchain/memory";
import { LLMChain } from "langchain/chains";

export const run = async () => {
  const memory = new EntityMemory({
    llm: new OpenAI({ temperature: 0 }),
    chatHistoryKey: "history", // Default value
    entitiesKey: "entities", // Default value
  });
  const model = new OpenAI({ temperature: 0.9 });
  const chain = new LLMChain({
    llm: model,
    prompt: ENTITY_MEMORY_CONVERSATION_TEMPLATE, // Default prompt - must include the set chatHistoryKey and entitiesKey as input variables.
    memory,
  });
};

3. Call the chain with some initial information

const res1 = await chain.call({ input: "Hi! I'm Jim." });
  
console.log({
    res1,
    memory: await memory.loadMemoryVariables({ input: "Who is Jim?" }),
  });

Internally, the EntityMemory automatically updates it’s internal knowledge base that the user is a person called Jim.

4. Call the chain with new information

const res2 = await chain.call({
    input: "I work in construction. What about you?",
  });
  console.log({
    res2,
    memory: await memory.loadMemoryVariables({ input: "Who is Jim?" }),
  });

The EntityMemory will now update it’s knowledge about Jim, and now the LLM knows that the user works in construction.

Potential Challenges with Entity-Based Memory in langchain

Of course this method is has major flaws currently, and might not be the best type of memory to use in your application. Here’s some potential dealbreakers.

High latency and cost

They rely on large LLM to perform entity extraction and summarisation, which introduces a lot of latency and cost into the product you’re building.

2. Unreliable process to update the knowledge base

At the current-state, LLMs are unreliable. The the extractions and updating process relies on having the LLMs outputting the data in the right format with the right content.

Conclusion

While it might not be an excellent choice to use in your applications right now, I believe there’s some promise in the concept of updating internal knowledge bases worth exploring.

Let me know what you guys want to learn about next, and i’ll drop a tutorial blog!

Side note

If you’re wondering how I came up with this article, I actually built a tool, Genie, that enables you to understand open-sourced projects easily. Here’s a walkthrough of how I came up with the materials using Genie