High-Level Overview of AI LLM Framework Architecture

This article analyzes the high-level architecture of an AI framework, discussing its individual components and their functions in the ecosystem. The AI framework should simplify the integration of traditional software with Large Language Models (LLMs).

Its role should be to provide the programmer with tools that will allow them to build advanced solutions that integrate artificial intelligence with software already operating in an organization. This innovative approach should enable the creation of software that allows running multiple AI applications and agents, opening the door to extremely sophisticated solutions.

Example Use Cases of AI Framework

To better understand the capabilities of the AI framework, let’s look at some examples of applications that should be possible to be build using it:

AI Sales Agent

A tool that should search for potential clients, analyse the specifics of their business, and prepare sales proposals for human approval. AI agents like this should look for relationships and contacts that enable making the first sales outreach.

AI Real Estate Research Agent

A tool that should monitor the real estate listings market and search for listings matching given criteria. It should prepare communication proposals to obtain missing information, assist in the formal and technical process of evaluating and purchasing real estate.

AI Slack Discussion Summarizer App

This intelligent application should be able to analyse a discussion on Slack, extract conclusions, tasks, responsible people, and next steps to take.

AI Framework Modules

The AI framework should provide the developer with a set of different modules delivering contract definitions, interfaces, and implementations of generic abstractions.

This solution should be a robust foundation on which you can build your own solution using proven patterns, adding your own implementations of individual modules, or using those prepared by the community.

Prompt and Chaining Module

Should be responsible for building prompts, i.e. programs for language models, and chains of calling those prompts, which are executed one after the other. This module should enable the implementation of various techniques used in Language Models (LMs) and Large Language Models (LLMs). It should also enable combining prompts with the model and creating prompt groups that provide one function across multiple LLM models.

Models Module

Should be responsible for handling and connecting LLM models to the software, making them available to other parts of the system.

Communication Module

Should be responsible for handling and adding new communication channels with users, whether in the form of a chat conducted in one of the messengers, or in the form of APIs and webhooks exposed for integration with other systems.

Tools Module

Should be responsible for providing functionality to add tools used by AI applications, e.g. the ability to read the contents of a website from a link, read a PDF file, search for information online, or send an email.

Memory Module

Should be responsible for memory management and allow adding additional memory function implementations for AI applications, allowing to store current state, data, and tasks that are currently being performed.

Knowledge Base Module

Should be responsible for managing access and allow adding new sources of organizational knowledge, e.g. regarding processes, documentation, instructions, information about everything captured electronically in the organization.

Router Module

Should be responsible for routing information from the outside world, coming from the communication module, to the appropriate AI application. Its role should be to determine the user’s intent and launch the right application. If the application was launched earlier and has not yet completed its operation, it should restore it and pass data from the communication module.

AI Applications Module

Should allow adding specialized AI applications focused on performing specific tasks such as automation or partial automation of processes. An example solution could be a Slack or Teams chat summarizer application. Such an application may consist of one or more prompts chained together, use tools, memory, and leverage information from the knowledge base.

AI Agents Module

Should contain more advanced versions of applications that autonomously converse with the LLM model, automatically or semi-automatically performing assigned tasks.

Accountability and Transparency Module

The Accountability and Transparency module logs all interactions between users and the AI system. It tracks queries, responses, timestamps, and authorship to distinguish between human-generated and AI-generated content. The logs provide visibility into autonomous actions taken by the AI and messages between the model and software.

Users Module

In addition to basic user management functionality, this module should maintain user account mapping across integrated systems from different modules.

Permissions Module

Should store user permission information and control user access to resources, ensuring they only have access to appropriate resources and applications.

AI Framework Component Architecture

To better visualize how the different modules of the AI framework architecture interact with each other, the following component diagram provides an overview:

The diagram shows the relationship between key components of the framework:

• The Prompt and Chaining Module builds prompts for AI Models. It chains prompt calls to create more complex logic.
• The Memory Module provides memory management via the Memory abstraction. The Knowledge Base Module gives access to knowledge sources.
• The Tools Module provides tools that can be used by AI Applications and Agents.
• The Router Module routes queries to appropriate AI Applications. Applications are managed in the AI Applications Module.
• The Communication Module handles communication channels like chat.

This component architecture illustrates how the different modules work together to enable building sophisticated AI solutions. The modular design allows easily expanding capabilities by adding new components.

Modules in Action Example

To illustrate the cooperation of AI framework modules, let’s analyse an exemplary information processing path in the system:

• The user sends a query via chat using the Communication Module.
• The Router Module analyses the content and identifies the appropriate AI Application from the Applications Module.
• The application retrieves required data from the Memory Module to regain conversation context.
• It then builds appropriate prompts using the Prompts Module and passes them to the AI Model from the Models Module.
• If needed, it executes Tools from the Tools Module, e.g. searching for information online.
• Finally, it returns the response via the Communication Module to the user.
• It stores important information in the Memory Module to continue the conversation.

Thanks to this scheme of operation, framework modules should cooperate with each other to enable complex scenarios by AI applications and agents.

Summary

The AI framework should provide comprehensive tools for building modern AI-based systems. Its flexible, modular architecture should allow easy expandability of functionality and integration with an organization’s existing software. Thanks to the AI framework, programmers should be able to quickly design and implement diverse innovative solutions using language models. Having ready-made modules at their disposal, they should be able to focus on business logic and application functionality. This should make the AI framework have the potential to significantly accelerate digital transformation across many organizations.