🤗 Hugging Face Transformers Agent Explained

Want to understand how AI Agents work? This article is for you.

The Hugging Face Transformers Agent in action. Solves tasks using a Large Language Model and AI tools.

Artificial Intelligence (AI) is very hyped. The hype started with ChatGPT which is a large language model (LLM). You can have a very advanced conversation with ChatGPT and other LLMs. For example ChatGPT can write blog posts for you. You can ask ChatGPT questions (prompts) and it will return back answers.

Since ChatGPT has only been trained on data up to 2021, it is just making things up and it cannot fully replace me yet. So the below is MADE BY A HUMAN. 😊

The next step after LLMs are intelligent AI agents like LangChain and the just released Hugging Face Transformers Agent.

An intelligent agent (IA) is an agent acting in an intelligent manner; It perceives its environment, takes actions autonomously in order to achieve goals, and may improve its performance with learning or acquiring knowledge.

The Basics

In order for me to understand how the Hugging Face Transformers Agent works, I spent some hours building the transformers-agent-ui data app you saw at the top.

I will use screenshots from the app to explain you how it all works.

First of all you have to give the agent a task

The task given to the Agent

The agent can take additional inputs like the boat_image. The format can be text, image, audio, video, tabular data and more.

An additional input argument given to the Agent

Then you run the Agent

The Transformers Agent will then convert the task into a cleverly engineered prompt and send that to the LLM.

Take a minute to understand the prompt. Its amazing.

A small part of the prompt sent to the LLM by the Agent

The prompt is much longer than what is shown here. But now I understand why prompt engineering is such a hyped skill.

The cleverness of the LLM and the powerful tools that are available are what makes the potential of the agent so incredible. In theory it can use the 100.000s of AI tools available on the Hugging Face platform! Currently it has access to only a tiny subset though.

The LLM will return an explanation of what the agent will do as well as the Python code 🐍 to execute.

The explanation returned by the LLM to the Agent

The Python code returned by the LLM to the Agent

This is where things will often break. Either because the code does not work or because (for security reasons?) the agent is not allowed to execute it.

But here it works

That’s all… almost.

The Details

The Hugging Face Transformers Agent gives you access to LLM models from Hugging Face

The Hugging Face LLM agents

and Open AI

The OpenAI LLM Agent

The Hugging Face LLMs are currently free to use while the OpenAI LLM is not. The OpenAI LLM works best.

In order to use the agent(s) services you will have to provide token(s). The token is just a text string. You can think of it as a password.

Here tokens are provided via environment variables

When the agent runs it will by default run the tools locally on your machine. I don’t have a GPU on my machine, so it can take many 10s of seconds for a single run. Luckily for many tools it is also an option to run them remotely on Hugging Face servers.

Run everything remotely if possible (and secure)

Want to try the data app?

Go to transformers-agent-ui to get started. Its early days, so don’t expect too much.

Panel. The powerful data exploration & web app framework for Python

The app was built using the HoloViz Panel data app framework. Actually a 1.0 Release Candidate. Please give the project a ⭐ on Github. Its such an awesome project.

Learn More

Check out this article by my friend

Sophia Yang

: 🤗Hugging Face Transformers Agent | by Sophia Yang | May, 2023 | Towards Data Science