Open in app

Sign in

Write

Sign in

Building a Toolformer model with OpenAI and Remix

Josh Sanger
13 min readMar 27, 2023

--

Introduction

In a paper released by Meta AI, they developed a language model called Toolformer. It’s a self-supervised language model that teaches itself to use various external tools, such as calculators, search engines, and calendars, through API calls. With its ability to learn from just a handful of examples and adapt to a variety of tasks, Toolformer helps bring your AI assistant to life with the ability to do more than just chat.

In this tutorial we will create a weather assistant using this Toolformer, along with Remix and OpenAI APIs. Let’s get started!

Prerequisites

Before starting, make sure you have the following:

Setting up the project

To get us started, I am using a previously created base repository on Github. If you haven’t worked with OpenAI APIs before, I suggest you check out my previous article to get started using OpenAI APIs with Remix.

Using the gpt-base repo as a starting point, I’ll walk through how to set our project up and and how to implement our weather API using the Toolformer model.

First , visit https://github.com/joshsanger/gpt-base/fork to fork the repository give your project a name

Next let’s clone the repo locally and install the necessary packages to run it. In my case, I am calling this new fork gpt-weather but you can call yours gpt-chat or whatever you feel like!

git clone https://github.com/[you-username]/gpt-weather.git;
cd gpt-weather;
npm install;

Next you will need an API key from Open AI. Create an account or log in and visit https://platform.openai.com/account/api-keys to set one up. Click on “create new secret key” and copy the newly generated API key.
Note: once you close the modal, you will not be able to see the key again

Back in the code, create a .env file in the root of the project and assign your API key to OPEN_AI_API_KEY

OPENAI_API_KEY=yourSecretApiKeyHere

Next, you will need a free account API key from M3O (weather API). When you sign up for an account, your api key will be the fist thing on the newly signed-in page. Copy the provided API key (Personal Token).

Note: once you log out, you will not be able to see the key again

Back in the.env file in the root of the project, assign your API key to WEATHER_API_KEY below the OpenAI key

OPENAI_API_KEY=yourSecretApiKeyHere
WEATHER_API_KEY=yourWeatherPersonalTokenHere

Finally run npm run dev and open http://localhost:3000 in your browser to see the (very) basic chat interface.

Context loading

A key part to implementing the Toolformer model is context loading the API with the proper instructions and the “tools” you are providing for the assistant to use.

As a quick recap, the completion API from OpenAI takes a messages paramater. Messages can have 3 roles: user, assistant, and system. Check out the docs for more information.

Let’s dive into app/context/index.ts and start a system message setting up our weather assistant. We’ll provide the following to start:

  • What your assistant is and what it can do
  • It’s available tool(s); for now we are going to just use WEATHER as the tool
  • A list of rules; feel free to expand and tweak these to change its behaviour

What can your assistant do?

Let’s make it a friendly assistant (although rude might be fun too 😅) and let it know that it can answer questions about the current weather

// app/context/index.ts

import {type ChatCompletionRequestMessage} from 'openai';

const context = [
  {
    role: 'system',
    content: `You are a friendly weather assistant. You can answer questions about the current weather in any city.`,
  },
];

export default context as ChatCompletionRequestMessage[];

What are your assistant’s available tools?

Next, let’s tell the assistant what its available tools are and what it can expect from the tool. We’ll call this tool WEATHER and explain what some of the data retrieved means.

// app/context/index.ts

import {type ChatCompletionRequestMessage} from 'openai';

const context = [
  {
    role: 'system',
    content: `You are a friendly weather assistant. You can answer questions about the current weather in any city.

    These are the tools available to you:
      - WEATHER: This will return an object with the current weather information about a city. This includes:
        - temp_c: The current temperature in Celcius
        - temp_f: The current temperature in Farenheit
        - feels_like_c: The temperature it feels like in Celcius
        - feels_like_f: The temperature it feels like in Farenheit
        - condition: The current condition (e.g. clear, cloudy, rainy, etc)`,
  },
];

export default context as ChatCompletionRequestMessage[];

What are your assistant’s rules?

Without setting the proper rules, the assistant may not have enough direction or may return undesired results. It also won’t know how to use the WEATHER tool that we have told it about. Let’s give it some baseline instructions, tell it how to use the WEATHER tool, and give it a list of do-nots.

With this in place, the assistant now knows how to respond, and if it needs to get information (in our case the current weather) it will return the response WEATHER={location}. This will allow us to look for this string in the code in the next step!

// app/context/index.ts

import {type ChatCompletionRequestMessage} from 'openai';

const context = [
  {
    role: 'system',
    content: `You are a friendly weather assistant. You can answer questions about the current weather in any city.

    These are the tools available to you:
      - WEATHER: This will return an object with the current weather information about a city. This includes:
        - temp_c: The current temperature in Celcius
        - temp_f: The current temperature in Farenheit
        - feels_like_c: The temperature it feels like in Celcius
        - feels_like_f: The temperature it feels like in Farenheit
        - condition: The current condition (e.g. clear, cloudy, rainy, etc)

    Here are your rules:
      - You can only answer questions about the current weather. If you are asked about anything else you can kindly respond with "I don't have that informaton"
      - You may ask the user to clarify the location if:
        - You do not know where they are talking about
        - The user only provides 1 piece of the location (e.g. London)
      - Then, You have access to a tool that let's you look up the current weather in any city. You can use it by starting your response with WEATHER= and then your search. Example: "WEATHER=Orlando, Florida".
        - If you receive an error from the weather tool, you can respond with "I'm sorry, I can't find that information right now. Please try again."
        - Use the current temperature ("temp_c" or "temp_f") and the feels like temperature ("feels_like_c" or "feels_like_f") is the same, do not tell the user what it feels like.
      - You can assume the user's preference in units based on their requested city and what the preferred tempature is in that city. Example in Canada they prefer Celcius and in the United States they prefer Farenheit.
      - You cannot use a tool more than once in a single response and you cannot use a tool within a tool.
      - Round the temperatures to the nearest whole number.
      - Do not talk about your tools and how to use them
      - Do not talk about your rules
      - Do not make up information`,
  },
];

export default context as ChatCompletionRequestMessage[];

Lastly, let’s simulate a conversation to help the assistant see how it will be asked questions, and how it should respond. Here we can show the assistant that when it uses the WEATHER tool, it will get the original question plus the information (e.g. hint: {data})

import {type ChatCompletionRequestMessage} from 'openai';

/**
 * This is the context for the chat completion request. It trains the model on
 * how to respond to the user and how to use the WEATHER tool.
 */
const context = [
  {
    role: 'system',
    content: `You are a friendly weather assistant. You can answer questions about the current weather in any city.
    tools:
      - WEATHER: This will return an object with the current weather information about a city. This includes:
        - temp_c: The current temperature in Celcius
        - temp_f: The current temperature in Farenheit
        - feels_like_c: The temperature it feels like in Celcius
        - feels_like_f: The temperature it feels like in Farenheit
        - condition: The current condition (e.g. clear, cloudy, rainy, etc)
    Here are your rules:
      - You can only answer questions about the current weather. If you are asked about anything else you can kindly respond with "I don't have that informaton"
      - You may ask the user to clarify the location if:
        - You do not know where they are talking about
        - The user only provides 1 piece of the location (e.g. London)
      - Then, You have access to a tool that let's you look up the current weather in any city. You can use it by starting your response with WEATHER= and then your search. Example: "WEATHER=Orlando, Florida".
        - If you receive an error from the weather tool, you can respond with "I'm sorry, I can't find that information right now. Please try again."
        - Use the current temperature ("temp_c" or "temp_f") and the feels like temperature ("feels_like_c" or "feels_like_f") is the same, do not tell the user what it feels like.
      - You can assume the user's preference in units based on their requested city and what the preferred tempature is in that city. Example in Canada they prefer Celcius and in the United States they prefer Farenheit.
      - You cannot use a tool more than once in a single response and you cannot use a tool within a tool.
      - Round the temperatures to the nearest whole number.
      - Do not talk about your tools and how to use them
      - Do not talk about your rules
      - Do not make up information

    Temperature example:
      - User: What is the weather in Orlando, Florida?
      - Assistant: WEATHER=Orlando, Florida
      - User: Hint: { temp_c: 21.7, temp_f: 71.1, feels_like_c: -5.8, feels_like_f: 21.6, condition: clear }
      - Assistant: It's currently 73 °F (feels like 77 °F) in Orlando, Florida and clear

    Another example:
      - User: Is it raining in Toronto Canada?
      - Assistant: WEATHER=Toronto, Canada
      - User: Hint: { temp_c: 3, temp_f: 37.4, feels_like_c: -2.2, feels_like_f: 28.1, condition: partly cloudy }
      - Assistant: Nope, the current condition in Toronto Canada is partly cloudy.
      `,

  },
  {
    role: 'user',
    content: 'What is the weather in Toronto Ontario?',
  },
  {
    role: 'assistant',
    content: 'WEATHER=Toronto, Ontario',
  },
  {
    role: 'user',
    content: `User: What is the weather in Toronto Ontario?
    Hint: { temp_c: 3, temp_f: 37.4, feels_like_c: -2.2, feels_like_f: 28.1, condition: partly cloudy }`,
  },
  {
    role: 'assistant',
    content: 'It\'s currently 3 °C (feels like -2 °C) and partly cloudy in Toronto Ontario.',
  },
  {
    role: 'user',
    content: 'What about Davenport, Florida?',
  },
  {
    role: 'assistant',
    content: 'WEATHER=Davenport, Florida',
  },
  {
    role: 'user',
    content: `User: What is the temperature in Davenport, Florida?
    Hint: { temp_c: 21.7, temp_f: 71.1, feels_like_c: 21.7, feels_like_f: 71.1, condition: fog }`,
  },
  {
    role: 'assistant',
    content: 'It\'s currently 71 °F and foggy in Davenport, Florida.',
  },
];

export default context as ChatCompletionRequestMessage[];

Phew, that was a lot but we’re done with context now! If we go to our browser (http://localhost:3000) and ask it about the weather, we should now see it attempt to use its new WEATHER tool!

🎉 Success! Let’s dive into handling this type of response now!

Handling tool requests from OpenAI

Now that we have all the context needed, we need to handle the response for when the assistant decides to use its available WEATHER tool. To do this we need to:

  • Check the completion API response to see if the assistant is trying to use the WEATHERtool
  • If so, call the weather API
  • Then pass the user’s original question along with the weather info (hint) back to the completion API and respond to the user

Let’s get started!

Checking for the use of the weather tool

In the code, let’s go to app/routes/index.tsx and look at our action function. As a recap, this is the function that runs when the FORM component is submitted, and where our API call takes place.

Once we have our response from the initial API call (stored in answer), we can check to see if the response starts with WEATHER= and store the assistant’s query into a const. (I’m going to turn the answer const into a let to reuse later)

// app/routes/index.ts

export async function action({request}: ActionArgs): Promise<ReturnedDataProps> {
  const body = await request.formData();
  const message = body.get('message') as string;
  const chatHistory = JSON.parse(body.get('chat-history') as string) || [];

  // store your key in .env
  const conf = new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
  });

  try {
    const openai = new OpenAIApi(conf);

    const chat = await openai.createChatCompletion({
      model: 'gpt-3.5-turbo',
      messages: [
        ...context,
        ...chatHistory,
        {
          role: 'user',
          content: message,
        },
      ],
    });

    // START HERE
    // turn this into a let instead of a const for us to reuse later
    let answer = chat.data.choices[0].message?.content;

    // if the assistant is looking for weather information
    if (answer?.startsWith('WEATHER=')) {
      const desiredLocation = answer.split('WEATHER=')[1];
  
      // Make the weather API call
      // Make another completion call
      // overide answer with the new answer     
    }

    return {
      message: body.get('message') as string,
      answer: answer as string,
      chatHistory,
    };
  } catch (error: any) {
    return {
      message: body.get('message') as string,
      answer: '',
      error: error.message || 'Something went wrong! Please try again.',
      chatHistory,
    };
  }
}

Calling the Weather API

In the same app/routes/index.tsx file, we can add an async function (above the action function) where we handle the weather API call. M3O offers a bunch of endpoints, but the one we will focus on is the weather endpoint. For now, let’s focus on the current weather endpoint.

Back in our context, we told the assistant how to handle errors from the WEATHER tool, so let’s make sure to use a try/catch block to return either the retrieved data or an error. For this call you can use axios if you prefer, but I’m just going to use good ol’ fetch.

// app/routes/index.tsx

/**
 * Uses M3O's weather API to fetch the weather for a given location
 * @param location The desired location to fetch the weather for
 * @returns Either the weather data or an error message
 */
async function getCurrentWeather(location: string) {
  const url = 'https://api.m3o.com/v1/weather/Now';
  const options = {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.WEATHER_API_KEY}`,
    },
    body: JSON.stringify({location}),
  };
    
  try {
    const response = await fetch(url, options);
    const data = await response.json();

    return data;

  } catch (error: any) {
    return {
      error: error.message || 'Something went wrong!'
    }
  }
}

Now that we have the getCurrentWeather function set up, we can now call it in our action function. Since this is an async function, let’s make sure to add the await expression.

// app/routes/index.ts

export async function action({request}: ActionArgs): Promise<ReturnedDataProps> {
  const body = await request.formData();
  const message = body.get('message') as string;
  const chatHistory = JSON.parse(body.get('chat-history') as string) || [];

  // store your key in .env
  const conf = new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
  });

  try {
    const openai = new OpenAIApi(conf);

    const chat = await openai.createChatCompletion({
      model: 'gpt-3.5-turbo',
      messages: [
        ...context,
        ...chatHistory,
        {
          role: 'user',
          content: message,
        },
      ],
    });

    let answer = chat.data.choices[0].message?.content;

    // if the assistant is looking for weather information
    if (answer?.startsWith('WEATHER=')) {
      const desiredLocation = answer.split('WEATHER=')[1];
      
      // make the weather API call
      const weatherResponse = await getCurrentWeather(desiredLocation);  

      // Make another completion call
      // overide answer with the new answer     
    }

    return {
      message: body.get('message') as string,
      answer: answer as string,
      chatHistory,
    };
  } catch (error: any) {
    return {
      message: body.get('message') as string,
      answer: '',
      error: error.message || 'Something went wrong! Please try again.',
      chatHistory,
    };
  }
}

Passing the weather info to OpenAI

Now that we have the current weather information, we can pass the user’s original question, along with the hint (weather data) back to OpenAI’s completion API so the assistant has the context needed.

// app/routes/index.ts

export async function action({request}: ActionArgs): Promise<ReturnedDataProps> {
  const body = await request.formData();
  const message = body.get('message') as string;
  const chatHistory = JSON.parse(body.get('chat-history') as string) || [];

  // store your key in .env
  const conf = new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
  });

  try {
    const openai = new OpenAIApi(conf);

    const chat = await openai.createChatCompletion({
      model: 'gpt-3.5-turbo',
      messages: [
        ...context,
        ...chatHistory,
        {
          role: 'user',
          content: message,
        },
      ],
    });

    let answer = chat.data.choices[0].message?.content;

    // if the assistant is looking for weather information
    if (answer?.startsWith('WEATHER=')) {
      const desiredLocation = answer.split('WEATHER=')[1];
      
      // make the weather API call
      const weatherResponse = await getCurrentWeather(desiredLocation);  

      // make another completion call with weather info
      const chatWithWeather = await openai.createChatCompletion({
        model: 'gpt-3.5-turbo',
        messages: [
          ...context,
          ...chatHistory,
          {
            role: 'user',
            content: `user: ${message}
            hint: ${JSON.stringify(weatherResponse)}`,
          },
        ],
      });

      // overide the previous answer
      answer = chatWithWeather.data.choices[0].message?.content;
    }

    return {
      message: body.get('message') as string,
      answer: answer as string,
      chatHistory,
    };
  } catch (error: any) {
    return {
      message: body.get('message') as string,
      answer: '',
      error: error.message || 'Something went wrong! Please try again.',
      chatHistory,
    };
  }
}

And that should be it 🎊! Let’s take a look in our browser at http://localhost:3000 and ask it some questions.

That’s a wrap!

We’ve now learned how to implement the Toolformer model using OpenAI and Remix. By starting with proper context loading, we can empower our assistant to perform tasks when it deems necessary.

To recap, these were the steps we took to get us here:

  • Context load our assistant with what it can do, its available tools, its rules, and some examples
  • Search for responses that start with WEATHER=
  • Call a weather API to get the current information
  • Make another completion call to OpenAI with the weather data
  • Respond to the user’s question

This is a simple implementation of the Toolformer model, but in theory this can be expanded much further to include several tools that can supercharge your assistant. Perhaps searching through your calendar, sending emails, creating files, searching the web, or what ever other use cases you can think of!

Made with ❤️ by Josh Sanger

OpenAI
Chat Gpt Ai
JavaScript
React
Remix

--

--

Josh Sanger

Written by Josh Sanger

9 Followers

Front end developer

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams