Using Vercel’s AI SDK to build your next chat bot.

Sharing the things I learned

Photo by Volodymyr Hryshchenko on Unsplash

Not too long ago, Vercel announce their brand new AI SDK. This open source tool makes it really easy to build apps with large language models and provide text streaming on the server, allowing you to create a typing effect for better user experience.

After experimenting and playing around with this new library, I was able to create my own chat bot using the Chat GPT 3.5 turbo model from open AI, and the OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 model from HuggingFace. The major tools to build this project were Next.js, Tailwind css, the Vercel AI SDK, and Vercel for deployment. The project can be found here and the repository to can be found here.

Things I learned

I’d like to share some interesting things I learned.

1. Streaming/Blocking UI: Like I have mentioned before, this SDK utilizes the streaming UI, which allows you to create the typing effect. Traditionally, a lot of responses are made using Blocking UI, which generates resposnses when the whole response is ready to be displayed. This has been a problem because it usually takes some time for these language models to generate a response. Using the Streaming UI, we can mitigate this problem by displaying responses in parts as they become available.
2. Prompt Engineering : Prompt engineering is a term used in Artificial intelligence that enables Large Language models to generate better, high quality, desired outputs. According to Brex’s documentation on prompt engineering. “You’ll have to come up with mechanisms to constrain the model to achieve reliable results, incorporate dynamic data that the model can’t be trained on, account for limitations in the model’s training data, design around context limits, and many other dimensions.” As software engineers it’s extremely important to weigh in the tradeoffs between your choices, why you want to generate one response over the other, or why you should choose a model over the other. In my project, I was able to compare the differences between Chat GPT 3.5 and Open Assitant to look at the differences between the two models. As expected, Chat GPT 3.5 turbo was by far the more superior model, it was able to generate better and more concise reponses. However, Open assistant was able to generate reponses faster, at a more noticeable speed.
3. Serverless/Edge Functions: Instead of managing and maintaining your own servers, you rely on a service(vercel,netlify,etc) to handle that for you. This enables you to concentrate on your business needs and develop a higher-quality application, without having to worry about traditional server infrastructure and maintenance.
4. The AI SDK API: The SDK provides two different hooks , useChat, and useCompletion. In my project, I was able to use the useChat hook, which allows us to implement the conversational interface, and streaming from the AI provider. The api calls were straight foreward and easy to use, and provided an amazing developer experience. More info can be found here.

That is all I have for the AI SDK. I’ve been consistently using Vercel to make(Next.js) and deploy my projects and was excited to see this new library. Please email yu24c@mtholyoke.edu if you have any questions or suggestions! And once again, Thank you for reading my blog!