Build an OpenAI Powered Home Assistant

Welcome to the OpenAI Home Assistant project. The goal of the home assistant project is to explore the power of OpenAI Whisper and OpenAI Functions.

OpenAI Whisper is a powerful multilingual speech-to-text transcriber, and OpenAI Functions allow you to extract structured data from unstructured prompts. Together, these AI models bridge the messy unstructured world of natural language, both speech and written, with the structured world of software development. This bridge opens up exciting possibilities for UX designers and App developers enabling more accessible and intuitive app experiences.

There are plenty of scenarios where you will want to transcribe speech and extract structured data from prompts, for example, call center apps. The “Home Assistant” scenario is generic enough that most people will be familiar with it, and it’s a scenario we interact with day-to-day using natural language.

Video Introduction

Watch the 5-minute video introduction for this project.

Home Assistant Orchestration

The home assistant is a simple orchestration of OpenAI Whisper and OpenAI Functions using a state machine. The home assistant can turn on and off imaginary lights, get weather conditions to ground OpenAI prompts, and more. It’s an example of how you can use natural language with OpenAI Whisper and OpenAI Functions to build a home assistant that you can extend.

The agent flow is:

1. Record speech
2. Transcribe speech to text with OpenAI Whisper
3. Call OpenAI Chat Completion API with the transcribed text and function definitions to extract a function name and arguments.
4. Run code for the function and pass arguments.
5. Clean up
6. Rinse and repeat

Project source code and documentation

The project source code and documentation are Open Source and published on GitHub.

Questions?

Ask away in the comments section.

Cheers Dave