Building a voice-input and text-based AI chatbot with OpenAI and Python — Chatbot UI Development (Part 2)

This blog series will guide you through the development of a voice-input and text-based AI chatbot using OpenAI and Python, with a focus on Flask. The series is divided into two parts:

• Part 1: We will cover the application setup and delve into chatbot management, covering the process of converting speech to text and generating chatbot responses.
• Part 2: We will cover chatbot UI development of how to record audio from the microphone and make it into Blob before sending the data to server.

In this blog, we will focus on Part 2. If you’re keen on diving into the backend development of the AI chatbot, please visit our separate blog for Part 1.

Chatbot UI Development

1 Create Template

First, let’s create a template for the AI chatbot. You have the option to either customize your own chatbot interface or use the one I’ve provided as a reference. (Click here to see the template)

2 Configure script.js

If you’re using my AI Chatbot template, you can refer to the following code to get insights of how to record audio from the microphone and make it into Blob before sending the data to server.

Frequently Used Variables

let mediaRecorder;
let mediaStream;
let recordedChunks = [];

const userInputField = $("#userInput"); 
const chatbox = $("#chatbox"); 
const loadingDiv = $("#loading div"); 
const responseFailedDiv = $("#response-failed div"); 

Function 1: startRecording()

// Function to handle recording and saving the audio
  function startRecording() {
    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then(function (stream) {
        // Create a MediaRecorder object and assign the stream to it
        mediaRecorder = new MediaRecorder(stream);

        // Store the MediaStream object
        mediaStream = stream;

        // Event listener for data available during recording
        mediaRecorder.ondataavailable = function (event) {
          recordedChunks.push(event.data);
        };

        // Event listener for stopping the recording
        mediaRecorder.onstop = function () {
          // Combine the recorded chunks into a single Blob
          const recordedBlob = new Blob(recordedChunks, { type: "audio/wav" });

          // Clear the recordedChunks array for future recordings
          recordedChunks = [];

          // Show the "Save" button and hide the "Stop" button
          $("#loadingBtn").removeClass("d-none");
          $("#stopBtn").addClass("d-none");
          $("#recordBtn").addClass("d-none");

          // Show a message to indicate recording has stopped
          // alert('Recording stopped. Click "Save Audio" to save the recording.');

          // Call the function to handle saving the audio
          saveAudio(recordedBlob);
        };

        // Start recording
        mediaRecorder.start();

        // Show the "Stop" button and hide the "Record" button
        $("#stopBtn").removeClass("d-none");
        $("#recordBtn").addClass("d-none");
        $("#loadingBtn").addClass("d-none");

        // Show a message to indicate recording has started
        // alert('Recording voice... Click "Stop" to stop recording.');
      })
      .catch(function (error) {
        console.error("Error accessing user media:", error);
      });
  }

• This function is triggered when the “Record” button (#recordBtn) is clicked.
• It accesses the user’s microphone using the navigator.mediaDevices.getUserMedia() method to create a MediaStream object.
• It sets up a MediaRecorder to record audio data from the microphone and stores the audio data chunks in the recordedChunks array.
• It sets up event listeners for ondataavailable and onstop to handle data recording and stopping.

Function 2: saveAudio(audioBlob)

// Function to handle saving the audio to the server using Ajax
  function saveAudio(audioBlob) {
    const formData = new FormData();
    formData.append("audio", audioBlob, "recorded_audio.wav");

    $.ajax({
      type: "POST",
      url: "/speech-to-text",
      data: formData,
      processData: false,
      contentType: false,
      success: function (data) {
        const { status, result } = data;
        if (status == "success") {
          $("#userInput").val(result);
        } else {
          console.log(result);
          alert("Something went wrong. Please try again.");
        }
        $("#recordBtn").removeClass("d-none");
        $("#stopBtn").addClass("d-none");
        $("#loadingBtn").addClass("d-none");
      },
      error: function (error) {
        alert("Something went wrong. Please try again.");
        console.error("Error saving audio file:", error);
        $("#recordBtn").removeClass("d-none");
        $("#stopBtn").addClass("d-none");
        $("#loadingBtn").addClass("d-none");
      },
    });
  }

• This function is called after the audio data is converted into Blob.
• It is responsible for sending the recorded audio to the server.
• It receives an audioBlob as input, which is the recorded audio data, and accepts the speech-to-text result.

Function 3: displayChatbotResponse(message)

function displayChatbotResponse(message) {
    // Append the chatbot response to the chatbox
    $("#chatbox").append(
      $('<div class="d-flex flex-row bg-light py-4">') // Create a new div element with classes for chatbot response
        .append(
          $('<span class="material-symbols-rounded mx-3"> smart_toy </span>')
        ) // Create a new span element with smart_toy icon
        .append($("<span>").text(message.content)) // Create a new span element with the chatbot response text
    );
  }

• This is a custom function defined to display the chatbot’s response in the chatbox.

Event Listener 1: stopBtnButton Click

// Event listener for "Stop" button click
  $("#stopBtn").on("click", function () {
    // Stop the recording
    mediaRecorder.stop();

    // Stop the MediaStream
    if (mediaStream) {
      mediaStream.getTracks().forEach((track) => track.stop());
      mediaStream = null;
    }
    // Get the last recorded audio Blob (if any) and call the function to save it
    if (recordedChunks.length > 0) {
      const recordedBlob = new Blob(recordedChunks, { type: "audio/wav" });
      saveAudio(recordedBlob);
    }
  });

• This event listener will call the saveAudio function to handle saving the audio.
• It stops the recording and MediaStream as well as combines the recorded audio chunks into a single Blob before saving the audio.

Event Listener 2: recordBtnButton Click

// Event listener for voice recording button click
  $("#recordBtn").on("click", startRecording);

• This event listener will call the startRecording function to initiate audio recording.

Event Listener 3: sendUserInputBtnButton Click

$("#sendUserInputBtn").on("click", function () {
    responseFailedDiv.addClass("d-none"); // Hide the response-failed div initially

    const userInput = userInputField.val(); // Get the user input value
    userInputField.val(""); // Clear the user input field

    // Append the user input to the chatbox
    chatbox.append(
      $('<div class="d-flex flex-row-reverse py-4">') // Create a new div element with classes for user input
        .append(
          $('<span class="material-symbols-rounded mx-3"> person </span>') // Create a new span element with person icon
        )
        .append($("<span>").text(userInput)) // Create a new span element with the user input text
    );

    // Send user input to the AI chatbot endpoint using AJAX
    $.ajax({
      type: "POST",
      url: "/ai-chatbot",
      data: { input: userInput },
      beforeSend: function () {
        loadingDiv.removeClass("d-none"); // Show the loading div before making the AJAX request
      },
      complete: function () {
        loadingDiv.addClass("d-none"); // Hide the loading div after the AJAX request is complete
      },
      success: function (data) {
        const { status, messages } = data;
        if (status === "success") {
          displayChatbotResponse(messages[messages.length - 1]); // Display the chatbot response
        } else {
          responseFailedDiv.removeClass("d-none"); // Show the response-failed div if there is an error
        }
      },
      error: function (error) {
        console.log("Error:", error);
        responseFailedDiv.removeClass("d-none"); // Show the response-failed div if there is an error
      },
    });

• This event listener will transmit the user’s input to the server and, at the same time, modify the UI chatbot element based on the loading status and result status.

This is how the chatbot is processed on the client side. For a comprehensive view of the entire code structure, please refer to the following scripts:

• speech_to_text.js:
  https://github.com/henghuisan/practice-with-open-ai/blob/main/app/static/js/speech_to_text.js
• ai_chatbot.js:
  https://github.com/henghuisan/practice-with-open-ai/blob/main/app/static/js/ai_chatbot.js

We’ve completed Part 2!

I hope your chatbot successfully handles both user voice and text inputs and effectively generates responses.

Thank you for reading.