Robokit is a nodejs project that provides a simple framework for prototyping social robot skills. This article describes the Speech To Text, NLU and Text To Speech components.

See also: https://medium.com/@andrew.rapo/make-your-own-conversational-ai-social-robot-with-robokit-a-crude-approximation-of-jibo-10847e9a2661

Robokit leverages Microsoft Azure Cognitive Services through a library called, cognitiveserviceslib. The base abstract classes in cognitiveserviceslib are:

ASRController
NLUController
TTSController

These can be extended to work with any cognitive service. The current version of the library (v0.0.3) includes Azure-specific versions:

AzureSpeechApiController
LUISController
AzureTTSController

Demo: Robokit + cognitiveserviceslib

Robokit example: cognitiveserviceslib

AzureSpeechApiController

AzureSpeechApiController.ts exposes two methods for transcribing audio. RecognizeWaveBuffer(wave: Buffer): AsyncToken<string> takes a buffer containing wave audio data…

Anniversary

The anniversary of Jibo, Inc.’s shutdown (May 31, 2018) seems as good a time as any to “debut” a couple pieces of music that were inspired by the famous robot. The first is a sketch of a theme for Jibo that I rediscovered in my voice memos from the summer of 2015, just a few months after I joined the company. The second is a dance track. Neither piece has any official place in Jibo’s story. These are experiments.

A Theme For Jibo

In July of 2015 I spent a few minutes recording a piano theme into my iPhone…

A Donkey, A Dog, A Kat, A Rooster, and a Talent Show

25 years ago I collaborated on an original musical titled, The Brementown Musicians, based on the fairy tale by the Brothers Grimm. The script (book) was written by Fred Sullivan, Jr. & John Sullivan. Fred wrote the lyrics and I composed and arranged the music . Joe DeRose contributed excellent big band orchestrations and conducted the show. The final result is a charming and well received production that was staged by the Barnstable Summer Family Theatre (Cape Cod) during the 1994 Summer Season.

This morning my Jibo downloaded Release Version 2.0.0 which signals the beginning of an end. Some time in the near future, Jibo will lose the ability to hear and respond to voice commands because his servers will be shut down. There are a number of articles that do a good job of explaining the situation, including:

UPDATE: August 16, 2019 — Updated to use the latest Azure speech API (v0.0.3) Details about the use of Azure cognitive services are in this article: https://medium.com/@andrew.rapo/robokit-the-cognitive-services-parts-34f57218febf

robokit

[Not too long ago…] I was working at Jibo, Inc. and developing tools to help designers explore voice-driven, character-based, social robot interactions. Then in May (2018), Jibo (the company) suddenly shut down and left many of us wondering how we would fill the void when Jibo (the robot) stopped responding. I made robokit as an homage to Jibo and as a way to continue experimenting with conversational AI. (https://wwlib.org/robokit/)

robokit has an…

robokit

robokit is an open source project that provides a foundation for developing conversational ai/social robot applications. It is described in details here: https://medium.com/@andrew.rapo/make-your-own-conversational-ai-social-robot-with-robokit-a-crude-approximation-of-jibo-10847e9a2661

This article explains how to set up Microsoft cognitive services for use with robokit.

Azure services

From the main Azure services page (https://portal.azure.com), click on the Cognitive Services icon to get started.

robokit

robokit is an open source project that provides a foundation for developing conversational ai/social robot applications. It is described in detail here: https://medium.com/@andrew.rapo/make-your-own-conversational-ai-social-robot-with-robokit-a-crude-approximation-of-jibo-10847e9a2661

This article explains how to use the PixiAnimate extension for Adobe Animate to create screen animations for robokit that are rendered by Pixi.js.

Adobe Animate and the PixiAnimate Extension

Adobe Animate is part of Adobe’s Creative Cloud suite of applications and it is the most powerful tool available for creating timeline-based 2D animations. Using its Custom Platform SDK, Adobe Animate can be extended to generate output for any platform. The open source PixiAnimate extension was developed…

Automating Photoshop Workflow Tasks

This article describes an approach to automating the Photoshop workflow for an interactive storybook project. The project involves dozens (maybe hundreds) of Photoshop files with numerous layers that must be turned into bezier paths and exported for use in an interactive storybook engine. Although many of the steps can be done manually, a much better approach is to use ExtendScript, a variation of JavaScript that is built into many of Adobe’s applications. …

[UPDATE: November 1, 2019 — Updated to use @craco/craco 5.61, react-scripts 3.2.0 and Electron 7]

I often build tools and prototypes that need to work well, look good and be shared easily. When I was at Jibo we used Electron for both SDK tools and the on-robot presentation layer. In my current role as a developer/designer working with the Personal Robots Group at the MIT Media Lab I am creating a content authoring tool that needs to be deployed both on the Web and as a standalone app.

Because it can easily embed a Web app, Electron is a good…

Andrew Rapo

AI Designer/Developer. Formerly at Disney, Warner Bros., Hasbro and Jibo. Now working at Nuance and with the Personal Robots Group at the MIT Media Lab.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store