Interactive Fiction Actions (Part 2)

In the first part of the series, I explained how I used the Actions SDK to host the ifvms.js ZVM interpreter to play IF games using voice with the Google Assistant.

However, I experienced some issues with handling typical conversational responses from users. The IF interpreters can handle basic variations of the text adventure commands, but it’s easy to say common variations that it can’t understand.

To have interactions be more natural and conversational, I’ll need an NLU (Natural Language Understanding) solution. API.AI has Actions on Google integrations built in and great NLU, but you can use any NLU solution you’d like with the Actions SDK. Let’s see how we can make our games more conversational using API.AI.

API.AI

The Actions SDK doesn’t support its own NLU. If you go this route, you are responsible for parsing the meaning of the raw text from user input.

One option is to port the adventure contents to the API.AI web GUI by configuring various intents for the expected user input and contexts for the various locations and states within the game.

Alternatively, you can also use API.AI to handle common variants of the commands before it is passed on to the IF interpreter. For example, for handling directions, you can create a list of entities like north, south, north-west, etc. and then use those entities in an intent to handle conversational ways of giving directions like “take me south”, “I want to go north”, “let’s go south-east”, etc.

Once API.AI matches the intent, the direction value is passed on to the action logic as an argument via fulfillment and the action then invokes the interpreter with the low level command like “go south”. Declaring the intents for expected user phrases also helps the assistant voice recognition to be biased towards the typical responses and make the voice recognition more accurate.

For any intents not handled by API.AI, the default fallback intent fulfillment will just pass the user’s raw text input to the IF interpreter.

This API.AI sample was used as the starting point:

'use strict';
process.env.DEBUG = 'actions-on-google:*';
let Assistant = require('actions-on-google').ApiAiAssistant;
let express = require('express');
let bodyParser = require('body-parser');
let app = express();
app.use(bodyParser.json({type: 'application/json'}));
app.post('/', function (request, response) {
  const assistant = new Assistant({request: req, response: res});
function responseHandler (assistant) {
assistant.tell('Hello, World!');
}

const actionMap = new Map();
actionMap.set('input.welcome', responseHandler);
assistant.handleRequest(actionMap);
};
if (module === require.main) {
let server = app.listen(process.env.PORT || 8080, function () {
let port = server.address().port;
console.log('App listening on port %s', port);
});
}

The action map was extended to support the intents defined in the API.AI project:

app.post('/', function (request, response) {
const assistant = new Assistant({request: request,
response: response});
const WELCOME_INTENT = 'input.welcome';
const UNKNOWN_INTENT = 'input.unknown';
const DIRECTION_INTENT = 'input.directions';
const DIRECTION_ARGUMENT = 'Directions';
const LOOK_INTENT = 'input.look';
const INVENTORY_INTENT = 'input.inventory';
const EXAMINE_INTENT = 'input.examine';
const EXAMINE_ARGUMENT = 'Thing';
const REPEAT_INTENT = 'input.repeat';

const welcomeIntent = (assistant) => {
runner.start();
};

const unknownIntent = (assistant) => {
if (assistant.getRawInput() === 'quit') {
assistant.tell('Goodbye!');
} else {
// pass the user input to the interpreter
assistant.mappedInput = assistant.getRawInput();
runner.start();
}
};

const lookIntent = (assistant) => {
// map intent to game command
assistant.mappedInput = 'look';
runner.start();
};
  ...
  let actionMap = new Map();
actionMap.set(WELCOME_INTENT, welcomeIntent);
actionMap.set(UNKNOWN_INTENT, unknownIntent);
actionMap.set(DIRECTION_INTENT, directionsIntent);
actionMap.set(LOOK_INTENT, lookIntent);
actionMap.set(INVENTORY_INTENT, inventoryIntent);
actionMap.set(EXAMINE_INTENT, examineIntent);
actionMap.set(REPEAT_INTENT, repeatIntent);

assistant.handleRequest(actionMap);
});

When we tested an action using this experiment, some of the feedback we got from users were that the game needed better hand holding in the first couple turns. So the fulfillment logic was updated to add hints at the end of the first two prompts to the user:

  • ‘In this interactive adventure, you can use commands to get around or examine objects, including yourself. Try saying things like “examine me” or “look around”. What do you want to do?’
  • ‘You’re looking for the pig, so try going in different directions to explore the farm. You can also examine or pick up objects. What do you want to do next?’

The no-input prompts were also adjusted to give additional help on how to play the game:

  • ‘Try a command like “look around” or “go north”.’
  • ‘If you’re still there, let me know what you want me to do next.’
  • ‘We can stop here. Let’s play again soon.’

Since the prompts to the user could sometimes be long, we also added SSML breaks between sentences to better pace the story telling.

Next Steps

This experimental project showed that it is possible to render existing IF file formats on voice-enabled devices. Although the IF interpreter isn’t currently designed for this use case, it should be relatively easy to better support actions. Some text adventures provide menus and other interactive steps which currently aren’t handled by the voice interpreter, but the stories could be edited to consider this use case.

We have open sourced the code for the experimental IF action. Follow these instructions to deploy it on your own Google Home device.

We are giving away the code to enable all of you to publish your stories on Google Home and extend the long history of text adventures to new audiences. We also hope you’ll contribute improvements. We are eager to see the stories you’ll come up with!