AI in Games: Simple Intent Classification

As I’m writing this, ChatGPT is the big thing with AI. With companies like Google scrambling now to launch competitor projects, we have no idea which company will be leading the field in a year, but honestly, that doesn’t matter. This technology has made quite an impact on our world and seems like it's going to stick around for a while, so let’s talk about it!

ChatGPT recently revolutionized the field of natural language processing, the way in which computers understand and interpret the languages we use to. One of the things it enabled is for anyone with an internet connection and a web browser to receive immediate responses to a large majority of common questions, as it has knowledge in many fields, from programming to math, to history, to finance, and way more. It still has its limitations though, as when it comes to things more niche, the AI often won’t know anything about it.

Fortunately, the same ideas and technology enabling ChatGPT can also be utilized to address these smaller applications. Here’s a practical application: I have a small game where players are thrown into a large world with a huge variety of different game mechanics. New players often get confused and ask for help in chat, but the staff members that are meant to assist them are not always on (or sometimes miss the message).

In an ideal scenario, an AI would be able to (like ChatGPT) understand and answer questions. Unfortunately, due to the tendency for text-generating models to make things up when they don’t know the answer and the practical limitations of training a model for this application, we are going to start off simple: let’s make a model that can sort messages where players are asking for help from everything else.

For this, I am using Python with sklearn and pandas. Here are the imports (will go over each one as they come up in the code):

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC

To make a model to separate these messages, we first need data to train on. CSV files are often used for small datasets like the one that we are going to need. The CSV file that we are going to create will look like this:

label,message
chat,yo
chat,yo
chat,wsg
chat,nothing much
chat,no news is good news
chat,also do you want to vc?
chat,Oh
needs assistance,what do you do on this server?
needs assistance,how do you get items
needs assistance,how do you level up
needs assistance,what is the best way to level up?
needs assistance,how are you supposed to level up
...like another 100 more...

The top of the data file contains the column names: label and message. The message contains the text of a chat message, and the label identifies whether the message is a normal chat message or a player asking for help.

To import and organize this data, we will use pandas:

# create a dataframe of the csv file
df = pd.read_csv('data.csv')
# Assign a numerical id based on the category, in this case "chat" is first, so it has id 0 and
# "needs assistance" has id 1
df['category_id'] = df['label'].factorize()[0]

Now we need to convert the data into a format that the computer can read.

NOTE: explain tokenization + counting (count vectorizer) and normalizing (TFIDF transformer)

# transform text using count vectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)

# transform text again with tfidf transformer
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

Now that the data is imported it is time to train a model on the dataset. After testing a couple of different models provided by sklearn, I have chosen the LinearSVC model due to its superior performance.

# fit the model using the encoded data from the tfidf vectorizer against the training data
clf = LinearSVC().fit(X_train_tfidf, y_train)

Then just a little bit of code is needed to run chat messages the model hasn’t seen before to test its performance and simulate a live environment:

# simple function for making a prediction with the model
def predict(input_string: str):
    input_string = input_string.replace(',', '')
    return clf.predict(count_vect.transform([input_string]))[0]


# input messages
messages = [
    "how do you get gold",  # obvious person needing help
    "yo",  # obvious chat message
    "whats up",  # obvious chat message
    "how are you"  # tricky message because it's a question but the person doesn't need help
]

# loop through input messages and predict category
for message in messages:
    print(f"[{predict(message)}] {message}")

Running the code above produces the output:

[needs assistance] how do you get gold
[chat] yo
[chat] whats up
[chat] how are you

In order to be able to actually quantify how good this machine is performing, we need to change up some of the previous code. Initially, we used all of the data in the dataset as training data, but this time we will split it up into a training dataset and a testing dataset. The training dataset will again be used by the model to train and learn from, and the test dataset will be used to simulate data. Here’s what the code looks like:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(df['message'], df['label'], test_size=0.33, random_state=0)

Now we can use the X_test and y_test variables as new testing data the model hasn’t seen. After running this through the model, the best way to visualize the data is to look at it with a confusion matrix. Here’s the code to create the matrix (source):

from sklearn.metrics import confusion_matrix
from matplotlib import pyplot as plt

y_pred = clf.predict(count_vect.transform(X_test))
conf_matrix = confusion_matrix(y_true=y_test, y_pred=y_pred)

fig, ax = plt.subplots(figsize=(7.5, 7.5))
ax.matshow(conf_matrix, cmap=plt.cm.Blues, alpha=0.3)
for i in range(conf_matrix.shape[0]):
    for j in range(conf_matrix.shape[1]):
        ax.text(x=j, y=i, s=conf_matrix[i, j], va='center', ha='center', size='xx-large')

plt.xlabel('Predictions', fontsize=18)
plt.ylabel('Actuals', fontsize=18)
plt.title('Confusion Matrix', fontsize=18)
plt.show()

Here’s what the matrix is saying:

• There were 35 normal chat messages that the model correctly classified
• There were 16 help chat messages that the model correctly classified
• There were 2 chat messages the model incorrectly classified as a help message
• The model didn’t recognize any help messages as chat messages

From this data, the accuracy of the model can be calculated to be 96%, which is definitely high enough in this case to be a useful tool for identifying when players need help.

Despite the accuracy of the model, unfortunately, it still isn’t able to entirely eliminate the need for human interference. All this model can do is tell when people need help, not actually help them. So you may now be wondering why this model is even useful at all. Well, it is a great starting point towards creating a completely automated help system. Here’s what a roadmap could look like using this simple project as a starting point:

1. Detect whether a player is asking for help or not (this project)
2. Detect what type of thing the player is asking for help with, and serve them a pre-determined help response about the topic
3. Some sort of system using a model like GPT-3 that can synthesize a response personalized to what the player is asking

Takeaways

• Implementing simple NLP like what we discussed above requires significantly less code than I thought
• Simple systems like this are not powerful enough to fully replace any human positions