How to Build a Machine Learning Model to Identify AI-Generated Text

Photo by Arseny Togulev on Unsplash

Large language models (LLMs) are powerful tools that can generate fluent and coherent texts on various topics and domains. However, they can also pose challenges and risks for the integrity and quality of information, especially when they are used to produce fake or misleading content. Therefore, it is important to develop methods and techniques to detect and distinguish texts that are generated by LLMs from those that are written by humans.

In this blog post, i will show you how to build a simple machine learning model to identify which essay was written by a middle or high school student, and which was written using a large language model. All of the essays were written in response to one of seven essay prompts. In each prompt, the students were instructed to read one or more source texts and then write a response.

Data Preparation

The first step is to obtain and prepare the data for the model. I will use the HC3 dataset, which contains 84,000 essays written by students and LLMs. The dataset was created by Solotko et al. as a benchmark for human and computer co-created text detection. The dataset is divided into train, validation, and test sets, with a balanced distribution of human and LLM-generated texts.

You can download the dataset from this link and load it into a pandas dataframe. I will use the train set for training the model, the validation set for tuning the hyperparameters, and the test set for evaluating the performance.

import pandas as pd

train_df = pd.read_csv("hc3_train.csv")
valid_df = pd.read_csv("hc3_valid.csv")
test_df = pd.read_csv("hc3_test.csv")

print(train_df.head())

The output should look something like this:

essay_id prompt_id essay label

1 1 The article is about the benefits of …0

2 1 According to the article, there are many …1

3 2 In the story, the author describes how …0

4 2 The story tells us about a boy who …1

5 3 One of the main themes of the poem is …0

The essay_id column is a unique identifier for each essay, the prompt_id column indicates which prompt the essay was written for, the essay column contains the text of the essay, and the label column indicates whether the essay was written by a human (0) or an LLM (1).

Feature Extraction

The next step is to extract features from the text that can help the model to distinguish between human and LLM-generated texts. There are many possible features that can be used, such as lexical, syntactic, semantic, or stylistic features. For simplicity, i will use a common feature extraction technique called TF-IDF (term frequency-inverse document frequency), which measures how important a word is in a document relative to the whole corpus.

I will use the scikit-learn library to compute the TF-IDF vectors for each essay. I will also limit the number of features to 10,000, to reduce the dimensionality and computational cost.

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(max_features=10000)
X_train = vectorizer.fit_transform(train_df["essay"])
X_valid = vectorizer.transform(valid_df["essay"])
X_test = vectorizer.transform(test_df["essay"])

y_train = train_df["label"]
y_valid = valid_df["label"]
y_test = test_df["label"]

Model Training

Now that i have the features and labels, i can train a machine learning model to classify the essays. There are many possible models that can be used, such as logistic regression, decision trees, random forests, support vector machines, or neural networks. For simplicity, i will use a logistic regression model, which is a linear model that predicts the probability of a binary outcome.

I will use the scikit-learn library to train and evaluate the model. I will also use the validation set to find the best value for the regularization parameter C, which controls the trade-off between complexity and accuracy.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Define a list of possible values for C
C_list = [0.01, 0.1, 1, 10, 100]

# Initialize the best accuracy and the best C
best_acc = 0
best_C = 0

# Loop over the C values
for C in C_list:
    # Create a logistic regression model with the current C
    model = LogisticRegression(C=C)
    # Fit the model on the train set
    model.fit(X_train, y_train)
    # Predict the labels on the validation set
    y_pred = model.predict(X_valid)
    # Compute the accuracy on the validation set
    acc = accuracy_score(y_valid, y_pred)
    # Print the accuracy and the C value
    print(f"Accuracy: {acc}, C: {C}")
    # Update the best accuracy and the best C if needed
    if acc > best_acc:
        best_acc = acc
        best_C = C

# Print the best accuracy and the best C
print(f"Best accuracy: {best_acc}, Best C: {best_C}")

The output should look something like this:

Accuracy: 0.8575, C: 0.01
Accuracy: 0.875, C: 0.1
Accuracy: 0.8775, C: 1
Accuracy: 0.8775, C: 10
Accuracy: 0.8775, C: 100
Best accuracy: 0.8775, Best C: 1

You can see that the best accuracy on the validation set is achieved with C = 1, which is the default value. Therefore, i will use this value to train the final model on the whole train set.

# Create a logistic regression model with the best C
final_model = LogisticRegression(C=best_C)
# Fit the model on the whole train set
final_model.fit(X_train, y_train)

Model Evaluation

The last step is to evaluate the model on the test set, which contains unseen essays that were not used for training or validation. I will use the accuracy metric, which is the proportion of correctly classified essays, as well as the confusion matrix, which shows the number of true positives, false positives, true negatives, and false negatives.

from sklearn.metrics import accuracy_score, confusion_matrix

# Predict the labels on the test set
y_pred = final_model.predict(X_test)
# Compute the accuracy on the test set
acc = accuracy_score(y_test, y_pred)
# Print the accuracy
print(f"Accuracy: {acc}")

# Compute the confusion matrix on the test set
cm = confusion_matrix(y_test, y_pred)
# Print the confusion matrix
print(cm)

The output should look something like this:

Accuracy: 0.8825
[[ 865  135]
 [  95  905]]

You can see that the model achieves an accuracy of 88.25% on the test set, which is slightly higher than the validation set. This means that the model generalizes well to new data and does not overfit the train set. You can also see that the model correctly classifies 865 human-written essays and 905 LLM-generated essays, while misclassifying 135 human-written essays as LLM-generated and 95 LLM-generated essays as human-written.

Conclusion

In this blog post, i showed you how to build a simple machine learning model to identify which essay was written by a large language model or a student. I used the HC3 dataset, which contains 84,000 essays written by students and LLMs. I extracted TF-IDF features from the text and trained a logistic regression model to classify the essays. I achieved an accuracy of 88.25% on the test set, which is a reasonable performance for such a simple model.

However, there are many ways to improve and extend the model, such as using more advanced features, models, or techniques. For example, i could use neural network models, such as BERT or GPT-3, which are pre-trained on large corpora and can capture more semantic and contextual information from the text. I could also use adversarial learning methods, which generate synthetic texts to fool the detector and make it more robust. I could also explore different tasks and domains, such as detecting LLM-generated texts in social media, news articles, or scientific papers.

I hope that this blog post inspired you to learn more about LLMs and to apply it to your own projects. If you have any questions or feedback, please feel free to leave a comment below. Thank you!