Using Transformer(BERT) for Sentiment Analysis : “Inside Out 2” Review in IMDb

source : Research Gate

Sentiment Analysis ialah proses menganalisis teks digital untuk menentukan apakah nada emosional pesan tersebut positif, negatif, atau netral. Pada analisis kali ini menggunakan model BERT(Bidirectional Encoder Representations from Transformers).

BERT adalah deep learning model yang telah memberikan hasil canggih pada berbagai tugas Natural Language Processing (NLP). BERT berfungsi sebagai model kontekstual. Alih-alih membuat satu kata, memberikan representasi untuk setiap kata dalam kosakata. BERT menghasilkan representasi untuk setiap kata dalam kalimat yang didasarkan pada kata lain.

Dalam artikel ini, tujuan utama saya adalah implementasi, jadi saya tidak membahas arsitektur BERT secara mendalam.

Import Package

import nltk
import requests
from bs4 import BeautifulSoup
import string
from nltk.corpus import stopwords
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv1D, MaxPooling1D, Dropout, Flatten
import json
import time
from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
from datasets import Dataset, DatasetDict
import pandas as pd
import numpy as np

Scrapping “Inside Out 2” Review in IMBD

Pada langkah awal ini, kita melakukan scrapping pada bagian reviews, setiap scrapping halaman akan mengambil 25 baris data, sehingga kita perlu menambahkan syntax “num_pages = 10” untuk mengambil 10 halaman.

def fetch_reviews(url):
    headers = {'User-Agent': 'Mozilla/5.0'}
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        page_content = response.content
        soup = BeautifulSoup(page_content, 'html.parser')
        reviews = soup.find_all('div', class_='text show-more__control')
        return [review.get_text(strip=True) for review in reviews]
    else:
        print("Failed to retrieve the webpage.")
        return []

def load_all_reviews(base_url, num_pages):
    all_reviews = []
    for page in range(1, num_pages + 1):
        url = f"{base_url}?start={((page - 1) * 10) + 1}"
        print(f"Fetching reviews from: {url}")
        reviews = fetch_reviews(url)
        if reviews:
            all_reviews.extend(reviews)
        else:
            break
    return all_reviews

# Base URL without pagination parameters
base_url = "https://www.imdb.com/title/tt22022452/reviews"
num_pages = 10  # Number of pages to load

# Fetch all reviews
reviews = load_all_reviews(base_url, num_pages)

# Print or process the reviews
for i, review in enumerate(reviews, 1):
    print(f"{review}\n")

output yang akan didapatkan :

Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=1
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=11
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=21
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=31
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=41
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=51
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=61
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=71
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=81
Fetching reviews from: https://www.imdb.com/title/tt22022452/reviews?start=91

Review 1 : Inside Out 2 talks about "growing up" in an emotional way and depicts how growing up is challenging inside our thoughts. The film introduces the emotions we had from the previous movie: Joy, Anger, Sadness, Fear and Disgust. We are also introduced to new emotions: embarrassment, Ennui (boredom), Envy and Anxiety.Riley is about to go to high school but the story focuses on her preparation to officially join the FogHorn Hockey team. As the story progresses, Riley finds herself challenged with her hopes to join the team. Her friendship was wounded, her emotions went all over the place and the pressure got into her head. But eventually, everything goes into their proper places.This sequel film may be animated and made for kids but as an adult, it just hits differently. I can say that this film was partially made for adults and for anyone who is in a bad mental situation. It reminds its viewers how growing older can be tough, how overthinking affects everything else, how decisions that are made today may affect what's next. But it is also a strong reminder that problems, fear, worries and anxieties will come. They are all inevitable but everything will pass. Take a deep breath, stay calm and do not overstress everything. Having a calm mind also affects the way we think. We may feel pressure at any given time but we have to sit back and relax.Do not overstress things at the present. Live at the moment. The future is there but we live at the present, right here, right now. Enjoy life. Remember that you are not alone on whatever you are facing. Keep your friends and family close. And finally, God is always with us.Inside Out 2 is a great film. Every audience will have a different way of enjoying this but just remember, you are who you are. Embrace yourself.#InsideOut2 #moviereview.

Review 2 : I can't believe I'm saying this, but this is one of the best Pixar movies to date. I did not expect to like this movie very much. Disney and Pixar have been vastly disappointing and even controversial recently, but Inside Out 2 rivals Pixar's best movies like Up, Monsters inc., Ratatouille, The Incredibles and Toy Story in quality.I found the story very moving. The best animated movies ever made appeal not just to children, but resonate with the more mature audience as well. And that's what this movie does.The theme was very poignant. I loved the introduction of the new characters and especially how they incorporated anxiety into the transformation of puberty. Anxiety is not fun, but many people struggle with it.The ending was great too. I love that they didn't wrap it up in a cliche way.Great job Disney and Pixar!

Preprocessing Data

Kemudian kita perlu melakukan preprocessing data untuk menghilangkan stopwords pada review sebelum lanjut melakukan labelling pada data.

import re
import nltk 
import warnings
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from spacy import load
from nltk.tokenize import word_tokenize
from urllib.parse import urlparse
warnings.filterwarnings('ignore')

nltk.download("omw-1.4") # Open Multilingual WordNet
nltk.download("stopwords")
nltk.download("wordnet")
nltk.download("wordnet2022")
nltk.download("punkt")

# Define preprocessing functions
def remove_punctuation(text):
    return text.translate(str.maketrans('', '', string.punctuation))

def to_lowercase(text):
    return text.lower()

def remove_stopwords(text):
    stop_words = set(stopwords.words('english'))
    return ' '.join(word for word in text.split() if word not in stop_words)

def preprocess_reviews(reviews):
    preprocessed_reviews = []
    for review in reviews:
        review = remove_punctuation(review)
        review = to_lowercase(review)
        review = remove_stopwords(review)
        preprocessed_reviews.append(review)
    return preprocessed_reviews

# Apply preprocessing
preprocessed_reviews = preprocess_reviews(reviews)

# Display preprocessed reviews
for i, review in enumerate(preprocessed_reviews):
    print(f"Preprocessed Review {i+1}: {review}")

Output yang akan didapatkan :

Preprocessed Review 1: inside 2 talks growing emotional way depicts growing challenging inside thoughts film introduces emotions previous movie joy anger sadness fear disgust also introduced new emotions embarrassment ennui boredom envy anxietyriley go high school story focuses preparation officially join foghorn hockey team story progresses riley finds challenged hopes join team friendship wounded emotions went place pressure got head eventually everything goes proper placesthis sequel film may animated made kids adult hits differently say film partially made adults anyone bad mental situation reminds viewers growing older tough overthinking affects everything else decisions made today may affect whats next also strong reminder problems fear worries anxieties come inevitable everything pass take deep breath stay calm overstress everything calm mind also affects way think may feel pressure given time sit back relaxdo overstress things present live moment future live present right right enjoy life remember alone whatever facing keep friends family close finally god always usinside 2 great film every audience different way enjoying remember embrace yourselfinsideout2 moviereview
Preprocessed Review 2: cant believe im saying one best pixar movies date expect like movie much disney pixar vastly disappointing even controversial recently inside 2 rivals pixars best movies like monsters inc ratatouille incredibles toy story qualityi found story moving best animated movies ever made appeal children resonate mature audience well thats movie doesthe theme poignant loved introduction new characters especially incorporated anxiety transformation puberty anxiety fun many people struggle itthe ending great love didnt wrap cliche waygreat job disney pixar

Labelling Data

Berikutnya, kita melakukan pelabelan pada data, yaitu “Positive”, “Negative” dan “Netral” kemudian download data yang telah diberikan label dengan format .csv dengan menggunakan syntax berikut.

from textblob import TextBlob
import pandas as pd

# labelling pada sentimen
def label_sentiment(text):
    analysis = TextBlob(text)
    if analysis.sentiment.polarity > 0:
        return 'positive'
    elif analysis.sentiment.polarity < 0:
        return 'negative'
    else:
        return 'neutral'

# pelabelan sentimen
labeled = [(review, label_sentiment(review)) for review in preprocessed_reviews]

# Konversi ke DataFrame dan simpan
df_labeled = pd.DataFrame(labeled, columns=['text', 'label'])
df_labeled.to_csv('labeled.csv', index=False)

df_labeled

Output :

Implementing Transformer

Pada tahap ini, kita akan memulai dengan memuat dan memproses data, kemudian melatih model BERT, dan akhirnya mengevaluasi kinerjanya.

Kita membagi dataset menjadi set pelatihan dan validasi menggunakan train_test_split. Ini memastikan kita memiliki data untuk pelatihan dan evaluasi model kita.

Selanjutnya, kita mengonversi label kategorikal menjadi label numerik menggunakan LabelEncoder. lalu, mengonversi data kita ke dalam format yang kompatibel dengan perpustakaan Hugging Face Datasets.

Memuat tokenizer dan model BERT. Tokenizer akan menangani konversi teks menjadi token yang dapat dipahami model, dan model akan dilatih untuk klasifikasi urutan. Atur format dataset kita agar kompatibel dengan PyTorch.

Berikutnya, kita mendefinisikan metrik untuk evaluasi, yang dalam hal ini adalah akurasi. Menetapkan argumen pelatihan, seperti laju pembelajaran, ukuran batch, dan jumlah epoch.

Kita membuat objek trainer, yang menangani loop pelatihan dan evaluasi. Lalu, trainer.train,Ini akan melatih model BERT pada dataset kita.

data = pd.read_csv('/content/labeled_reviews.csv')
import torch
from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
from datasets import Dataset, load_metric
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

# Load the dataset
# Assuming `data` is a pandas DataFrame with 'text' and 'label' columns
train_texts, val_texts, train_labels, val_labels = train_test_split(
    data['text'].tolist(), data['label'].tolist(), test_size=0.2, stratify=data['label']
)

# Convert labels to numeric
label_encoder = LabelEncoder()
train_labels_numeric = label_encoder.fit_transform(train_labels)
val_labels_numeric = label_encoder.transform(val_labels)

# Convert pandas DataFrame to Hugging Face Dataset
train_data = Dataset.from_dict({'text': train_texts, 'label': train_labels_numeric})
val_data = Dataset.from_dict({'text': val_texts, 'label': val_labels_numeric})

# Load BERT tokenizer and model
model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=len(label_encoder.classes_))

# Tokenize the data
def tokenize_function(examples):
    return tokenizer(examples['text'], padding="max_length", truncation=True, return_tensors="pt")

train_data = train_data.map(tokenize_function, batched=True)
val_data = val_data.map(tokenize_function, batched=True)

# Create data loaders
train_data.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label'])
val_data.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label'])

# Define metrics
metric = load_metric('accuracy')

def compute_metrics(p):
    predictions = torch.argmax(torch.tensor(p.predictions), dim=-1)
    return metric.compute(predictions=predictions, references=p.label_ids)

# Define training arguments
training_args = TrainingArguments(
    output_dir='./results',
    eval_strategy='epoch',  # Updated argument name
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    weight_decay=0.01,
)

# Create Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=val_data,
    compute_metrics=compute_metrics,
)

# Train the model
trainer.train()

# Evaluate the model
eval_results = trainer.evaluate()
print(eval_results)

Output yang akan didapatkan :

Classification Report

Setelah model dilatih dan dievaluasi, kita dapat menggunakan model untuk membuat prediksi pada set validasi dan menghasilkan laporan klasifikasi untuk melihat metrik seperti presisi, recall, dan F1-score.

import numpy as np
from sklearn.metrics import classification_report
# Get predictions and labels for classification report
predictions = trainer.predict(val_data).predictions
pred_labels = np.argmax(predictions, axis=1)
true_labels = val_data['label']


# Print Classification Report
report = classification_report(true_labels, pred_labels, target_names=label_encoder.classes_)
print("Classification Report:")
print(report)

Output yang akan didapatkan :

Model klasifikasi sentimen yang dilatih menunjukkan performa yang sangat baik dengan akurasi keseluruhan sebesar 98%. Untuk kelas “positive”, model memiliki presisi 98% dan recall 100%, yang berarti hampir semua prediksi “positive” benar dan semua instance “positive” dikenali. Untuk kelas “negative”, presisi mencapai 100% namun recall lebih rendah di 75%, menunjukkan semua prediksi “negative” benar tetapi ada beberapa instance “negative” yang tidak dikenali. Secara keseluruhan, model ini menunjukkan keseimbangan yang sangat baik antara presisi dan recall, dengan F1-score tinggi untuk kedua kelas.

Predict Sentiment in New Sentence

Pada tahap ini kita melakukan prediksi sentimen pada kalimat baru untuk melihat hasil prediksi oleh model yang telah kita latih.

Langkah pertama adalah mengambil model yang telah dilatih dari objek trainer. Objek trainer.model menyimpan model yang telah melalui proses pelatihan, sehingga kita dapat menggunakannya langsung untuk prediksi.

Prediksi numerik diterjemahkan kembali ke label teks menggunakan label_encoder. Ini memungkinkan kita untuk melihat prediksi dalam bentuk label yang dapat dimengerti, seperti "positive" atau "negative".

Contoh penggunaan fungsi predict_sentiment untuk melakukan prediksi sentimen pada dua teks baru.

trained_model = trainer.model
import torch

# Misalkan label_encoder adalah LabelEncoder yang digunakan untuk pelatihan
def predict_sentiment(texts, trained_model, tokenizer, label_encoder):
    model.eval()  # Set model to evaluation mode

    # Tokenisasi input
    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")

    # Pindahkan input ke GPU jika tersedia
    input_ids = inputs['input_ids'].to(model.device)
    attention_mask = inputs['attention_mask'].to(model.device)

    # Ambil prediksi dari model
    with torch.no_grad():
        outputs = model(input_ids=input_ids, attention_mask=attention_mask)
        logits = outputs[0]  # Outputs adalah tuple, dan logits adalah elemen pertama

    # Ambil prediksi dari output logits
    predictions = torch.argmax(logits, dim=-1).tolist()
    
    # Decode predictions to text labels
    predicted_labels = label_encoder.inverse_transform(predictions)
    return predicted_labels

# Contoh penggunaan
texts = ["Every Scene was incredible!", "I love this film."]
sentiments = predict_sentiment(texts, trained_model, tokenizer, label_encoder)
print(f"Sentiments: {sentiments}")

Output :

Model BERT yang telah dilatih melakukan prediksi sentimen pada kedua kalimat tersebut dan memberikan hasil bahwa kedua kalimat tersebut memiliki sentimen “positive”. Output ini menunjukkan bahwa model BERT yang telah dilatih mampu mengidentifikasi sentimen positif pada kedua kalimat yang diberikan.

References :

https://huggingface.co/docs/transformers/training?source=post_page-----931ed142e37--------------------------------