MCD-UNISON
Published in

MCD-UNISON

WhatsApp group chat analysis with python

In the world of social media, the WhatsApp conversation groups are one of the most popular ways to stay in contact with family, friends and coworkers. These groups contain a lot of information and data that can be analyzed in an fun and interesting way; therefore, inspired in Saiteja Kura’sWhatsapp Group Chat Analysis using Python and Plotly”, I decided to analyze one of my close friends’ group chat to see what interesting results I could find.

WhatsApp conversation data.

The first step of this process is to obtain the conversation on TXT format. The easiest way to export an entire chat history, not including video or photos from your cell phone, is to use the built-in “Export Chat” feature, following these steps.

1. Open an individual or a group chat.
2. Tap the Menu button shown in the image below.
3. Tap the More button.
4. Select Export Chat.
5. Tap Without Media from the options that are given.
6. Select an option to share TXT file.

Importing libraries.

The following analysis uses a series of tools to develop and easy mechanism to generate charts and graphics. The imported libraries where:

import re
import regex
import pandas as pd
import numpy as np
import emoji
import plotly.express as px
from collections import Counter
import nltk
from nltk.tokenize import word_tokenize
from nltk.probability import FreqDist
from nltk.corpus
import stopwords
import datetime
import matplotlib.pyplot as plt
import seaborn as sns
from os import path
from PIL import Image
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
%matplotlib inline

Creating a DataFrame from txt file.

Regular expressions are fundamental to classify information in a chat; hence, we will use some functions based on these expressions’ use.

With the following code, plus the support of the previous functions, we will select the TXT file from the conversation, and we will form each of the initial DataFrame columns with which we will work, and we will call it “chat”.

Exploring “chat” DataFrame.

It is time to know more about new DataFrame, with help of head() pandas method, we are be able to see columns and few records

chat.head()

Using an alias for each author.

In some cases, doing this type of analysis it is better to take care with identity of the group members, for that it is possible to assign an alias to each participant at author and message column where they are mentioned

ducktales_group = list(chat.Author.unique())
ducktales_group
aliases = ['Pluto', 'Donald', 'Mickey', 'Goofy']
chat['Author'].replace(ducktales_group, aliases, inplace=True)
chat.head()

Using aliases in all messages that require it

for(name, alias) in zip(ducktales_group, aliases):
chat.Message = chat.Message.str.replace(name,alias)

Making adjustments and adding new columns.

Using another pandas method called info()we can have a lot of information about columns of DataFrames, such as format, number of nan and the count of records per column, in this way we found that format of DateTime column is string, we will transform this column to a suitable date and time format to be able to analyze these aspects.

chat["DateTime"] = pd.to_datetime(chat["DateTime"])chat.info()

With DateTime column, we will get 4 new columns weekday, month_sent, date, hour.

#new column weekday
chat['weekday'] = chat['DateTime'].apply(lambda x: x.day_name())
# new column month_sent
chat['month_sent'] = chat['DateTime'].apply(lambda x: x.month_name())
#column date
chat['date'] = [d.date() for d in chat['DateTime']]
#column hour
chat['hour'] = [d.time().hour for d in chat['DateTime']]

With Messages column, we can get 4 new columns emojis, urlcount, Letter_Count & Word_Count.

#column urlcount
URLPATTERN = r'(https?://\S+)'
chat['urlcount'] = chat.Message.apply(lambda x: re.findall(URLPATTERN, x)).str.len()
#column Letter_Count
chat['Letter_Count'] = chat['Message'].apply(lambda s : len(s))
#column Word_Count
chat['Word_Count'] = chat['Message'].apply(lambda s : len(s.split(' ')))

Function to create emojis column

Finally a DataFrame to analyze our chat group

chat.tail()

Analyzing Data

This chart shows the number of messages per day

date_grouped = chat.groupby('date')['Message'].count().plot(kind='line', figsize=(20,10), color='#A26360')

In a chart that shows the number of messages per day by authors, we can see that usually when one of them starts a conversation there is a response from the rest of the members.

Fridays are the favorite days to chat with friends

What time of day is it most common to send messages in this group?

hour_grouped_msg =  (chat.set_index('hour')['Message']
.groupby(level=0)
.value_counts()
.groupby(level=0)
.sum()
.reset_index(name='count'))
fig = px.bar(hour_grouped_msg, x='hour', y='count',
labels={'hour':'24 Hour Period'},
height=400)
fig.update_traces(marker_color='#EDCC8B', marker_line_color='#D4A29C',
marker_line_width=1.5, opacity=0.6)
fig.update_layout(title_text='Total Messages by Hour of the Day')
fig.show()

If want to know which day of the week of each month had the greatest number of messages sent, to obtain these data

1. Group data by month and day of the week, in addition to counting the messages sent
2. Make a pivot_table with DataFrame obtained from in the previous step, having as columns
days of the week and as rows the months of the year and the count carried out as the values to evaluate.
3. Using Plotly, perform a HeatMap with the ImShow function

Let’s coding…

grouped_by_month_and_day = chat.groupby(['month_sent', 'weekday'])['Message'].value_counts().reset_index(name='count')grouped_by_month_and_daymonths= ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']days = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']pt = grouped_by_month_and_day.pivot_table(index= 'month_sent', columns= 'weekday', values='count').reindex(index=months, columns= days)fig = px.imshow(pt,
labels=dict(x="Day of Week", y="Months", color="Count"),
x=days,
y=months
)
fig.update_layout(
width = 700, height = 700)
fig.show()

Using some pandas methods, is possible to know that the DataFrame has 9679 rows, 3764 of them are multimedia messages, average of words per message is 3.23, average of letters per message is 20.92 and average of daily messages is 23.67.

total_messages = chat.shape[0]
media_messages = chat[chat['Message'] == '<Multimedia omitido>'].shape[0]
average_message_words = chat['Word_Count'].mean()
average_message_letters = chat['Letter_Count'].mean()
average_message_day = chat.groupby('date')['Message'].count().mean()
print('Total Messages ',total_messages)
print('Media Message', media_messages)
print('Average Words by Messages', round(average_message_words, 2))
print('Average Letters by Messages', round(average_message_letters, 2))
print('Average Message Per Day', round(average_message_day, 2))

We can have more information on behavior of authors, analyzing data such as
Number of messages sent per author

qty_message_author = chat['Author'].value_counts()
qty_message_author.plot(kind='barh',figsize=(20,10), color=['#D4A29C', '#E8B298', '#EDCC8B', '#BDD1C5', '#9DAAA2'])
qty_message_author

Average number of messages sent daily per author

Number of multimedia messages sent by author

Number of words sent per message from each author

Its possible to know the most used words in chat, following this steps

1. Using nltk stopwords, to remove some words from our data
2. Generate a new DataFrame copying chat DataFrame selecting author and message columns
3. Separate each word of each message to make a row with each of them
4. Use "remove_emoji" function, to not consider emojis as common words
5. Remove empty or NaN rows
6. Unifying every laughing text just in a "jaja"
8. Grouping most common words then count their repetitions
9. Using plotly make a nice chart with top 10 common words

What about most common words by author?

WordCloud.

It’s time to create a word cloud with all “chat” words, for this we will use WordCloud library.

Using a couple of new functions to plot our chart and to eliminate links to web sites that could be found in messages.

#function to display wordcloud
def plot_cloud(wordcloud):
# Set figure size
plt.figure(figsize=(40, 30))
# Display image
plt.imshow(wordcloud)
# No axis details
plt.axis("off");

#function to remove urls from text
def remove_urls(text):
url_pattern = re.compile(r'https?://\S+|www\.\S+')
return url_pattern.sub(r'', text)

Preparing text to plot WordCloud

chat_word_cloud = chat[['Message']].copy()chat_word_cloud['Message']= chat_word_cloud['Message'].apply(remove_emoji)
chat_word_cloud['Message']= chat_word_cloud['Message'].apply(remove_urls)
chat_word_cloud['Message']= chat_word_cloud['Message'].replace('nan', np.NaN)
chat_word_cloud['Message']= chat_word_cloud['Message'].replace('', np.NaN)
chat_word_cloud['Message']= chat_word_cloud.Message.str.replace(r"(a|j)?(ja)+(a|j)?", "jaja")
chat_word_cloud['Message']= chat_word_cloud.Message.str.replace(r"(a|j)?(jaja)+(a|j)?", "jaja")
text = " ".join(review for review in chat_word_cloud.Message.dropna())wordcloud = WordCloud(width = 3000, height = 2000, random_state=1,
background_color='black', colormap='Set2', collocations=False,
stopwords = stopwords).generate(text)
# Plot
plot_cloud(wordcloud)

Emojis.

What data can we get from emojis in group chat?

A sum of all different used emojis.

total_emojis_list = list(set([a for b in chat.emoji for a in b]))total_emojis = len(total_emojis_list)print('Sum of all used Emojis', total_emojis)

A new DataFrame shows the top ten emojis ordered from highest to lowest based on the number of repetitions.

total_emojis_list = list([a for b in chat.emoji for a in b])
emoji_dict = dict(Counter(total_emojis_list))
emoji_dict = sorted(emoji_dict.items(), key=lambda x: x[1], reverse=True)
emoji_df = pd.DataFrame(emoji_dict, columns=['emoji', 'count'])
emoji_df.head(10)

A Plotly TreeMap chart shows variation in the amounts of use of each emoji.

fig = px.treemap(emoji_df, path= ['emoji'],
values = emoji_df['count'].tolist(),
)
fig.show()

What if analyze the number of emojis sent by each author, with their respective Plotly TreeMap?

Emojis Distribution by Pluto

Emojis Distribution by Donald

Emojis Distribution by Mickey

Emojis Distribution by Goofy

Conclusions.

It was fun and a great knowledge experience for me to have done this analysis. It is curious to realize that you can obtain a lot of interesting information in tribal things of daily life, like in this case a group of conversations from a social network.

--

--

--

Espacio de divulgación y discusión creado por estudiantes y profesores de la Maestría en Ciencia de Datos de la Universidad de Sonora (México). Proyectos, temas y discusión de lo que se hace para desarrollar la CdD en el estado de Sonora.

Recommended from Medium

A²I : Sensibility in Data around us — By us, of us and for us:)

Minimal Pandas Subset for Data Scientists

Is MLB Attendance per Game Significantly Different When It’s Cold?

Big Data analytics and optimization in Python

Linear regression in Python

How 150 Data Scientists Gathered to Hack for Social Good

Identify NBA ‘Empty-Calorie’ Scorers Based on Points Per Game and Value Over Replacement Player

Essentia: Mining Domain-Specific Paraphrases With Word-Alignment Graphs

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Luis Rafael Arce

Luis Rafael Arce

Industrial Engineer, full-stack web developer, master’s student in data science

More from Medium

Python Data Science Toolbox (Part 2): Using iterators in PythonLand

Facets — Data Visualization Tool for ML Datasets

How to specify location for bars and ticks for barplots in matplotlib

Analyze and Visualize Articles Published at GeeksforGeeks Using Plotly