YouTube Comment Scraping Made Easy with Apps Script
The first step in conducting research involves acquiring an appropriate dataset. Dataset plays a crucial role in ensuring the accuracy and dependability of the results we obtain. Several studies have highlighted the significance of collecting a high-quality dataset, including studies in social sciences, economics, and healthcare.
Fortunately, YouTube is a valuable resource that we can use for collecting datasets, particularly regarding user-generated content and social interactions. On YouTube, you can find various types of data such as comments, the number of likes for the comments, and reply comments. These data points can provide us with unique insights into viewers’ opinions, emotions, and behaviours. To collect YouTube comments, there are several methods we can use;
- YouTube Data API: The YouTube Data API allows you to retrieve comments associated with a YouTube video by making API requests. By leveraging the API’s features and parameters, such as search queries, video IDs, and pagination, you can extract comments programmatically.
- Python Libraries: Python offers you several libraries, such as youtube_dl, pytube, and youtube-comment-scraper, which facilitate scraping YouTube comments. These libraries provide functions and interfaces to fetch comments directly from a video URL.
- Web Scraping with Selenium: Selenium is a web automation tool, you can use it in conjunction with Python or other programming languages to automate the process of navigating to a YouTube video page, dynamically loading comments, and extracting them from the page’s HTML structure. This method requires knowledge of web scraping techniques and the ability to work with XPath or CSS selectors.
Another approach we can use to scrape YouTube comments is through the utilization of Apps Script. Apps Script is a scripting platform developed by Google, that provides a user-friendly interface that enables easy automation and interaction with various Google services, including YouTube’s API.
Methods like the YouTube Data API and web scraping typically require some degree of technical knowledge. However, with Apps Script, you can easily collect YouTube comment datasets with less effort and a lower barrier to entry. By using simple JavaScript code, we can easily develop scripts and automate the process of collecting YouTube comment datasets using Apps Scripts.
Let’s look at the step-by-step procedure for scraping comments from a specific video and saving them in a Google Sheet using Apps Script.
Navigate to Google Sheets and create a blank Sheet.
Choose the video that you wish to extract comments from.
Copy the video ID from the URL of the selected video. The video ID can be found after the “v=” in the Youtube video URL and is typically a sequence of characters. ( Please note that sometimes the video URL may include additional information such as starting times or playlist details. When copying the video ID, ensure that you exclude any such parts.)
Paste the copied video ID into cell A1 of the newly created spreadsheet.
Click on the “Extensions” menu located in the menu bar and Select Apps Scripts. You will be directed to a new Apps Script project.
Copy and paste the following code snippet into the window.
function scrapeComments(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var result=[['Name','Comment','Time','Likes','Reply Count','Reply Author','Reply','Published','Updated']];
var vid = ss.getSheets()[0].getRange(1,1).getValue();
var nextPageToken=undefined;
while(1){
var data = YouTube.CommentThreads.list('snippet', {videoId: vid, maxResults: 100, pageToken: nextPageToken})
nextPageToken=data.nextPageToken
for (var row=0; row<data.items.length; row++) {
result.push([data.items[row].snippet.topLevelComment.snippet.authorDisplayName,
data.items[row].snippet.topLevelComment.snippet.textDisplay,
data.items[row].snippet.topLevelComment.snippet.publishedAt,
data.items[row].snippet.topLevelComment.snippet.likeCount,
data.items[row].snippet.totalReplyCount,'','','','']);
if(data.items[row].snippet.totalReplyCount>0){
parent=data.items[row].snippet.topLevelComment.id
var nextPageTokenRep=undefined
while(1){
var data2=YouTube.Comments.list('snippet', {videoId: vid, maxResults: 100, pageToken: nextPageTokenRep,parentId:parent})
nextPageTokenRep=data2.nextPageToken;
for (var i =data2.items.length-1;i>=0;i--){
result.push(['','','','','',
data2.items[i].snippet.authorDisplayName,
data2.items[i].snippet.textDisplay,
data2.items[i].snippet.publishedAt,
data2.items[i].snippet.updatedAt]);
}
if(nextPageTokenRep =="" || typeof nextPageTokenRep === "undefined"){
break
}
}
}
}
if(nextPageToken =="" || typeof nextPageToken === "undefined"){
break;
}
}
var newSheet=ss.insertSheet(ss.getNumSheets())
newSheet.getRange(1, 1,result.length,9).setValues(result)
}Click on the “+” sign in the services tab located on the left-hand side of the screen. A pop-up window will appear. In that window, select “YouTube Data API v3” from the options provided. After selecting it, click on the “Add” button to include the YouTube Data API as a service. You should now see that YouTube has been successfully added as a service in the left-hand pane.
Click on the “Run” button located at the top of the window to execute the code. The first time you run the code, it may prompt you to grant permission to the Google account of your choice. Please provide the necessary permissions to proceed. If you encounter a safety warning, select the “Advanced” option and select “Go to the project(unsafe)”. From there, grant access to the project to proceed with the execution of the code.
After a few seconds, the code will complete its execution. You can monitor the progress from the execution log.
Now if you check the Google Sheet you created, you will notice that a new sheet has been added automatically. This new sheet should contain all the comments that were successfully scraped from the selected video.
Indeed, the process is quite straightforward. To scrape comments from another video, simply copy and paste the YouTube video ID into cell A1 of the Google Sheet again. Then, run the Apps Script code once more, and it will create a new sheet to store the comments from the new video. This allows you to easily scrape comments from multiple videos by repeating the process for each video you want to extract comments.
Thanks for reading ❤
