Extracting Main Keywords from Web Scraped Text with BeautifulSoup, Pandas, and NLTK in Python

Introduction

Web scraping is an essential technique for extracting information from the internet. In this article, we'll demonstrate how to use Python libraries such as BeautifulSoup, Pandas, and NLTK to extract main keywords from web scraped text and insert them into a Pandas DataFrame.

Photo by Amanda Jones on Unsplash

If you are not able to visualise the content until the end, I invite you to take a look here to catch-up!

Setting up the Environment

First, install the required libraries:

pip install beautifulsoup4 pandas nltk

Web Scraping with BeautifulSoup

Let's start by scraping an article's content using BeautifulSoup:

import requests
from bs4 import BeautifulSoup

def scrape_article(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    
    article = soup.find('div', class_='article-content')
    text = article.get_text(strip=True)
    return text

url = 'https://www.example.com/article'
text = scrape_article(url)

Replace https://www.example.com/article with the target article's URL and adjust the soup.find() arguments to…