The Startup
Published in

The Startup

The guy in the chair’s name is Web. Photo by Adam Sherez on Unsplash

Web Scraping All the Links With Python

pip install beautifulsoup4, requests
from bs4 import BeautifulSoup
import requests
URL = "https://www.comnetwork.org/newsletter-archive/"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
for a_href in soup.find_all("a", href=True):
print(a_href["href"])
archive_links = ["http://mailchi.mp/4bd5002a15e1/february2018", "http://mailchi.mp/comnetwork/16may2018"]
for link in archive_links:
page = requests.get(link)
soup = BeautifulSoup(page.content, "html.parser")
for a_href in soup.find_all("a", href=True):
with open("newsletter_links.txt", "a") as linkfile:
linkfile.write(a_href["href"]+"\n")

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Python Is Rad

Python Is Rad. I’m a software engineer with an addiction to building things with code. https://twitter.com/PysparkIsRad