Slice of Py: Web Scraper
I have been building and modifying web sites and apps for the last month daily. Don’t get me wrong I love it, but wanted a small project to learn something new. I gravitate towards Python when not using JavaScript since I want to get better at data analysis and eventually get into programming robots.
Web scraping is awesome. I can’t believe it’s this easy to scrape the internet and get the information in such an organized fashion. Here is the code just to get the html on my personal blog:
my.py
import requestsfrom bs4 import BeautifulSoupr = requests.get(“https://bhankee.github.io")c = r.contentsoup = BeautifulSoup(c, “html.parser”)print(soup)
This will literally output the html for the page and to make it indented and more visual all you have to do is add prettier.
On top of this if you want to look for a div with a class of “awesome” all you do is:
awesome = soup.find_all(“div”, {“class”: “awesome”})This will return only the html with class of awesome, awesome right?
So why is this so cool since you can always right click and inspect for the html?
- Sites that compare prices scrape for data
- To find information on multiple job sites
- anything else that takes data and uses it in a central location for specific purpose
- An excuse to play with Python
This is a really powerful tool for having less then 10 lines of code. There are many other attributes that you can do to specify data and format it in a way that is useful. I encourage you to check it out and think about the ways this could come in handy in your own personal coding journey!
Enjoy your slice of Py!
- Brad
