Python Web-Scraper Demo

Its me again!!! This time I needed spend more time on my Python and I wanted to do something other than the usual snake games and number games. This time i chose something that is dear to my heart….Pokemon(admit it you all love Pokemon. We all wanted a Mew)

This wont be a long tutorial. I will just be explaining the code I chose to use and also talk about some the issues I ran into while attempting this project before using our secret weapon…Google.

What I Used

VSCode was my IDE

Python and a few libraries ( Panda, Beautiful Soup, requests)

Note that the above Python Libraries and functions must be installed.

The source code can be found on my GitHub @ https://github.com/DuranIsrael/python-webscraping-demo.

As you can see it does not require a ton of code to create a web-scraper. I also learned there are a quite a few different ways to do this but the above was the easiest for me to sort out(I wanted to get as far as I could on my own before researching help. I call it “Python Logic Training”).

Most things are explained in the code but I do want to explain the libraries I used:

Pandas: Pandas is an open source Python package that is used for data science/data analysis and machine learning tasks. I used the Panda package to generate my csv file that holds my results.

Requests:The Requests function allows you to send HTTP/1.1 requests using Python. Basically it allows you to get the information you need from a source.

Beautiful Soup: Beautiful Soup is a Python web scraping library that allows us to parse and scrape HTML and XML pages. You can search, navigate, and modify data using a parser. It’s versatile and saves a lot of time. (www.topcoder.com)

As you can see I chose a PokemonGo Trainer site for my project since, well I frequently enjoy playing PokemonGO. Since one of my issues was linking up with other trainers I figured why not solve my problem and get some practice in!?

The Results!

Boom! Plenty of trainers to add to my friends list(the website chosen are a list of trainers are do not mind adding friends, which is why they signed up for the website. So its ok.)

Issues And Conclusion

So one thing you all should know is that you CAN NOT scrape just any site. Sites like Amazon has a blocker that will block you from doing these things….or its beyond my current skill level.

One of the major issues I ran into was reforming my logic based on understanding what I can or cannot do with these libraries and functions. I was able to request the basic information, but when it came to try to narrow it down to just the trainer name and trainer code I struggled (thats when I called on the power of Google). So I Googled as well as watched tons of tutorials to make sure I get a good understanding of how to correctly invoke commands.

So that is my first ever web-scraper using Python! Next I will take it up a level and see what other problems I can solve using Python! Hey if you have any pointers or advice to make this better send me a message and lets grow together. Remember, iron sharpens iron!