Web Scraping using Beautiful Soup and Selenium for dynamic page

Rahul Nayak
Feb 16, 2019 · 6 min read
Image for post
Image for post

Web Scraping

“the construction of an agent to download, parse, and organize data from the web in an automated manner.”

Why is Python a suitable language to use for Web Scraping?

Why Selenium? Isn’t Beautiful Soup enough?


Installation

pip install bs4 selenium

Selenium for JavaScript link buttons

Image for post
Image for post

For Selenium to work, it must access the browser driver.

Get Trip Advisor review page and click relevant buttons

Beautiful Soup for extracting data

Practical

JOKE of an airline. You act like you have such low fares, then turn around and charge people for EVERYTHING you could possibly think of. $65 for carry on, a joke. No seating assignments without an upcharge for newlyweds, a joke. Charge a veteran for a carry on, a f***ing joke. Personally, I will never fly spirit again, and I’ll gladly tell everyone I know the kind of company this airline is. No room, no amenities, nothing. A bunch of penny pinchers, who could give two sh**s about the customers. Take my flight miles and shove them, I won’t be using them with this pathetic a** airline again.
My first travel experience with NK. Checked in on the mobile app and printed the boarding pass at the airport kiosk. My fare was $30.29 for a confirmed ticket. I declined all the extras as I would when renting a car. No, no, no and no. My small backpack passed the free item test as a personal item. I was a bit thirsty so I purchased a cold bottle of water in flight for $3.00 but I brought my own snacks. The plane pushed off the gate in Las Vegas on time and arrived in Dallas early. Overall an excellent flight.
Original flight was at 3:53pm and now the most recent time in 9:28pm. Have waisted an entire day on the airport. Worst airline. I have had the same thing happen in the past were it feels like the are trying to combine two flights to make more money. If I would have know it would have taken this long I would have booked a different airline without a doubt.
Made a bad weather flight great. Bumpy weather but they got the beverage and snack service done in style
Flew Spirit January 23rd and January 26th (flights 1672 from MCO to CMH and 1673 CMH to MCO). IF you plan accordingly you will have a good flight. We made sure our bag was correct, and checked in online. I do think the fees are ridiculous and aren't needed. $10 to check in at the terminal? Really.. That's dumb in my opinion. Frontier does not do that, and they are a no frill airline (pay for extras). I will say the crew members were very nice, and there was decent leg room. We had the Airbus A320. Not sure if I'd fly again because I prefer Frontier Airlines, but Spirit wasn't bad for a quick flight. If you get the right price on it, I would recommend it... just prepare accordingly, and get your bags early. Print your boarding pass at home!
worst flight i have ever been on. the rear cabin flight attendents were the worst i have sever seen. rude, no help. the seats are the most cramped i have every seen. i looked up the seat pitch is the smallest in the airline industry. 28" delta and most other arilines are 32" plus. maybe ok for a short hop but not for a 3 or 4 hour flight no free water or anything. a manwas trying to get settle in with his kids and asked the male flight attendent for some help with luggage in the overhead andthe male flight attendent just said put your bags in the bin and offered no assitance. my son got up and help the manget the kidscarryons put away
I was told incorrect information by the flight counter representative which costed me over $450 i did not have. I spoke with numerous customer service reps who were all very rude and unhelpful. It is not fair for the customer to have to pay the price for being told incorrect information.
We got a great price on this flight. Unfortunately, we were going on a cruise and had to take luggage. By the time we added our luggage and seats the price more than doubled.
Fun crew. Very friendly and happy--from the tag your bag kiosk to the ticket desk to the flight crew--everyone was exceptionally happy to help and friendly. We find this to be true of the many Spirit flights we've taken.
Not impressed with the Spirit check-in staff at either airport. Very rude and just not inviting. The seats were very comfortable and roomy on my first flight in the exit row. On the way back there was very little cushion and narrow seats. The flight attendants and pilots were respectful, direct, and welcoming. Overall would fly Spirit again, but please improve airport staff at check-in.

Conclusion

YML Innovation Lab

Imagine a non-billable team that’s daily RFP is to…

Rahul Nayak

Written by

Just a guy

YML Innovation Lab

Imagine a non-billable team that’s daily RFP is to experiment, break things, and embody the bleeding edge of technology. Enter YML’s Innovation Lab; a team of AI and machine learning experts dreaming up the future of technology.

Rahul Nayak

Written by

Just a guy

YML Innovation Lab

Imagine a non-billable team that’s daily RFP is to experiment, break things, and embody the bleeding edge of technology. Enter YML’s Innovation Lab; a team of AI and machine learning experts dreaming up the future of technology.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store