One of my final projects for Flatiron is to build a CLI. The major requirements were to have a CLI, pull data from an external source using Nokogiri and implement both list and detail views of information.
On my first day of the bootcamp back in March I began contemplating what I would do for this project. After a few weeks of working thru the material I had decided that I would use Pollen.com and pull the Pollen levels and types.
Fast forward to the end of August. I was very excited to make this gem because I know that it is something I would use everyday. I built out my CLI with fake data to make sure it worked, built out my classes and was excited to see that objects were being returned! WOO HOO!
The excitement was short lived. Little did I know I made a serious noob mistake. I was 99% done with my gem. All I had left was to scrape the data from Pollen.com. I had just spent my whole Labor day weekend getting this far, but I still had a whole day Monday to get the last bit done.
I got my coffee and settled into what I thought would be a easy session. Hours later and multiple parties involved my project was dead. Pollen.com at the time of this post is written in an early version of Angular.
When I was asking Nokogiri to scrape the data I needed from the webpage there was a lag time between when Pollen.com would load the data for the day. This resulted in the data not being pulled over for me to scrape. You can see the data in the picture on the left. Its there … but in my terminal below there is #(CDATA….).
The lesson here is start with scraping… if you can’t scrape what you want then why build out a gem?
I took a long walk and cleared my head. I began to think about what I was going to do instead.
I decided that I would build out a gem to get the newest releases from one of my favorite authors websites. And the first thing I would do is to see if I can scrape it. After a little trial and error I had the info I needed. I then began to build out my CLI.
Since I work full time it took me another week to build out the rest of my project.
The most important thing I learned on this project was to start first with the unknowns. I knew that I could build out a CLI as I had done so already. I did not know if the site I chose for my first project was scrapeable. Lesson Learned!
Here is a link to my github repos if you would like to see either project.