Get Data from the web with Web Scraper

Intro

There is a huge amount of data online that is accessible with just a few clicks. Web scraping is the process of automating the manual search part with the browser and instead just scraping the info you want from each website. In the tutorial, I will be demonstrating how we can achieve this in Flutter to in the end create an app that displays the world’s current population.

final product

Tutorial Portion

The first thing is first, huge shoutout to https://github.com/tusharojha for creating the package https://pub.dev/packages/web_scraper that we will be using for the tutorial

Setup

The first thing we have to do is add the package to our flutter project in the pubspec.yaml, The link for the dependency can be found here https://pub.dev/packages/web_scraper#-installing-tab-. Our pubscpec.yaml should look like this:

impoted web_scraper in pubspec.yaml

In main.dart we can delete the MyApp class and we will recreate it as a statless widget that returns a MaterialApp, (optionally already set ‘debugShowCheckedModeBanner: false’ to not the debug banner):

creating MaterialApp

We will then create our Homescreen which will be a stateful widget, that has a scaffold with a center widget, for now, the widget will just be a circular loading indicator:

creating HomeScreen

Dont forget to set the home as HomeScreen() in main.dart

setting HomScreen as home

If we run this there is nothing special just the progress indicator:

loading indicator

Lets now add some variables to keep track of the state of the app; inside _HomeScreenState. We will also create and import our WebScraper now

instantiate variables

Then for initializing all the variables we will first override setState and call an async method that we created called _getData(). We will handle all the asynchronous calls in _getData()

create _getData() function

We will now build out the _getData() method: we need to create a webscraper with our desired wesite url (in our case‘https://worldpopulationreview.com'). Then we want to check if it can load our desired page (since our website has the data in the main page this will just be ‘/’).

setting website checking if page loads

HTML Time !!

Now we can get what we want from the webpage. We need to first open the page in a browser and inpect the elements (in chrome cntl+alt+J)

using chrome inspect tool to get elemnet’s name

We can get the element’s name from the first options in properties. For the one which we want it is a div of the class center so our name will be div.center. We create a list of maps to get all the elements that match this name:

creating a list of maps of html elemnts with specified name

Once we have that we can call setState and set our popNum that we defined earlier to the first element’s title that we get and set the bool loaded to true so the whole method should look now like:

setting new state with popNum as the first result

Bringing it all Together

The last thing we need to do now is stop the progress indicator when we get the result and display the popNum in a Text widget instead. We can do this by using the dart ternary operation to check if the loaded bool is true, and show the Text if it is true.

ternary operation for setting text or progress indicator

So now our app should open with the loading indicator and then show the text we scraped once it is ready

final result

Conclusion

Now we can easily scrape any static elements from websites with HTML. This is particularly useful if we want to, for example, compare the first price result of each seller’s website and show them all, we would have to create a getData method for each site we are scraping then return it in a list.

For the souce code: source

Originally published at https://www.dailyflutter.monster.