Flutter web scraping in practice

Image credit: @mamatha.bbb article

In this article I am going to introduce you to web scraping in Flutter/Dart

Some time ago I started developing this package chaleno, at that time I wanted start using web scraping in Flutter/dart. I developed the first version of the package but stoped and now that we have Dart in backend growing fast I believe is a good time to keep working on it.

This package contains a set of functions that make it easy to web scrape websites. Its multi-platform, supports mobile, desktop, and the web.

I’m still working on it and there’s features missing so feel free to give your contribute 🙂❤️

Now that already we met each other let’s jump to the code 💪🏿

First you need to add the package in your pubspack.yaml

chaleno: ^0.0.6

This is the last version so far, make sure to always check the pub dev to get update about the last version.

Once it’s installed you can import it like this

import 'package:chaleno/chaleno.dart';

Now we’re ready to go

As example we’ll use Filipe Deschamps Newsletter

If you like to read about technology this newsletter is for you! 🙂 I really love it. It’s only available in Portuguese but I believe will have english support soon

We’ll extract from this newsletter:

• Avatar url
• Newsletter title
• Subscribers number

A quick introduction to Chaleno package

To start working with chaleno, we need to create an instance of Chaleno like this.

final chaleno = await Chaleno().load('https://example.com');

Now you can use chaleno object to fit your needs

As you can read here chaleno#importants-methods-and-propriets this package offer us a bunch of method and properties that we can easily use while web scrapping.

We have different ways to extract data with this package

• Get an element by id
• Get elements by class name
• Get elements by tag name
• Get an element by selector
• Get elements by selector

Extracting data

You can see here different ways to use those methods

Different ways to extract data using chaleno package

It return a Result object that contains some nice properties that you can access through your past query.

As you can see there are different ways to extract data using chaleno.

And you can access Result object properties like in the example below. You can take text, src, href, html

Cool 😎 Now you’re ready to play around with this package

Credit: https://giphy.com/gifs

Now that you know how to work with chaleno package we can start our little example (web scrape Filipe Deschamps newsletter)

First lets create our flutter project by running flutter create newsletter_scrapping

After that let’s open our main file, delete all unnecessary code and let this file like this

I added a HomePage class to our MaterialApp(home:HomePage()) , don’t worry we’re going to create it now

I added our HomePage and I took the opportunity to stylise to looks like with the Newsletter Filipe Deschamps. Basically I added an image, a Text, TextField and a button. Note that those values are hardcoded, it doesn’t come from web page yet. we’ll add the method to do it now.

That’s how it looks

Now let’s add a method to load the newsletter and extract the avatar, title and subscribers number. First we’ll need to inspect the page to see which selectors to we need to access to extract the avatar, title and subscribers number, when we click on inspect element we’ll see exactly what’s selectors we need

Inspect element result, we can see the selectors

We already now which selectors we need so let’s start extracting data

And that’s it 🙂 super simple.

We added scrapData method responsible to load the url (https://filipedeschamps.com.br/newsletter) and extract the avatar, title and subscriber number. After that we call this method on initState method.

We can see the final result, avatar, title and subscriber number extracted from url.

Here’s the complete code of our HomePage

That’s chaleno package, hope you enjoyed this tour 😊

If you liked this content left a clap 👏 and follow me for more Flutter tips 🙂