Create your first Minion with Project Enferno

Dave.. the minion

This is a simplified tutorial to help you get started with project Enferno, in this article you will learn how to use some cool features like periodic background tasks, the Mongo Engine ORM, and the Mail functionality included in the framework.

Unlike the original Minions created by Dr. Nefario, our minion is a piece of code that will be periodically performing some tasks for us.

So what will our Minion do ?

It will help us find the best apartment in Berlin ! how ? it will check Craig’s List website periodically, import the most recent 100 apartment listing, and email us the best available options. Let’s get started !

Download and install the framework

You can get it from github or you can follow this article.

Create the “Apartment” model

Create a “models.py” file inside the public directory, inside that file let’s create our “Apartment” model, add the following code:

from extensions import db

class Apartment(db.Document):
    title = db.StringField()
    price = db.IntField()
    url = db.StringField()

Working with Celery background tasks

By default, Enferno ships with a powerful task queue already pre-configured for you. in order for our Minion to check Craig’s List, we need to do the following:

1. Scrape the website and parse the listings.
2. Save the listings into our Mongo Engine model.
3. Perform the above periodically

For the first task, we are going to use a very handy library called “pyquery”, it will help us get parts of the page HTML using jQuery-like syntax.

Install the pyquery library by executing the following:

pip install pyquery

The “tasks.py” file is the place to have all your periodic/background celery tasks, let’s add our first task there. replace your “tasks.py” file with the following code:

# -*- coding: utf-8 -*-
from celery import Celery
from celery.task import periodic_task
from datetime import timedelta
from settings import Config
from mongoengine import connect
celery = Celery('tasks',broker='redis://localhost:6379/2')
from pyquery import PyQuery as pq
from public.models import Apartment

URL = 'http://berlin.en.craigslist.de/search/apa'


db = Config.MONGODB_SETTINGS['DB']
connect(db)

@periodic_task(run_every=timedelta(hours=1))
def get_apartments():
    page = pq(URL)
    page.make_links_absolute(base_url=URL)
    for row in page('.row'):
        apartment = Apartment()
        apartment.url =  pq(row)('.pl a').attr('href')
        apartment.title = pq(row)('.pl a').text()
        apartment.price = int(pq(row)('.price').text()[1:])
        if len(Apartment.objects.filter(url=apartment.url)) == 0:
            apartment.save()

Pretty simple ! with pyquery we are able to parse different components of the HTML and save them directly into our Mongo Engine model, we use the URL as a unique identifier to make sure we won’t import the same apartment twice.

By adding a simple decorator “@periodic_task” we tell Celery to run this task every hour in the background.

Sending the reports

For the sake of learning simplicity, our minion will grab the 5 cheapest apartments and send them to us. let’s first configure the mailing functionality that ships with Enferno. open the “settings.py” file and change the “SMTP” settings to your own:

MAIL_SERVER = 'smtp.gmail.com'
MAIL_PORT = 465
MAIL_USE_SSL = True
MAIL_USERNAME = 'user@gmail.com'
MAIL_PASSWORD = 'password'
SECURITY_EMAIL_SENDER = 'user@gmail.com'

Let’s create another periodic task, that will send us the cheapest five apartments every four hours.

Add the following code to your “tasks.py” :

from flask_mail import Message
from extensions import mail
@periodic_task(run_every=timedelta(hours=3))
def send_report():
    top = Apartment.objects.order_by('price').limit(5)
    email = ''
    for a in top:
        email += '\n -------- \n %s - %s %s' % (a.title, a.price, a.url)

    msg = Message(
            subject='Apartment Report from your Minion !',
            sender = 'user@gmail.com',
            recipients=['youremail@gmail.com'],
            body=email
            )
    mail.send(msg)

That’s it ! simply run or deploy your app, and your Minion will be working for you while you’re asleep.

Here is a screenshot of the report I got:

Think about other possibilities you can achieve with this, like downloading some cool torrents, or watching bitcoin price changes, or finding some great job listings !

It’s time to put your minions to work for you !

You can download the source code from Github.

Hints:

• To run Celery task queue, make sure you activated your virtual environment, then execute the following:

celery -A tasks worker -B 

• You can test specific celery tasks manually by running the Enferno Shell, for example:

./manage.py shell
from tasks import get_apartments
get_apartments()