Divide and conquer — Gather javascript bricks to make the job done

Build the rails and automate the train

In the previous days at MentorLéo, you learnt to build a Node.js server (day 1), to store stuff with a database (day 2), and to make those available online (day 3).

In the #week2 you used superagent to fetch data from a remote server.

It’s time to assemble the bricks.

The 3 weeks of this portfolio project follow the divide and conquer approach. There’s a complex problem: the whole portfolio. The problem has been divided into simpler subproblems:

  • build a simple interface
  • style it
  • fetch job offers

There’s a 4th point which is the reason why we bother doing a backend. In the week 1 & 2 have you seen that everytime you refresh the page, you must wait 2–3 seconds for getting those job offers ? Well, this is not suited for an awesome platform. Because:

Users hate waiting.

The backend’s job is to pre-fetch the data. When the frontend ask for the data to the backend, the data is already available. No 2–3 second latency each time an user land on the page 😍

Part 1: Seeding

Today you’ll build a route GET /seed that wipes the database, fetches jobs from the github api, and stores these jobs in the database.

Here’s how to wipe a collection: https://docs.mongodb.com/manual/reference/method/db.collection.remove/

  • You already know to fetch jobs with superagent (week 2 day 5).
  • You already know how to store jobs in the database (week 3 day 2).

Gogo Léo ! Make stuff happen ! 💪

Part 2: Automation

Allright, you have done a seeding route. But you need to call that route each time you want to refresh the data. How boring ?

The best developers are lazy. They hate doing boring tasks over and over.

Let’s hire a robot. Bots don’t care what they do if you pay them with electricity.

We will use the free plan of Zapier 😎

  • Sign up
  • Make a zap
  • Choose a “Schedule” trigger: we want to automate the action every day (like at 3:00 am)
  • Choose a “Webhook” action: we want to strike the GET /seed route on the production server !
  • Type the correct url/route according to your CleverCloud setup

Aaaaand, it’s done. Thanks Zapier, your product is freaking effective ⚡️

Part 3: Security

Right now, anyone who can guess the seeding route can call it. It’s pretty bad. Imagine someone calls this route many many times each second. It would wipe everything, fetch the jobs and write to the DB. These operations are pretty heavy, and would cause a weird behaviour for the end user.

Let’s add a security token. It is a password that will be only known by the Node.js server and the zapier automation. The way the two will communicate that password is over HTTP headers.

Headers are meta information contained in every HTTP request. They describe the request url, the request method (GET, POST, remember ?) the content type, the content length and various information.

You will use a custom header. Simply a header that only your application will understand. The name below is great:


Here’s how to check a custom header in Node.js:

Run your server, and then type the command below:

curl --header "x-mentorleo-token: MENTORLEO" localhost:8080/seed

This command will strike the GET /seed route on your server with the custom header.

Now, obviously if you take MENTORLEO as a token you’ll have the same security problem than at the beginning since everyone now knows the default password I gave you 😂

Final steps:

  • Generate a password here: http://passwordsgenerator.net/
  • Edit your zapier automation: Add the custom header x-mentorleo-token in the “Webhook” action.

Activate your zap and let the magic happen ! Your server will be updated automatically with fresh data everyday 🚀

The next and final day will be about plugging your frontend with the beautiful backend you built ❤️

→ click here to go to Day 5 👍

Call to action

Join us and get help from professional mentors !

Apply here, it will always be free ♥ https://mentorleo.co

Like what you read? Give Damien BRY a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.