Node.js + Typescript + Nightmare

I hope the title does not stand up for mistakes: Nightmare is a is a high-level browser automation library. Once I’ve made clear what Nightmare is, I’ll add that working with Node.js and Typescript is a pleasure. I come from an object-oriented programming background, and Typescript makes it much easier to program an application in JavaScript (Node.js), as I can develop code using the functional or the object paradigm (or both).

[DISCLAIMER: don’t expect any more explanation about what is Node.js, Typescript or Nightmare, or how can you install them … google it]

I needed to develop an application for scraping dynamic websites, so I decided to opt for a solution that would allow me to load the web page into a “container”, and then access that container to extract the data. Before I choose Nightmare, I tested Phantom.js; it’s the most recommended solution for Node.js for scraping scenerios. But the way it works was not in line with my way of looking at things (custom shell made difficult to debug, light integration with Node.js application, …), so I threw away first prototype 👎

After further research, I found Nightmare, which runs an Electron instance behind (Electron’s engine is based on Chromium). Nightmare is totally transparent and fully debuggable and overcomes many Phantom.js limitations. 👍

Nightmare API is based on Promises. I had never worked with promises before, but I did work with similar patterns in other languages (C#/python async). I performed some basic tests on a Node.js shell, and results were promising. Then, I developed a basic programmable scraper console app using Nightmare, and results were still good. 👍

Nightmare has a “unique” way of working: actions are queued and “theoretically” executed when you run a command then, run or end. Easy, right? Well, no. As soon as you need to perform a non-sequential flow, different from the one explained in all the tutorials, things start getting dirty, and actions never execute whenever you want. I didn’t know if the problem was my lack of experience with promises or lack of documentation about Nightmare command queueing and execution (or probably, both of them). 👎

Looking all over the Internet, I found several users who use Nightmare along with vo, and seemed pretty happy. Hands on, I adapted my code to use vo …. bingo! 👌

vo solves promises problems using generator functions. Technically speaking, generator functions sit between promises and async functions. I will show how I used vo and Nightmare with some demo code:

The basic Nightmare functions are commented inline. I will claim reader’s attention to the function scrape, which contains the main workflow (generator) function. In this function, I use an if-then-yield statement which “breaks” the sequential chain of promises. Also we can check how to call another workflow contained in another function (urlScraper), passing as parameter the Nightmare instance; this function uses a while loop, which flow was not feasible using only Nightmare, but it solves easily using vo.

So my recommendation is simple: use vo together with Nightmare if you need to perform complex flows. 👏