Scraping Profitwell for MRR using NightmareJS

Oscar Morrison
Jul 30, 2017 · 2 min read

We have a beautiful wallboard that Nelson created with Databox that has all our company’s major KPIs such as user activity, as well as MRR (monthly recurring revenue). Generally speaking, Databox has a good array of integrations and has the ability to connect with most of the services we use, but one Profitwell. We use Profitwell predominately for net new MRR, upgrades, downgrades and churn numbers.

Up until recently, the only way our MRR was updated was when our Head of CS Phil manually went into Profitwell and copied across the current MRR to a google spreadsheet that we fed into Databox.

Solution TL;DR

Use a web scraper to periodically get the number from Profitwell and feed it directory into Databox using their API.

Method

  • First I made a clone of one of my side projects Nightmare on Heroku this is essentially just a build pack and basic setup for using NightmareJS on a Heroku app.
  • Then I went and read the node databox docs and added databox sdk to the project. I created a datasource in databox, and retrieved my token.
  • Next I went and setup an environment file with username, password, and databox token.
  • And then wrote the following node script to retrieve my data and post it to databox:
  • The script should be pretty self-explanatory, but the main parts are that it logs into Profitwell, then it navigates to the growth page, then gets the data from the h1 data that has the current monthly MRR, it then posts this to Databox with js.mrr as the key and the mrr as the value.
  • Next, I pushed to heroku, and setup a heroku scheduler to periodically run the script (and update the MRR in databox).

Done.


Originally published at blog.oscarmorrison.com on July 30, 2017.

Oscar Morrison

Written by

Software Engineer. iOS Developer, Nerd, Chess Player, Motorcycle Rider. WWDC 2015 Scholarship Recipient. ♥︎ Evernote, Raspberry Pi, Arduino

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade