Autobot in action: taking results in saved searches and notifying me in Slack.

Separating the Notification Wheat from the Chaff

Sending carsales.com.au alerts straight to Slack with a little Zapier magic.

In explaining why I feel so strongly about ad blockers, notification spam and content marketing, I wrote a rather lengthy diatribe about what it feels like to be both an end user and front end developer in the industry. It didn’t feel right to leave this in, as it implies I was talking specifically about carsales.com.au. They are instead merely yet another notification and email in my inbox I want to automate away for good.

Put simply:

  • I want to know when something happens
  • You want to up sell me
  • Your advertisers want to influence me
There are two types of notifications in this world: those that you originally wanted to see, and those that a product owner wants you to see.

It’s an ambitious but naive goal, but I want to remove as many touch points with your business as possible, and receive information and alerts on my terms. I don’t care that your real estate company now offers home loans. Besides the possible conflict of interest, I know that it’s not for my benefit, but for yours. And hey, didn’t I sign up when you had houses in my budget and in my area?

A more traditional approach to the problem would be running some Node.js scripts on CRON jobs, scraping the target website and checking for new matches, firing them off as a tweet or PushBullet notification. Indeed this is something I’ve done in past while hunting for houses:

But a couple of things are happening in this space.

Slack is the new PushBullet is the new Twitter Bot

An increasing number of colleagues and friends are turning to personal or family Slack instances. Even on a free tier you get an unlimited amount of webhooks, with each one easily personalised via display name and avatar. Slack gives you all the nice things we’ve come to take for granted like link unfurling, media embedding and the like. And that’s not to mention the enormous amount of bots and integrations that you can add on to further your personal digital assistant goals.

Scrapers are Out

Websites are also pretty watchful for web scrapers, even if only targeting the SaaS ones for now. If the key value your business provides is aggregating information, and someone can scrape that for free, it makes sense that you want to protect that at all costs.

I was never lucky enough to try Kimono before it was shut down, but sites like carsales.com.au have blocked just about every service that has popped up in their wake.

Using Email Alerts Instead

The idea I had was to instead repurpose the antiquated but still somewhat useful feature of email alerts. There are no RSS feeds. There are no mobile push notifications. We make do.

The Solution

In brief, users can create up to 50 alerts on new carsales.com.au listings which can be sent to your email address near immediately. Zapier can watch your inbox for new emails matching particular filters. Zapier has its own email processing engine, which for this purpose, is garbage (more on that later). Zapier does, on the other hand, have a JavaScript task which acts similarly to a FaaS (functions as a service) invocation like AWS Lambda. There’s also a Slack integration to push out your results.

Using Zapiers Gmail, JavaScript and Slack Integrations in series.

A Note on Free Tier Limitations

From what I’ve seen so far, I really like Zapier. Mistakenly, I built the first version (pictured above) using the pro user “multi step workflow” on a trial membership, and then had to pare it back. I’ll explain later on.

Set Up Your Alerts

The most obvious first step is to configure your alerts on carsales.com.au. It should be as simple as performing your search and then following a “Save Search” or “Receive Alerts” prompt. Ensure you elect to receive your notifications immediately.

Configure Your Email Trigger

In Zapier you must create a new Zap with an email based trigger. This is where Gmail really shines, due to their great search syntax and more advanced filter options. Our use case is relatively simple.

Gmail users are specifically looking for “New Email Matching Search”.

And, once connected to your specific account, we’re looking for the particular filter below. This handles the three particular situations in which there is only “1 car found”, multiple “x cars found”, and when you have a multitude of results across many saved searches “x/new cars listed in your saved searches”.

subject:”car found” OR “cars found” OR “cars listed in your” from:carsales.com.au

Email Parsers Won’t Help Here

Throughout this process I tried countless email parsers. They ranged from the simple but elegant one baked into Zapier, to the supposedly machine learning powered algorithms. They all fail a pretty simple use case, repeaters.

Email parsers are primarily geared towards pulling out key information from legacy email formats, generated in plain text from servers running in someone’s basement. Zapier makes an enormous amount of sense here, where no or low-touch solutions mean you can pump an old email format through a parser, grab out the data fields you need, and then re-send a better looking email through Campaign Monitor or Mail Chimp. Quite amazing, actually.

But the moment you have dynamic data, say three cars instead of one, these tools might as well not even bother.

“JavaScript” Zapier Action to the Rescue

As mentioned earlier, Zapier has the ability to run a JavaScript action as a step in a Zap. Considering Node.js does not run in a browser environment, it has no need for a DOM parser. This is where tools like cheerio would normally come in, allowing you to parse a string of HTML and query it via a jQuery like syntax.

Except, Zapier’s FaaS environment is locked down, with no packages besides node-fetch to polyfill the ES fetch function, as well as strictly disabling npm. Drat.

Filthy Regex It Is

Stack Overflow was adamant that you should never parse HTML with RegEx. So of course I did exactly that.

Configure these two input fields to allow your code to parse them

Once you have configured your two input fields, namely the email subject and body, you should be able to use the following code. Replace with your webhook URL obviously.

Sending to Slack

The eagle eyed amongst you may have noticed that I post the result to Slack from directly within the code here. This is because after my trial expired the pro feature of multi-step Zaps was removed. Knowing full well that webhooks can be invoked from pretty much anywhere it was simple enough to do it from our function.

Free users need not apply.

In The Flesh

At some point in time between writing my paid Zapier version and my free Zapier version, I appear to have broken the link unfurler. The unfurler takes the URL, visits the site, and grabs key information like an image preview, the car location and price. Here it is when it was working.

And here it is more recently, with the unfurler remaining firmly furled.

I should probably fix that.