One year of tracking prices of local online shops — the results
Tl;dr: Last December I wrote and deployed a price tracker for a few local sites. This article explores the build process and the results of the data.
The Results
With that out of the way, let’s talk about results.
There are almost 29 thousand datapoints over 97 different products.
The statistics were made by using plain ol SQL queries as the datapoints are stored in an SQL database.
Note to self: add a few hundred more products.
The interesting thing is that I can make the same conclusions for all the websites!
Conclusions:
1 Contrary to popular belief, prices don’t go up before Xmas or Black Friday :) What happens is that everything is at the baseline, or basically “highest” price point and nothing is discounted (or discounts are tiny, like 5%) which may cause the feeling that everything is more expensive than usual as nothing is discounted.
2 Prices don’t go up a lot (during the span on 1 year) when they are introduced. Some products may go up, but it isn’t drastic
4 There are usually 2 discount types — a small one and a big one. The exact % depends on both the shop and the type of product.
5 Both discount types may have slight fluctuation, but not by much.
Price tracking 2024 ?
The biggest thing is to add more datapoints.
Finding a way to map the same product automatically between several websites would be good, but so far I have failed to find a sturdy way of doing it.
I don’t think it makes sense to add more shops as I have already added the biggest ones and in general the ones I am interested in.
Adding more metadata to the products, like adding their categories and so on might be a good choice for being able to make better conclusions. That data won’t change over time so in theory it can be gotten and enhanced at any point in time.
The Build Process
<Unless you are interested in a high level description of the tech stack and development timeline and app features skip this section>
For some reason in May 2021 I decided to try and write a price tracking app in one sitting.
The stack was Express (with plain ol JS) and no frameworks for the client.
In one sitting of a couple of hours I managed to create a web form in which you input a URL to one specific local online shop, the webpage gets scraped, the price gets filtered, stored in an SQLite Database and returned to the web UI which will then display it as a list.
Pretty much looks like this:
I don’t really remember what my idea with this was, but it never went anywhere beyond that.
Fast forward to December, 27th 2022. I decided to recreate the project from scratch using NextJS, supporting multiple websites aaaand to actually deploy it.
In a couple of sessions we had a NextJS app that scrapes data from 4 online shops, 2 of which are the biggest ones in the country (as far as I am aware).
The app is deployed on a Digital Ocean Droplet, is in NextJS with TypeScript and strict mode on.
The main features are, once again — you input a URL, it enters the DB and everyday a background job scrapes the website and makes a new price point.
This is how the starting screen looks:
And this is how an already tracked product looks (Your URL is usually the real URL, I edited it out).
Main features are:
- Track the prices on a daily basis
- Showcase the highest and lowest price
- You may put your email to subscribe for email updates
Yeah, the UI isn’t the best, but I am not really a FE developer :)
For the Database I used SQLite again as it doesn’t have a server and made deployment much easier. The ORM was prisma.
Scraping the prices is pretty straight forward — Since all local websites are dynamic I need something that runs JS and returns the fully loaded page. Cheerio does the heavy lifting for this. From there on it is a matter of writing the correct selector.
In short — it is a small app that was quick to write and uses the simplest technologies that get the job done and I was interested in using.
The code sadly isn’t open source. I might open source it next year, we will see.
