Is automating tasks significantly faster with AI ?

Recently I wrote that I am tracking the prices on some items from a few local online stores. The gist is, there is a small web app in which you can submit URLs you want tracked and every 24 hours it will create a new datapoint with its current price. The app works for 5 local shops and had arround 100 products during it’s first year. Currently, I am interested in extending the app’s usefulness with using as little hours as possible.

This article tries to challenge the concept that if you have partial solution and AI you should aim to automate more tasks than you normally would. Please see the image bellow:

The image still mostly makes sense, however I want to highlight that with AI we can move faster.

The price tracking software needs more products so that the dataset we are creating is more useful.

Naturally the question was “What is the most efficient way to extend the dataset ?”

Manual inputs are what I tried initially and it took too much time. Also it was a very annoying, uninspiring way of doing things. Hardly how I want to spend the very limited time I have to work on this project.

Manual inputs and baseline

I tried to gather some friends to help with manual inputs, that didn’t work.

I am painfully aware that you can spend a lot of time automating a task that needs to be done infrequently. This case is exactly like that. Once the product is being tracked we don’t care how it got there.

In two sessions I added products. The number of tracked products jumped from 100 to 300(ish). Adding the 200 products took me around 2 hours.

Can I do better with automation ?

I decided to challange myself and to see if I can write a script that will add more than 200 products in 90 minutes.

Yes, I am aware that when the script is done it may theoretically add thousands of items much faster.

I am interested if it may also beat my manual inputs!

Interestingly enough the answer is — yes.

I had already part of the problem solved in the main app — namely scraping webpages and extracting data from them.

Roughly what I had to do was:

1 Build a script that would scrape all products from a webpage

2 Make the script call the API that will start tracking them daily

3 Ensure that the functionality in 1 and 2 is encapsulated so that I can call with with an array of pages

The script was written in typescript mostly because I managed to reuse parts of the project that was also written in Typescript and NextJS. The pages for the websites are static so a normal node fetch worked combined with cheerio for parsing the html.

All in all, fairly straightforward. The script was done in arround 40 minutes. It would have been done faster if I had tested a bit earlier during the development process.

One thing that could be improved is the error handling in the script — in the timeframe I had I managed to build the happy path and have very basic validation. This is fine because the script was intended to run for a couple of hours and automate manual work, but if it was something that would have bigger scope like importing hundreds of thousands of products from different sites more time would need to be invested into hardening.

All in all for 70 minutes I managed to insert 900 products. Beforehand we had arround 300. Great numbers.

Manual input: ~2.2 products/minute
Automation: 12.8 products/minute

Note: The products I want to import were finite, the larger the dataset you want to import the higher the automation rate would be as most of the initial investment in writing the script would pay for itself.

I wouldn’t call this article conclusive. This is an anecdotal example that you may want to opt for at least partial automation.

The takeaway

First and most importantly you need a baseline with which you can compare things.

Secondly you need to understand which parts will benefit the most from being automated and which parts should remain manual. If I had tried to automate the page generating process instead of providing the list (mostly) manually I would have used up more time which would have made the gain from automation less. In this case the Paretto principle, or the 20/80 rule was to skip automating the page generation.

The example with the script is anecdotal, but the principles described in this section are applicable to most tasks that can be considered for optimization.

Next steps ?

The price tracker tracks over 1k products, which means that we should expect more than 400 000 datapoints over the course of 2024. The dataset is a few megabytes large right now. I will avoid optimizing it (or sharding it) if possible for the entirety of this year. Not sure if I can justify the time investment, even if it is just a couple of hours.

I considered adding more products but when I looked at the largest local website I saw hundreds of thousands of products. Tracking them is a task outside of my time budget. It will be great to find a way to extend the dataset with diverse products. Let’s see if I figure something out.