crontab, wget, Dropbox, and ifttt: updating files from the web with relative ease
This is for you if: you’re a developer who gets her hands dirty on some code, but you’re not above some simple, abstracted automation.
If you’re an transit nerd like me, you always like to have the latest data from your local transit operator. How else are we going to find out the latest service schedules? Surely not by checking a printed brochure or website!
Or maybe you’re interested in some other file on the web (though most things are RSS’ed, nowadays) that may be updated without you knowing. I for one am of the xkcd school of thought and I will always opt for automation over manually checking said file once every so often.
Because the Internet, there has to be a nice little service that checks a file’s last modified date and sends you an email if it changes, right? Well, I couldn’t find one after extensive (read: 7 minutes) of searching.
So I put together this little process that might be of help to you (for you Windows users, you might have to take a few extra steps). It kind of depends on you using some basic Terminal commands as well as storing the file on Dropbox. Oh, and you need ifttt and an email account, too ;)
Setting up the cron job
1. Open a Terminal window and install wget from homebrew (OS X only, Linux users should already have this command).
brew install wget
2. Navigate to your Dropbox folder of choice (ideally make a new folder just for this file) and use wget to download the file.
In the same window, try running the following wget with timestamping to see if the file updates. If it doesn’t, good. That means the file hasn’t been updated on the remote server yet.
wget -N http://url.of.file
3a. Create a new cron job.
3b. Add the following cron job entry (tap i to edit entry). The following will run the task every day at 12 pm.
0 12 * * * cd /path/to/Dropbox/folder && wget -N http://url/of/filename.ext
(to exit crontab editor, type ESC and then :wq) More on creating cron jobs.
Turn Dropbox into RSS and then into Email
Now that you have your file in Dropbox, you can check out https://www.dropbox.com/events to see all changes to your Dropbox files and view an RSS feed of these events.
This next step is a little janky, but I promise it works! In order to filter out only those changes to the particular folder you’re interested in, you’ll need to share the Dropbox folder with yourself. To do this, just right-click on the folder in Finder or your file system viewer and select Share this folder on Dropbox… That folder will then show up in the dropdown list like the one shown below.
Now, click the orange RSS button to get the link to the filtered RSS xml. Copy and paste this into the Feed URL for the Dropbox RSS to Email ifttt recipe.
That’s it! Now you’ve got a hands-free notification straight to your inbox every time that file is updated.
As mentioned above, you can even modify how frequently you check with the crontab settings. And, if you’re interested in archiving, you can add some additional commands to the cron job:
0 12 * * * cd /path/to/Dropbox/folder && wget -N http://url/of/filename.ext && cp filename.ext filename_$(date +%F-%T).ext