Some Optimizations…
Note: This post refers to an unreleased version of Keep (3.3.5). Expect it during the week of June 22.
When you save a link into Keep, it gets passed to a NodeJS server and the server makes an additional request (before saving) to fetch metadata associated with the link. The server uses a library called Metascraper to get the title and thumbnail image you see in the app:
In most cases this works fine (and fast), however I’ve noticed that for certain sites / links Metascraper takes a long time to fetch associated metadata and response times can exceed 5 -10 seconds (not ideal and very annoying as a user). Additionally it was failing at getting a thumbnail image URL for Instagram and Twitter links:
To be fair, I also tested out some of the dominant read-later apps and I don’t think I’m unique in this struggle. I saw similar results for Twitter and Instagram from many other apps.
I started looking into whether I could fetch a links metadata on device and just pass everything to my server. First stop was Apple’s LinkPresentation framework. LinkPresentation is responsible for the previews you see in iMesssage; for example, when you send someone a link:
Great. I thought this would work amazingly well until I started digging into the documentation. While LinkPresentation seems to do a really good job overall, there’s no way (as best as I can tell) to obtain the preview images source URL. LPMetadataProvider gives you the image’s data file. Keep is setup to store the thumbnail’s URL and the image is downloaded and cached by the app when browsing links.
So I searched a little for an iOS library that could grab a links open graph data. Surprisingly not much out there but I did find one that seems to work pretty well. It does a good job with Instagram links:
But…it didn’t fair so well for Twitter links. Furthermore, Twitter was also proving to be a problem for Metascraper on the server unless I set an explicit user agent for the scrape request. Getting a simple thumbnail image and title for all links was starting to turn into a game of whack-a-mole.
So there doesn’t appear to be just one tidy way of doing this.
There are no hacks, just less elegant solutions 🤦🏻♂️
Where I landed (for now) is that the app will attempt to fetch open graph data and, if present, will pass it to the server to get saved along with the original URL. If the app is unable to get a thumbnail and title, the server will then try to use Metascraper as a backup.
Result
So far while testing, in the majority of cases, saving links is now much much faster and an appropriate thumbnail and title are obtained for the majority of saved links. So it’s a win on that front.
Amazon.com links are still causing me problems and I suspect I’ll continue to discover new sites that pose problems. I suspect this will turn into a never-ending chase.
*It’s possible I’m using Metascraper or another library incorrectly. And to be honest, a lot of this web scraping stuff is over my head. I expected that fetching OG data on device would be slower than doing it from the server but I’ve found that even on poor network connections, doing it from device resulted in faster save times.
