How to fetch your Medium feed in 17ms using Cloudflare Workers
Getting more than just your latest 10 Medium posts in JSON format
TLDR: Live demo. Repository.
Why do I need to fetch a Medium feed?
Having a cup of warm coffee, 36 open tabs on Firefox Nightly and another 17 on Postman for research and testing while writing about topics that I find fascinating, such as infrastructure and APIs is something I always crave for.
On the other hand, because I want to be somewhat updated with web and software tendencies I usually go to Meetups, workshops and other events.
And often, the best content I get from these places are photos, interviews, a link to video or talk, not necessarily in a text format that I use on Medium.
Even if it is written material, like a code repository, I find that Github is much more suitable for sharing technical content because you can make use of the actual versioning of the code and because the industry is already familiar with looking into repositories for code.
Lately I’ve been playing around the idea of exploring new content formats, such as videos or streams, given the fact that different ways of telling an idea helps the viewer/reader to get a better sense of it.
You can’t really replace seeing the wonderful work flow of a Twitch streamer like Noopkat by only reading curated documentation of the technologies she uses.
I find that the escense of the informal, spontaneous thoughts is sometimes lost in written articles.
What does this have anything to do with the post, you may very well be wondering.
And that’s a good question.
typeof content !== ‘string’
Planning to create content that is not only written, I can’t have it all on a platform which main’s format is written.
That’s why a redo of my website it’s on its way so it can be my personal platform and have Medium as a feed and not the other way around.
Dan really has a point in his article, Why My New Blog Isn’t on Medium when he says that you can do whatever you want with your website.
In my case I want to provide in the future the possibility to switch between English or Spanish content as in Medium readers can’t filter posts by language.
Not only because of language related UX but also; what if I want to share photos from a tech event?
I could just upload a link to a Lightroom gallery and that’d be all. Imagine a post in Medium with only photos of an event.
It would look weird.
Don’t get me wrong, I love Medium. On the other hand, I prefer to publish wherever a story might be most relevant.
I know I could narrate some story beneath each photo and make it “Medium format”, however; some stories are best told in words, other have a photo or video as their champion story teller.
Retrieving Medium Feed
When it comes to fetching a Medium feed, three standard ways emerge to achieve this:
- Use the RSS feed from https://medium.com/feed/@yourUser
- Get latest JSON feed https://medium.com/@alekrumkamp/latest?format=json
- Consume feed from the official Medium API
As we will see next, any of these alternatives by themselves can prove to be inadequate, depending on your scenario.
1. Using RSS Feed
Remember when using the internet meant entering many forums and sites and most of them had an RSS icon to follow up whenever they uploaded something? Yes, those days prior to Youtube’s like and subscribe.
Nostalgia aside, I don’t know about you, but as someone who codes a lot in Javascript, I always prefer to have data in JSON format as opposed to XML, the reason being that it is much more transparent to do JSON.parse()/JSON.stringify() and boom! No dependencies, no custom parser needed.
If you are already familiar with XML and prefer this format, this may be an acceptable solution to get your Medium Feed.
However, there are some caveats:
- No more than 10 posts are fetched using this method, so if you need to bring all of your posts, you can discard this option.
- CORS issues will arise if you fetch from a browser as the route does not provide a Access-Control-Allow-Origin header.
2. Get latest JSON feed
Wow! The Latest posts feed and in JSON? Sounds like what you might need!
Yes, but no. Wait.
If you go straight to the URL and you will find that in fact, there is no valid JSON at all.
There’s one, but only after the string ‘’. This is done to prevent JSON Hijacking.
Because of this, you need something in the middle of the requests between Medium and your website to remove it if you want to be able to consume it.
Using this method we find the same two issues as seen using the RSS feed:
- No more than 10 posts are fetched, so if you need to bring all of your posts, you can discard this option.
- CORS issues will arise if you fetch from a browser as the route does not provide a Access-Control-Allow-Origin header.
- Some parsing is needed to remove the JSON Hijacking protection.
- It seated behind Cloudflare’s IUAM. I’m under attack mode, that is.
To make emphasis on why this last point is a big deal; I was ready to publish this article and its repository two days ago.
Suddenly all the requests using this method started to fail.
I started wondering what could have happened since I wasn’t really touching any of the code when it stopped working.
After a few moments of confusion and debugging I came to realize the request was no longer retrieving useful data but an HTML document provided by Cloudflare informing that I would be shortly redirected to the content.
To put it straight forward; if Medium is under a DDoS attack, even though you are not in any way involved, this method will immediately stop working.
3. Using official Medium API
At this point you might be wondering; doesn’t Medium have a formal API to solve this?
However, the fact that I have to send a mail to request an API key, just to consume a public resource; my Medium feed, is enough to not consider it, specially when there are other alternatives.
(For the record I did request it two days prior to the publication of this post and still got no reply).
Edit: I got a response from Medium three days later with the API Key.
Finding a fourth solution
Because of the need to retrieve a full feed and not just the latest 10 posts without spending my holidays’ time waiting for a response from Medium to receive an API key, I had to find another way to achieve it.
It started by examining monitoring network activity of a request to my profile when I discovered that Medium uses a Graphql API.
Since I’m only interested in fetching public information, my posts that is, I decided to go ahead and give it a try replicating a Graphql request that fetchs the posts in Postman.
And it worked!
The only caveat is that the requests use a userId and a to that represents the following page of posts to be brought from the API.
If I happen to get this missing information, I should be able to bring as many posts previews as I need.
Remember alternative number one with the RSS feed? I used it not to bring the feed but to get the userId.
As far as the to field I was concerned, it turned out every Graphql response returned a to object key which could be used to retrieve the next 10 posts of an user.
Bingo.
Finally, because this API does not provide a reliable way of obtaining image url, title and description of a post, I had to iterate through every post and scrap some of its HTML content, get a chunk of the document and look for their Open Graph tags inside the post, using the postUrl provided by Graphql.
Ale, parsing a piece of HTML of every post is a very inneficient way to get those attributes!
And I agree, but it also proved to be the most reliable source available without using an official API token.
Plus, because I’m using Cloudflare Workers, only the initial request is somewhat slow. All subsequent petitions fall behind Cloudflare caching.
Once I had all the required APIs calls I needed it was just a matter of time, refactors and debugging to come out with a repository that provides a clean interface to fetch all the posts you want using Cloudflare Workers at a incredible fast speed.
Hopefully it will save you some time if you encounter the need to fetch your Medium Feed (:
—
If you want to be up to date with the new content that’s coming up, you can find me on Twitter: @alekrumkamp.
Thank you for reading!
