Lessons Learned Migrating Our Blog to Medium
Earlier this month Keen moved our blog to Medium. I was the lead engineer on the project, and… well… it sucked. Most of the sucky parts were my fault, but I thought I should write this down anyways for two reasons: I need to vent, and perhaps I can save y’all from making the dumb mistakes I made.
Frankenblog
For as long as I can remember (so, probably forever) Keen has used a sort of Frankenstein blogging system. While our blog has served us well for many years, it has come time to recognize that, as Frankenstein is well known to be, our blogging backend was ugly, brutish, and caused a bunch of hassle.
I don’t want to go into too much detail — it’s quite painful — but I will say The Frankenblog included Tumblr API, Flask, WTForms, lots of button presses, and confusion.
#ThisShouldBeEasy
Obviously, that whole thing was sort of convoluted and painful. Moving to Medium has a bunch of upsides: it’s pretty, it has a nice editor, Medium gives us good “reach”, and generally we just like Medium as a company. Medium recently released their Medium for Publishers product which should make it really easy to brand our blog how we wanted. Win, win, win, win.
All in all, this seemed like a wonderfully easy project. I was the engineer and was paired with a designer, a PM, and a random smart dude. The work seemed simple — we store all of our blog posts (synced from Medium) in our own Mongo database, with all of the relevant metadata. Medium has an API, so this whole thing should have been a simple migration script that punts all of the content to Medium. Or, so I thought.
The Great Import
Turns out it wasn’t easy, and as I was halfway through the process, one of my fellow engineers shared a bit of wisdom that pretty well describes my whole experience:
Integrations are tough. Sometimes it’s not obvious what’ll happen.
The first real issue that I ran into was the idea of authors. Authors are very simple in our Frankenstein blog, but they are a bit more complex in Medium. Medium is a platform, which means authors have full profiles, and they’re not even necessarily specific to Keen’s publisher account. Authors are all just personal accounts. I, as an admin of a publisher account, can’t simply author content for any random Medium user — I need express permission to act on their behalf. In nerd terms, that means I needed everyone at Keen to send me a personal integration token. It took some time, but I eventually got all the tokens. Turns out people respond well to instructional GIFs.
Once I got all those tokens, it was time for the big import. This is where I really become dumb. Medium had no posted rate limits, and we don’t have that much content, so I dove into the import head first. I intended to just upload all of our posts in the initial import, and see how it went.
It went poorly. Off the top of my head, here are some of the mistakes I made. Each one required a separate full import to discover:
- Sent the paraphrased “description”, instead of the full post
- Sent posts that weren’t actually published on our own blog
- Sent posts too fast and hit rate limits (not actual rate limits, but Medium’s API barfed). Had no retry/failure mechanism.
- Wrote a bad closure, and sent the wrong author for most posts
- Didn’t have the latest dump of our Mongo database, so failed to send the latest posts
There were more silly mistakes I made, but I can’t remember them. Clearly, I was trying to move fast instead of smart. I had assumed this task would be easy, and fairly low risk. My focus on speed blinded me to the implications of my mistakes, and disregarded their easy fixes.
The most painful part of this is that Medium’s API does not offer a DELETE endpoint. The only way to delete posts is via their website, which requires 3 button clicks on each post page. With 200 posts, that took a long time; an especially long time when you have to do it multiple times. I also wasn’t doing any logging early on, so I had no choice but to delete everything every time — otherwise I would reupload content and get duplicates.
At some point the smart engineer in me woke up, and I added some logging as well as database fields to indicate which posts had been uploaded successfully. That, paired with some UI automation hacks I created, sped things up.
One funny thing to note here… Every time I took any action on behalf of a coworker’s Medium account, they got an email. Creating a draft, deleting a draft, editing a draft, publishing a draft, etc. — those all sent them an email. I got 100s of emails. I expect our more popular writers got thousands. Sry frands.
“Going Viral”
Once I got the import thing worked out, we started to make real progress. I ended up adding a lot of logging, which enabled me to script in the redirects (301s from flask to Medium) for our old blog. Once that was done, we were ready to start turning things on. I needed to get the bulk of our content available so that our designer could start crafting our pages. It’s hard to design without any content available!
In order to do that, I had to soft launch the blog. That meant turning on the domain (we moved from keen.io/blog to blog.keen.io) and publishing all of the posts that had been imported as drafts, but not actually directing traffic there. That was all pretty easy — I even set up a very basic (ugly) blog design so our designer would have something to build from. All of the published dates were wrong (more on that later), but the content was there — ready to hand off to Mr. Designer!
Before I could handoff, though, I got a message from another coworker.
Coworker: hey just a quick note to let you know our blog on medium looks pretty weird in terms of design layout
Me: Yeah, it’s not launched yet
Coworker: not launched, really? I have friends commenting on articles.
Turns out Medium is a social platform, so when you suddenly publish 200 posts you go viral! If you remember back to the beginning, I was publishing all of these posts to my coworker’s personal Medium accounts (which are then linked to our Keen publication). Some of those people have existing audiences/followers on Medium, and their networks just got a ton of notifications about new content being posted. And it was all linking to our unfinished and terribly ugly blog! And, to make it even better, the posts were in the wrong order because the dates were all wrong.
Medium Support, the Real 10x Engineer
The notifications were a problem I couldn’t solve. They went out, and there was simply no going back — our networks on Medium were gonna be shown our newly migrated content. I needed to get the Medium blog looking nice, and fix any remaining content issues, ASAP.
I pestered our designer to “plzzzz hurry”, and went to solve the last issue that I had control over — published dates. Medium’s API doesn’t have a way to set the published date on a post (or, at least, not on a post that you initially create as a draft), so everything was marked as being published on the day that I migrated. Medium’s support team rescued me here — luckily I had added a good bit of logging (finally) to the import scripts, and was able to easily map the Medium Post ID to the intended publication date.
I honestly can’t believe how helpful Medium was with this. I consider myself to have been a pretty annoying user at this point, but their response times were incredibly fast, and they never shied away from helping. I ended up having to ask them overwrite the published dates 3 times (because I, once again, moved too fast). Each time I expected them to give up on me, yet they responded quickly that the migration was done. They are wizards and saints, mashed together into a beautiful support team.
I Love Our Designer
Micah Wolfe is one of our designers at Keen, and he is a magician of a different sort. Given the urgency of making that page pretty, I had actually stared at it for quite awhile thinking of a way to spruce it up in the mean time. I honestly couldn’t think of anything. I simply don’t have an eye for this stuff, and had no ideas about how to make it better.
Micah, paired up with an awesome member of our content team, absolutely crushed it. I went to bed one night, and woke up the next day with a b-e-a-utiful blog.
And we launched! And it’s live! And now I get to learn from my mistakes, and never do things this poorly again.
The Moral of the Story
The lesson here is actually a pretty simple one. Some of it applies directly to Medium (ie: their API is simplistic, be careful of notifications, contact their support early, instead of trying everything yourself), but there’s a larger general engineering lesson that I have relearned:
Planning is worth it, even for the small stuff.
At Keen, we work in 1 or 2 week sprints. Small stuff, like migrating a blog, still gets a full week. I took this for granted, thinking I would churn the blog out in 2 days, and then twiddle my thumbs the rest of the week. That was a bad idea — sprints are a minimum of week for a reason. Taking the time up front to plan things out and predict the roadblocks and risks is vitally important. A small task becomes an inordinately large and stressful task, if you don’t consider the challenges.
Anyways, hope you like the blog! 🚀
