Hypermedia API Clients
Utopia or dystopia?
Barely relevant introduction
We started building our “yet another” in-house CMS about three or four month ago. Well, the first month was actually me just thinking about what do we actually need. The obvious question was “why another CMS? aren’t there plenty?” Indeed there are. Furthermore the notion of a custom CMS being a bad decision when developing a website/app/whatever is very well explained in a couple of articles out there. Yet we decided to create our own…
However good our shiny new CMS will be there is always a chance that our client would want to switch to another one. And I think this should be one of the most or even the most important qualities of any CMS — how easy it is to switch to another one. And I’m not talking about data migration here. There are various way to solve the data migration problem. I’m talking about rebuilding the website itself when you switch from Wordpress to Drupal or whatever system you want to switch to.
One of the solutions I see for this problem is having a layer between your front-end and your CMS that is constant. Change your CMS, write a plugin that will parse your data into the format your front-end understands and you’re good to go. As long as your front-end and your CMS speak the same language you don’t really care about what software stack you use on either side.
Long story short we need an API that our client apps can understand. As long as whatever CMS you use doesn’t break the API it doesn’t break your clients. But this is a really utopian view of APIs, isn’t it? We all know that APIs change and when they do the clients should change as well. So having to maintain a hundred clients on various versions (API versions?) is a rather frightening idea.
That’s why I started researching into Hypermedia APIs. I guess this phrase from Leonard Richardson and Mike Amundsen’s “RESTful Web APIs” (which is a must-read) sold the idea to me:
When APIs change, custom API clients break and have to be fixed. But when a website undergoes a redesign, the site’s users grumble about the redesign and then they adapt. Their browsers don’t stop working.
In REST terms, the website redesign is entirely encapsulated in the self-descriptive HTML documents served by the website. A client that could understand the old HTML documents can understand the new ones.
Utopian part
That’s where everything is nice and shiny and every day is like your birthday with presents and cakes. I’m not going to review the whole concept of Hypermedia. Luckily there is enough useful information around. So if the idea of Hypermedia and Hypermedia APIs is new for you, here are some links to get you started (If the HATEOAS acronym seems like a misspelled “hate oats” or “hate us” typed in an overly excited caps lock manner for you then you definitely should click through the links):
- Dave Goldberg’s Hypermedia Resources blog post. (He has a section for “hypermedia people”… I had some wicked dreams after thinking about this notion)
- This series of posts managed by one of the “hypermedia people” that I have already mentioned — Mike Amundsen.
- Kevin Sookocheff’s blog post on choosing a hypemedia type
- Roy Thomas Fielding’s dissertation (specifically chapters 5 and 6)
The last one is referenced in almost every single article/lecture/video/presentation about Hypermedia APIs. Yet I doubt every single person researching into the subject actually spent time reading it. It reminds me of an old Russian anecdote:
— I don’t understand all those people admiring Caruso… No voice, never in tune, lisps, burrs…
— Wow, you heard Caruso sing?
— Nah, Petrovitch sang me some of Caruso’s stuff.
I encourage you to read Fielding’s dissertation (or at least the mentioned chapters) to get on the same page in Hypermedia API discourse.
So things are pretty amazing — sort out your media type, application states, resource representations, profiles and you’re done. Welcome to the beautiful world of Hypermedia…
Dystopian part
Well not technically a dystopian but…
Imagine someone invented a cure to any disease that ever existed. It will cure you and make you immune to any other possible disease yet to come. The only problem though is that there is no defined way to consume this cure… Some people might inject it into their veins and it will work, some should use it in a pill form, some should sniff it with their ears…
As exaggerated as it can get that’s how I see it. Finding the information about Hypermedia clients is so hard that I started thinking the subject is pretty obvious for everyone but me. As if everyone was always writing Hypermedia clients and then at some point they said “Enough! We need hypermedia APIs to support our hypermedia clients”. So the best minds sat down and came up with Hypermedia API notion. And they wrote books and articles and blog posts about this notion, because you know, the client side is pretty obvious.
It’s not obvious for me. Of course, there is always a simple explanation — I’m just not intelligent enough. But then again — why should it stop me from trying to sort things out for myself? (there should be a smiley face here but I lothe those so just imagine me smiling)
In our day-to-day scenario our client is a thick javascript single page app. It consumes JSON and loads everything besides the main page through AJAX. We use AngularJS but it’s not relevant to the question I have no answer to at the moment. Almost every front-end framework today has a notion of routing and resource. Routing is what you use for your front-end specific deep linking scenarios, resource is what you use to work with your “RESTful” APIs (mind the quotation marks). Your user navigates through the pages (or UI states) you have defined in router and gets the resources that your app loads from the backend. Simple stuff, everybody does it.
The problem is that you have pretty much everything predefined. The states your application has, the data that it is able to request. If you add a resource your application won’t know about it. Taken my initial tasks — my CMS — this situation doesn’t really work well for me. What if my client wants to add another page (i’ll go with the page metaphor for the sake of a simple example)? Although there are ways to deal with it they are still based on the notion of client knowing a lot about the API. And anyway when creating your website with plain old HTML and you add another page and a link to it you don’t have to tell every browser to update to find the link to this page. And that is only pages I’m talking about. There is more to it…
So what do we do? Well there is this Fielding’s phrase to start with:
Hypermedia is defined by the presence of application control information embedded within, or as a layer above, the presentation of information.
Sounds exciting, doesn’t it? As far as I can understand he says:
Throw your routers away and base all your application state changes on the data you are receiving because this data should include the information about where to go next and where else can you go.
Nah… it can’t be. It seemed like I should change the way I think about front-end apps. And as human as I can be I don’t like changing the way I think about something. So I decided to Google this subject to death…and maybe find some information to justify me not changing the way to think about it. Isn’t that how we normally deal with things that are threatening our stable points of view?
What I managed to find so far
In this presentation Thomas Parslow he describes a backbone.js app that consumes some personal JSON-based media-type (there is no notion of media types in the presentation itself but that what it appears to be for me). The way Thomas deals with the lack of hypermedia controls in JSON media-types is quite interesting. He separates his messages into “head” and “body” sections (really HTML-esque). Protocol semantics description (allowed actions) as well as additional outgoing links related to the current state of the resource requested go into the “head” section. Application domain stuff goes into the “body” section.
Ok, it’s not HAL, Siren or Collection+JSON but I wasn’t looking for a specific media type client or for the “hypermedaiest” media type. I was looking at the ways of dealing with hypermedia using javascript clients. And unfortunately (or fortunately for my don’t-want-to-change-the-way-I-think task) Thomas uses hypermedia merely to get resources related to the one that he requested. I couldn’t even find any description of how he got to that particular resource in the first place. And the application state’s fate was left in the hands of the backbone router. So nothing too new really. We have got some fancy stuff in the form of related resources URLs of which we don’t have to hardcode but it’s not a game changer. And we still deciding the state transitions upfront.
Then I stumbled upon this presentation by… Mike. The presentation is a call to action. Let’s write generic hypermedia middlewares for our clients. Well, I’m in. And there are generic hypermedia libraries out there… even specifically for angular. But what about routing…deep linking…you know…models… (I’ll get back to Mike’s presentation a bit later)
I continued my “research”. Some of the results were clearly wrong (while marked as the correct answer). There are also valid concerns about “if I give the control over state to hypermedia what if I refresh the page or send a page in some random state to someone else”. Indeed what if so? HTML and browsers seem to have solved it somehow. That brings up the problem of mapping front-end router links to API links (calling it mapping is not a good idea, I’d say it’s passing a link through because you’re not really mapping anything upfront).
Jimmy Bogard states that hypermedia APIs are amazing and “imaging how easy it is now for any client” he even mentions the client his company has built. No details though. Well at least there is another article to say how cool hypermedia APIs are. I’m sorry for being sarcastic but come on. Show us a bit of your client building ideas…please.
This promised to tell me a lot. And it kind of does. But than again it really doesn’t. I guess I’m spoiled by thorough descriptions of thinking behind API designs. I don’t really need to see any code. I need to now about the approach.
This article by Jan Kronquist and more importantly this comments thread touches upon the very problem I was trying to find a solution to. And Hugo Josefson offered a realy interesting solution. I mean it’s the only concrete solution I have managed to find. Buried in the comments.
I would like to mention this article by… guess who. It’s a blockbuster about implementing a proper hypermedia API using node.js. But we are interested in the clients, aren’t we? I’ll quote Mike:
The next instalment in this series will explore the details of coding various types of hypermedia clients. Ones that provide a “faithful rendering” of the server’s responses, ones that maintain their own “custom view” of server replies in order to establish their own application interface, and ones that act as automated robots that solve specific problems without the need for human intervention at runtime. Mike Amundsen, Feb 19, 2013
Wait… February 2013?
I mean obviously there is this video and this slide deck. The presentation even deals with the same problem domain. But still it would have been nice to read the other two parts of this four-part article series.
I could go on for another 1000 words but the truth is I still have more questions about hypermedia API clients than answers. I hope there will be more answers in the comments to this post. I decided I’m going to put my main observation as a quote so the “hypermedia people” and genuinely smart people can tell me if I’m wrong without reading the whole post. So here it goes:
You can only use a hypermedia architectural approach to your benefit if your clients respect state transfers defined in the representations you receive from the server. In fact the only steps to change the state that should be available in your application are the ones defined by the representations.
This does not mean that your API or your app is wrong. This does not mean “oh my god my REST is not RESTful enough”. It actually means that it’s not RESTful at all. But that does not mean that there is something bad with your app or your API. It just means you’re not using a specific architectural paradigm for your API that was developed to solve a set of specific problems (which might not be your problems at all). And if the API you’re consuming is indeed a hypermedia API then if your client isn’t using it to it’s own benefit… well it just doesn’t benefit from it how else can I put it.
The previous paragraph can look like a bunch of nonsense but I had enough discussions and read enough threads to understand that one of the main reasons we don’t have enough going on around hypermedia discourse is that half of the discussions look like this:
Person A: Hypermedia is cool!
Person C: What did you say? My API is not good enough because I don’t have stupid links there?
Person D: Twitter doesn’t have links… so you’re wrong. I trust twitter more than I trust you… Why should I even trust you?
Person E: Where is person B?
So I though I might try to avoid those kind of comments. They are not really productive. Guys, your APIs are the best APIs this planet have ever seen. Thank you.
As it is an ongoing quest, I promise to share my further findings in other posts.
N.B. I’m interested in javascript-in-browser clients that don’t use HTML as a media type. However there is a notion of creating APIs with HTML as a media type and I don’t think it’s a bad idea. Further reading: this Jon Moore’s blog post, this one by Stephen Mizell, this talk by Gustaf Nilsson Kotte
another N.B. I hope that I’m wrong and everything is already sorted and I just don’t know enough or I don’t know where to look. If this is the case please point me in the right direction and then you can use this post to show other people “how not to write about hypermedia” or even “how not to write blog posts”.
