The Trials And Tribulations of Creating An SDK — Part 4

I’ve made it to the climactic ending of the tale of my SDK creation journey. In the last episode, I told you about how my first attempt at generating the SDK methods and classes was successful, but ultimately unsustainable. I also hinted that my next approach might not have been the best choice either. Let’s look at why.

There’s nothing technically “wrong” with generating code with a throwaway generator. It’s like using a brick to pound a nail — it’ll work in a pinch for a quick job like hanging a picture on the wall — but I wouldn’t recommend building a deck with a brick instead of a hammer. I’ve written more throwaway code generation scripts over the years than I can even remember, and most of the time they’ve done the job and gone into the scrap heap that is my /Projects/scratch/ directory rarely, if ever, to be seen again. But it was a lapse in judgement for me to think that this task was one of those throwaway jobs. An API changes often, and the documentation isn’t guaranteed to remain in the format that it was on the day that I scraped it with the jQuery code. And that’s why my next approach, while more stable and automated, was another misstep.

I got really tired of clicking on each nav section to generate the code. That’s definitely one of the “smells” of a bad process — when it involves manual, repetitive tasks. So how could I write a script to loop through every doc page and perform the code generation tasks with only a single “click” or command to start the process? My thoughts immediately went to Selenium. If you’re not familiar, it’s a tool for automating web applications — meaning you can programmatically browse through a web app via a script. It’s mainly used for testing purposes, but, certainly scraping web docs would work, right? So I set about writing a Selenium script in Groovy to scrape the online documentation and pull the necessary information to generate the SDK from each method’s HTML. Like my previous attempt, technically it “worked”. And it was easier too. I could start the script, watch it run and sip some coffee while I browsed cat memes on Reddit and scrolled through my Twitter feed for about 7 minutes while it did its thing.

So what’s the issue then? Well, the problem with this approach is the reliance upon the HTML format that I’m scraping. It’s brittle as hell — relying on things like xpath expressions to find certain things like parameters, body arguments, etc. A single change in the documentation format can break my entire script.

It didn’t take long to create, and in the short term it’ll work. I can regenerate when the API docs get updated (and they already have been— very shortly after my first pass with the new generator). But a better solution was there all along.

Oh, what’s this???

Finally doing what I should have done before I’d written a single line of code, the next day I opened up dev tools and reloaded the page. The obvious answer was right there, staring me in the face. Sure, it’s not “official” or even documented, but this file was where I should have started with my code generation efforts. What does this file look like?

JSON doc showing all of the Swagger YAML config locations

It’s a JSON doc that contains the endpoints for Swagger YAML config files each individual service in the REST API. The Swagger files each document every single method, parameter in full detail and are obviously what is being used by the documentation site to generate their HTML docs. This gold mine is what I should have used. Why? Because Swagger is the de facto standard for API documentation. The Swagger format is well defined and there are tons of tools that can be used to parse it into a native format that can be consistently re-used to generate the SDK. A simple script that parses one of the YAML files with the swagger-parser project might look like this:

Which gives me an amazing native Java object like so:

Object Browser representation of the Swagger object

Beautiful, structured, natively supported and consistent. An object that I can iterate over — one for each service — that I can count on to have the data I need to generate the SDK. It would be perfectly easy to loop over the index.json and parse each Swagger into an object that I can use to generate (and regenerate as necessary) my SDK.

My next step is to rewrite the generator, again, using this new method. And I’ve definitely learned — again — to step back and plan when taking on a new project. Sometimes my excitement and enthusiasm for writing code can get me in to trouble if I don’t take the time to fully architect my solution, but the important thing is I can understand the feel and smell of unmaintainable solutions and am not too stubborn to rewrite (as often as needed)!