The Trials And Tribulations of Creating An SDK — Part 3

I’ve previously told you about my motivations in creating an SDK as well as walked you through the initial structure of the SDK that I created using NodeJS to interface with the Oracle Cloud REST API. My intention for the third entry in this series is to start walking you through how I generated the code for the SDK, but, before I do that I want to address one of the foundational architectural decisions that I made when I created the SDK.

You know that JavaScript is a dynamic language. We could spend days discussing the merits or downfalls of dynamic languages, but I’d rather not get into that right now. TypeScript gives us the ability to statically type our variables, but absent of using TypeScript we’re stuck with the fact that JS simply doesn’t have typed variables and our IDEs can only go so far with the support that they provide to us, especially when working with an SDK such as the one I created. That means, although I could have designed the SDK to accept an Object and pass that along, it would end leading developers to refer to the original API documentation for the possible parameters and leaving it up to them to essentially build the entire request themselves. Meaning things like typos and nesting errors are really, really easy to be introduced.

To further illustrate my point, let’s take a look at the AWS JavaScript SDK. Let me first say, I’m normally a fan of how Amazon designs their APIs and SDKs. I’ve used them quite a bit over the years, but I was really surprised when I looked at the documentation for the JS SDK. Here’s an example using the SDK to call getObject on an S3 bucket:

AWS JavaScript SDK getObject

Now imagine you’re working in your application and you’ve created an instance of the S3 client. You type s3.getObject( and wait for your IDE to provide intellisense to help you know what parameters getObject accepts. Only you don’t get anything helpful at all because getObject accepts an Object that requires you to know the keys that it expects. So you head off to the docs, look them up, come back and paste them into your IDE. And then you do that again later on. And again. And all of the sudden you’re wondering why you’ve even plugged an SDK into your application because you’re spending almost as much time in the docs that you would have had you chosen to just call the REST API directly.

In the SDK I created, if you wanted to call that same getObject method, you’d create the client and you’d see something similar to this in your IDE:

Ah, OK. It wants an instance of GetObjectRequest! So you create an instance:

And thanks to the magic of modern day IDEs we can see, even in a dynamic language, the property type, whether or not it is required and a description of the property from the jsDoc comments within the SDK. No TypeScript required.

I hope that clarifies why I made the decision to design the SDK in this way. I know it may feel a bit different than how you might normally do things in Node-land, but it’ll make sense after a short while. 😏


Now that I’ve got that out of the way, let’s get into my mistakes. As I said at the end of my last post, I’d arrived at the conclusion that the best course of action for creating the SDK was to generate the methods and classes via the REST API documentation. In my excitement, I made my first mistake. It’s a mistake we’ve all made when writing code — and a mistake I should have known better than to make after writing code for 15 years: the mistake of not thinking of the future consequences of my decision. See, I figured I could quickly and easily scaffold out the code by writing a throwaway generator using jQuery to scrape the documentation pages and generate the methods and classes. Now, don’t get me wrong — it worked. After spending a few hours handling a few edge cases, I ended up with a script that I could paste into the Chrome console that would scrape the current doc and generate the code. I ran it a few times and pasted the results into the client files. It felt good. In 5 minutes I’d have an entire client generated! I got tired of clicking, so I added some code to the generator to copy the code directly into my clipboard. Awesome! The next client, slightly more complex, took another 5 minutes (even with the time saving of setting the clipboard). Hmmm…what if I put a global click handler on all of the nav items so that it generated without me having to manually running it. Cool, that saves a bit more time. Still, there are 17 clients — 700+ methods to generate!! In my excitement, I decided to keep grinding — ignoring one of the golden rules of programming: if it feels wrong, it likely is wrong.

A day later, I had finished generating the entire SDK. I tested it, and it worked.

Then I tested a few different methods.

Oh.

Shit. An edge case.

TFW you think you’re smart, but you realize you’re far from it.

I spent the next few days on and off catching edge cases, modifying the generator and re-generating some code. Nothing required a complete re-generation of the entire SDK, but it was still starting to feel unmaintainable. I hadn’t even arrived at the inevitable change or addition to the API, but I knew a better process had to be found.

And that’s where I made my next mistake.

I’ll tell you about that mistake, and how I overcame it, in the next post.