Perform complex conversions on demand and at low cost

Andrew Tate
Aug 20 · 4 min read
Photo by Tobias Fischer on Unsplash

I often come across scenarios where the data that my application needs is available, but not in a format it can readily consume.

Cloud functions and other microservices really shine here. They can convert data from one form to another on demand and are generally very inexpensive. Since they are distinct execution contexts, they don’t have to match the consuming application’s tech stack. This frees them to use the language and libraries best suited to the job of conversion.

They also totally encapsulate (hide) the often inelegant conversion code, keeping the consuming application’s logic simple and focused.

One conversion I routinely need is from static websites to RESTful APIs. Let me illustrate with a couple of examples.


Example 1: Library Catalog Results

Take the example of my local library website. I want to create a browser plugin to notify online shoppers when a book they’re looking for is available at the library.

Photo courtesy of Nicepik

The library website provides a searchable online catalog. However, all the pages on the site are rendered server-side. The library doesn’t expose a public web API, either.

As it stands, my only option is to send my search query to the library’s server and add logic to my extension to parse the returned search results HTML.

But what if we instead create a cloud function that accepts a request from my browser extension, makes that network call to the library’s server, parses the returned HTML, and sends back a nicely formatted JSON response?

Conversion API Architecture

We will have effectively converted my local library’s statically rendered website into a RESTful JSON API.

Now that’s cool.

Here’s my cloud function code. (I use the Serverless framework with AWS Lambda for development and deployment).

And here’s an image of the JSON that my application gets back from the new API. This lists the first entries in the library’s catalog for the word computer.

Hopefully, you can already see the power in this microservice data-conversion architecture.


Example 2: Computer Science Faculty

Let’s say we’re building an app to recommend faculty mentors to potential graduate students at the University of Utah based on their research interests.

Fortunately, a list of faculty, complete with research blurbs and even contact information, is available on the university’s website.

Let’s follow the same pattern to create a web API. Here’s the code:

University of Utah Faculty API Function Code

Here’s the response from the new faculty API:

See the API working live here.

This bears many similarities to the library example. But do you notice differences?

Here’s an important one: Faculty change very rarely.

In the last example, it was necessary to poll the library website every time the cloud function received a search request from the consumer. However, this is often not the case.

In this faculty example, actually checking the static website for updates is rarely necessary. That frees us to cache the faculty data and simply poll the website for updates on an infrequent basis. Making this change decouples the data source from the consuming application. We’ve moved from one cloud function to two:

Cached Conversion Architecture

On the one hand, I like this decoupling because if you don’t control the data source (static website or otherwise), unannounced changes to it may require an update to the conversion code.

Without this decoupling, the conversion failure would likely be immediately apparent to an end user.

With it, the consuming application will probably continue to function just fine for a little while. This gives you time to update the conversion code with few hiccups.

The only downside, of course, is the extra complexity of making sure the cache is always fresh enough to be viable for your consuming application.


Conclusion

While this article is geared particularly toward converting from static websites to RESTful APIs, let me reiterate the general applicability of this cloud-function conversion pattern.

It provides a modular, targeted, independent environment in which to perform the sometimes complex conversion operations that are frequently required by modern applications.

Better Programming

Advice for programmers.

Andrew Tate

Written by

A technologist with an interest in software architecture and development processes. I write about both the machine and the machine that makes the machine.

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade