Photo by Darby Lee on Unsplash

Confident JS series: Part 1 – Encoding and Decoding payloads for saner applications

How a small detail can make your application more reliable

Daniel Kuroski
Jun 2 · 10 min read

Series overview

Due to the dynamic async nature of JavaScript it’s common to deal with unpredictable scenarios, weird errors, and state inconsistencies.

In this series, I’ll show some concepts that can be introduced incrementally in your application to improve or even solve those situations. I also want to bring more discussions about topics I think are important and that I’m not seeing many discussions in the community.

This is part 1 of a series:

Encoders and decoders

Those are generic terms, we hear them when we talk about URL, Base64, character encoding, and even in digital circuits.

But I want to talk most specifically about IO encoding/decoding operations in the Front-end, most commonly used to handle JSON data.

JavaScript is a single-threaded dynamic typing language, this gives us a lot of flexibility, speed, and power, also, it makes the language “easier” to be introduced.

But all of that comes out with a cost, we face a lot of complex async operations and due to its dynamic nature, it may leave us with cryptic errors, unpredictable state, and hard-to-maintain applications.

Below you can see a list of errors, on Sentry, from a few Homeday’s applications, even if they are easy to understand like the t.toUppercase is not a function some times it’s not easy to track the origin of that error.

Sentry log from Homeday

The 4th error Cannot read property ‘uid’ of undefined was fixed by a colleague and I, and the origin was actually in a deep part of the system that has no direct relation with NavigationItem component.

It occurred because some service was sending a variable and assumed the value would always be present. That was not the case and component was accessing nullable variable.

And in worst cases, we might have “silent errors” (the ones that are not logged), in which users face inconsistent states in our app and see something as weird as:

A user opening one important screen of your application

To show you a real example:

Accessing an undefined property OR object interpolation

Above is a component that display’s “User profile data” and it’s easy to reach this kind of situation.

When accessing a property in an object like `user.name`, without properly knowing the actual structure, it might lead us in this kind of situation.

No error is thrown and the user ends up seeing this component.

One other case is when you are interpolating a property that may be an object, then you end up printing [Object object].

By not handling properly our data like in the previous case, can result in a bad user experience, and maybe it can even cost you a client.

Requesting data from an external API, accessing an object property, handling promise rejections/exceptions cases, not properly testing…

It’s far too common to find applications that don’t handle properly those situations

What we can do about this now

A regular application

We will talk about handling unpredictable situations inside of the application in next parts, but one thing that is easy to do and that has a huge impact is dealing with things that are coming/going outside of the context of the application.

The previous image demonstrates what’s usually done in most JS applications.

We are consuming data from entry points and assume we know the data structure.

Then we bring that data in, use it and that may or may not fail because of numerous reasons.

Bringing data from external sources into your application without defining a contract is just like dealing with a bomb, it might explode.

This gives us a high degree of uncertainty, if we don’t have an explicit contract with external resources and something went wrong, we can’t say where the actual problem is. It might be:

What we can do

Usually, we deal with errors when we’re gonna use the data structure, but most times we just ignore them or do not handle them properly or even not handle them at all (e.g: Exceptions, Promise rejections, Error params in callback functions).

One thing we can do to reduce our errors is handling them at the very beginning (in the Edges).

We can do that by decoding data coming from external resources, and encoding them to transform the data to the required contract of our output.

This way, you guarantee that everything that is inside of the application is reliable and well known, and the same goes for data leaving.

Side note

Despite all of those problems, extensive Code Reviews, and planning/refactoring the architecture of our app (when relevant), can solve a lot of the problems, but still it’s not 100% guarantee that our app is bulletproof.

Strong type languages with proper type definitions and tests can help us avoid a lot of those problems.

Code time

Let’s work with a simple example and see better what is the problem and how to solve it.

Here we have a common scenario, our application init makes use of an api service, this service will make a “fake” HTTP call to return an object containing price and vat from a purchase.

rawData is just a representation of what we receive from an API call while api function just returns a resolved Promise by parsing rawData

After having a response, we just log the result, we expect to see Total price is 11.5 (price + vat).

But actually what happens is this:

Total price is 101.5

But why “101.5”? Our fake API is returning price: "10" on line 2 and this is a String .

This means we are doing "10" + 1.5 and in JavaScript when we try to sum a string with a number, it will concat the results and we end up with 101.5 .

To fix that, we just need to change:

// from
const rawData = JSON.stringify({
price: "10",
vat: 1.5
})
// to
const rawData = JSON.stringify({
price: 10,
vat: 1.5
})

The problem here is that you can’t simply trust blindly in the API, because in the real world:

Typescript doesn’t guarantee that for us

If we rewrite our code to use Typescript, we will still have the same problem:

If we run this program:

Total price is 101,5

Typescript does its best to help us architecture our application and catch problems in compile-time, but at the end of the day, this code compiles to JS and we “inherit” the same problem.

But there is one major difference from the previous example:

We eliminated one degree of uncertainty by defining the shape of our data

We explicitly defined that we are expecting numbers, if the application is logging the wrong result because the API response is returning a string, this means the API is breaking the expected contract.

But what can we do then?

We can decode the data, it can be achieved by manually checking the data structure (the shape and types), like

const api = () => Promise
.resolve(JSON.parse(rawData))
.then((data) => {
if (!data.price || !data.vat) return Promise.reject("Invalid result")

if (
typeof data.price !== 'number'
|| typeof data.vat !== 'number'
) return Promise.reject("Invalid data type")
return Promise.resolve(data)
})

But that just seems too much work 😅

I will show you a powerful library that helps us dealing with decoding operations.

Inspired by the Elm JSON decoding package, we can use https://github.com/nvie/decoders to help us out, this way we can create a declarative schema of our data that will be validated at runtime.

On line 1, I’m importing the library.

On lines 3–6, I’m defining the “shape” of my data structure.

On line 8, we must wrap our encoder into this guard function, ProductGuard is now the decoding function, which we can use to validate any JSON and confirm it’s an actual product. We do so by just calling ProductGuard(payload)

And in line 15, I’m making use of the ProductGuard , please note that then(ProductGuard) is the same as doing then((payload) => ProductGuard(payload))

The result is cool, if we run our code, the application will fail with a clear message:

Error message pointing where are the errors in the JSON

This way, you guarantee your types at runtime.

If you are using Typescript we can also provide more information to it and validate our types statically.

What about encoders?

The process is pretty much the same.

You can rely on the library to define your data structure and validate the data before sending data out of your application.

Relying on encoders can be particularly useful for having one place to handle all data transformation for the API and can be as simple as:

const userEncoder = (user) => ({
first_name: user.firstName,
last_name: user.lastName,
products: user.products.map(productEncoder)
})

And by following the pattern that was mentioned before, data should be handled in the “edges”, so sending something to the encoder should be the last thing before the actual request.

The previous encoder is ok, but definitely, it’s not safe, you can still use the same library we used for decoding:

Different example

Below, you can find a different example, please note that I simplified a lot the data structure, so things seem to be a bit redundant, but when you are dealing with bigger and more complex data, this can be particularly useful.

The example is a bit extensive, and if you are not using Typescript, you can just ignore that part (in the following articles I will show how to make use of the types in the declaration file without TS).

I hope that “pseudo-code” helps you to understand one way to structure your application, I introduced some rules on the data shape on purpose, since we have to deal with that kind of thing in the real world.

With that structure we have:

With that, we’ve pretty much covered the edges, and with that contract in place, we can safely use the data inside of our app with confidence that it is impossible to have something different than what we have defined.

Conclusion

I hope I could make some good points on why and how you should use decoders and encoders.

Introducing this kind of concept into your daily work is relatively easy, since you can just evolve incrementally, at Homeday we are slowly adopting them, and until now, we were able to discover and fix a few problems with our API contracts.

If you are using something like GraphQL, then you probably don’t need to worry about this, since you have a well-defined schema.

Furthermore, I will introduce a more advanced way to encode and decode your payloads by using fp-ts and io-ts libraries.

For more information about the library used here, please check the library repository at:

And the author Vincent Driessen have an amazing introductory article about his library:

I can’t stress enough how this can make your life easier, I’ve learned about those concepts while programming in Elm in past experiences, the syntax might not be familiar at first, but I strongly recommend to reading through the following two links, it can be life-changing 😁

https://guide.elm-lang.org/effects/json.html

I’ve made this presentation at Vue.js Berlin Online Meetup #11, if you are interested, please check the video.

Vue.js Berlin Online Meetup #11 Video

Thank you for reading this article, please stay tuned to the next parts.

Do you like what you see?

Give us a clap, leave a comment, or share anywhere. We appreciate your feedback.

Also, check out other articles on our blog. We deal with many interesting things:

Check out our openings at our dev team!

Sorry for the long post, here is a cat dressed as a potato.

Homeday

HOMEDAY – The next generation estate agent.