Confident JS series: Part 1 – Encoding and Decoding payloads for saner applications
How a small detail can make your application more reliable
Series overview
Due to the dynamic async nature of JavaScript it’s common to deal with unpredictable scenarios, weird errors, and state inconsistencies.
In this series, I’ll show some concepts that can be introduced incrementally in your application to improve or even solve those situations. I also want to bring more discussions about topics I think are important and that I’m not seeing many discussions in the community.
This is part 1 of a series:
- [You are here] Part 1 – Encoding and Decoding payloads for saner applications
- Part 2 – Types, JSDocs, and declaration files
- Part 3 – Test what improves your confidence
- Part 4 – A bit more functional
- Part 5 – Task, Either and Boxes
- Part 6 – Fp-ts and io-ts, now it’s time to get serious
Encoders and decoders
Those are generic terms, we hear them when we talk about URL, Base64, character encoding, and even in digital circuits.
But I want to talk most specifically about IO encoding/decoding operations in the Front-end, most commonly used to handle JSON data.
JavaScript is a single-threaded dynamic typing language, this gives us a lot of flexibility, speed, and power, also, it makes the language “easier” to be introduced.
But all of that comes out with a cost, we face a lot of complex async operations and due to its dynamic nature, it may leave us with cryptic errors, unpredictable state, and hard-to-maintain applications.
Below you can see a list of errors, on Sentry, from a few Homeday’s applications, even if they are easy to understand like the t.toUppercase is not a function some times it’s not easy to track the origin of that error.
The 4th error Cannot read property ‘uid’ of undefined was fixed by a colleague and I, and the origin was actually in a deep part of the system that has no direct relation with NavigationItem component.
It occurred because some service was sending a variable and assumed the value would always be present. That was not the case and component was accessing nullable variable.
And in worst cases, we might have “silent errors” (the ones that are not logged), in which users face inconsistent states in our app and see something as weird as:
To show you a real example:
Above is a component that display’s “User profile data” and it’s easy to reach this kind of situation.
When accessing a property in an object like `user.name`, without properly knowing the actual structure, it might lead us in this kind of situation.
No error is thrown and the user ends up seeing this component.
One other case is when you are interpolating a property that may be an object, then you end up printing [Object object].
By not handling properly our data like in the previous case, can result in a bad user experience, and maybe it can even cost you a client.
Requesting data from an external API, accessing an object property, handling promise rejections/exceptions cases, not properly testing…
It’s far too common to find applications that don’t handle properly those situations
What we can do about this now
We will talk about handling unpredictable situations inside of the application in next parts, but one thing that is easy to do and that has a huge impact is dealing with things that are coming/going outside of the context of the application.
The previous image demonstrates what’s usually done in most JS applications.
We are consuming data from entry points and assume we know the data structure.
Then we bring that data in, use it and that may or may not fail because of numerous reasons.
Bringing data from external sources into your application without defining a contract is just like dealing with a bomb, it might explode.
This gives us a high degree of uncertainty, if we don’t have an explicit contract with external resources and something went wrong, we can’t say where the actual problem is. It might be:
- The API that is not returning what we want
- The front-end application that is not handling the data correctly
Usually, we deal with errors when we’re gonna use the data structure, but most times we just ignore them or do not handle them properly or even not handle them at all (e.g: Exceptions, Promise rejections, Error params in callback functions).
One thing we can do to reduce our errors is handling them at the very beginning (in the Edges).
We can do that by decoding data coming from external resources, and encoding them to transform the data to the required contract of our output.
This way, you guarantee that everything that is inside of the application is reliable and well known, and the same goes for data leaving.
Side note
Despite all of those problems, extensive Code Reviews, and planning/refactoring the architecture of our app (when relevant), can solve a lot of the problems, but still it’s not 100% guarantee that our app is bulletproof.
Strong type languages with proper type definitions and tests can help us avoid a lot of those problems.
Code time
Let’s work with a simple example and see better what is the problem and how to solve it.
Here we have a common scenario, our application init makes use of an api service, this service will make a “fake” HTTP call to return an object containing price and vat from a purchase.
rawData is just a representation of what we receive from an API call while api function just returns a resolved Promise by parsing rawData
After having a response, we just log the result, we expect to see Total price is 11.5 (price + vat).
But actually what happens is this:
But why “101.5”? Our fake API is returning price: "10" on line 2 and this is a String .
This means we are doing "10" + 1.5 and in JavaScript when we try to sum a string with a number, it will concat the results and we end up with 101.5 .
To fix that, we just need to change:
// from
const rawData = JSON.stringify({
price: "10",
vat: 1.5
})// to
const rawData = JSON.stringify({
price: 10,
vat: 1.5
})
The problem here is that you can’t simply trust blindly in the API, because in the real world:
- We don’t always have API documentation
- If you have, keeping documentation up to date is hard
- Sometimes, what you have in the documentation is not true for the API behavior
- We can also have incomplete information
- APIs can change without you noticing
Typescript doesn’t guarantee that for us
If we rewrite our code to use Typescript, we will still have the same problem:
If we run this program:
Typescript does its best to help us architecture our application and catch problems in compile-time, but at the end of the day, this code compiles to JS and we “inherit” the same problem.
But there is one major difference from the previous example:
We eliminated one degree of uncertainty by defining the shape of our data
We explicitly defined that we are expecting numbers, if the application is logging the wrong result because the API response is returning a string, this means the API is breaking the expected contract.
But what can we do then?
We can decode the data, it can be achieved by manually checking the data structure (the shape and types), like
const api = () => Promise
.resolve(JSON.parse(rawData))
.then((data) => {
if (!data.price || !data.vat) return Promise.reject("Invalid result")
if (
typeof data.price !== 'number'
|| typeof data.vat !== 'number'
) return Promise.reject("Invalid data type") return Promise.resolve(data)
})
But that just seems too much work 😅
I will show you a powerful library that helps us dealing with decoding operations.
Inspired by the Elm JSON decoding package, we can use https://github.com/nvie/decoders to help us out, this way we can create a declarative schema of our data that will be validated at runtime.
On line 1, I’m importing the library.
On lines 3–6, I’m defining the “shape” of my data structure.
On line 8, we must wrap our encoder into this guard function, ProductGuard is now the decoding function, which we can use to validate any JSON and confirm it’s an actual product. We do so by just calling ProductGuard(payload)
And in line 15, I’m making use of the ProductGuard , please note that then(ProductGuard) is the same as doing then((payload) => ProductGuard(payload))
The result is cool, if we run our code, the application will fail with a clear message:
This way, you guarantee your types at runtime.
If you are using Typescript we can also provide more information to it and validate our types statically.
What about encoders?
The process is pretty much the same.
You can rely on the library to define your data structure and validate the data before sending data out of your application.
Relying on encoders can be particularly useful for having one place to handle all data transformation for the API and can be as simple as:
const userEncoder = (user) => ({
first_name: user.firstName,
last_name: user.lastName,
products: user.products.map(productEncoder)
})And by following the pattern that was mentioned before, data should be handled in the “edges”, so sending something to the encoder should be the last thing before the actual request.
The previous encoder is ok, but definitely, it’s not safe, you can still use the same library we used for decoding:
Different example
Below, you can find a different example, please note that I simplified a lot the data structure, so things seem to be a bit redundant, but when you are dealing with bigger and more complex data, this can be particularly useful.
The example is a bit extensive, and if you are not using Typescript, you can just ignore that part (in the following articles I will show how to make use of the types in the declaration file without TS).
I hope that “pseudo-code” helps you to understand one way to structure your application, I introduced some rules on the data shape on purpose, since we have to deal with that kind of thing in the real world.
With that structure we have:
- Appointment: data type used inside of the application
- AppointmentEncoded: formatted Appointment before sending it to some request
- AppointmentFormData: this is used in the form, to define the shape of the data you’ll be manipulating and as a parameter to the encoder. Sometimes the form structure is different from what you use internally in the application, if this is not the case, you can just ignore this
With that, we’ve pretty much covered the edges, and with that contract in place, we can safely use the data inside of our app with confidence that it is impossible to have something different than what we have defined.
Conclusion
I hope I could make some good points on why and how you should use decoders and encoders.
Introducing this kind of concept into your daily work is relatively easy, since you can just evolve incrementally, at Homeday we are slowly adopting them, and until now, we were able to discover and fix a few problems with our API contracts.
If you are using something like GraphQL, then you probably don’t need to worry about this, since you have a well-defined schema.
Furthermore, I will introduce a more advanced way to encode and decode your payloads by using fp-ts and io-ts libraries.
For more information about the library used here, please check the library repository at:
And the author Vincent Driessen have an amazing introductory article about his library:
I can’t stress enough how this can make your life easier, I’ve learned about those concepts while programming in Elm in past experiences, the syntax might not be familiar at first, but I strongly recommend to reading through the following two links, it can be life-changing 😁
https://guide.elm-lang.org/effects/json.html
I’ve made this presentation at Vue.js Berlin Online Meetup #11, if you are interested, please check the video.
Thank you for reading this article, please stay tuned to the next parts.
Do you like what you see?
Give us a clap, leave a comment, or share anywhere. We appreciate your feedback.
Also, check out other articles on our blog. We deal with many interesting things:
- 404: We have a problem
- How we achieved an automated color palette that designers and developers love
- Why and how we are doing E2E Tests
- VueJS components
- No-bullshit illustrations
- VueJS Component Library: Why we created our UI component library
- Testing vue-testing-library
- and more 😊!
Check out our openings at our dev team!
Sorry for the long post, here is a cat dressed as a potato.
