JavaScript. When the Invisible LINE SEPARATOR Breaks Your Integration With 3rd Party API

Alexander Ravikovich
Jan 26 · 5 min read

Preamble

In the company, I currently work we are dealing with many external APIs. This short story is about API integration that sometimes didn’t work because of a single invisible character. I’ll not provide the name of the company, so let just refer to it as a “Company” :)

In this specific case, the Company knows to send webhooks to our exposed and secured HTTP endpoints.

One day we started to see error logs, meaning we cannot verify the request signature, and that’s where our small journey starts…

Spoiler: “People still copy text from Microsoft Word with all these hidden characters…”

Webhooks

When we receive a webhook, we are required to do sha256 signature verification for security reasons. We must be sure the request was done by the Company and nobody changed it (MITM). Pretty simple stuff that described in Company’s documentation with step by step explanations how to implement it.

So far, so good.

But, we started to see that sometimes we are failing to verify the signature. There were not many failures, but it was an important flow, so we needed to fix it.

We did our first checks, didn’t find anything that can immediately show us the problem. Our verification code looks something like that. What can be wrong there? Especially when 98% of verifications are successful.

Simplified code just to show how simple is the verification itself

We decided to ask Company’s dev support to investigate, why do they send us the wrong signature sometimes.

The Company’s developers did their checks and replied that they don’t see any issues on their side, and they successfully verified the same data we are failing to verify.

After a few ping-pongs, we asked them to show us how do they create and verify the signature for some specific payload.

They send us a working example at Ruby, where we found the tiny difference in the payload they used in the example. The code itself is not relevant, so I don't include it here. What is relevant is the HTTP body payload.

This is not a real body. It’s just an example that reproduces the issue…

And this is how we saw the payload we sent them.

Our first reaction was like…

Why do they have \u2028 inside the string? We don’t have such a thing in our payload. Why do they change our example and claim it works? But, maybe it was done automatically…

We decided to see how does our payload looks in hex. Sublime to rescue…

Hmmm… There is an invisible character here. Googling helped us to find out what does it mean. It was a line separator character.

But still, by the end of the day, the payload is just a bunch of bytes. And if we apply the same actions on these bytes, and by actions, I mean generating signature using sha256 function, the result should be the same.

So maybe the issue is at one step before? What do we have before? We have a web server, that receives HTTP requests, parses the HTTP Protocol raw data to extract the body and parameters, and then, parses the JSON string (the body) to provide us Javascript Plain Object in the request object of Express.js middleware (forget to mention we use NODE.js as backend).

In our authorization middleware, we encode (stringify) body object to get a JSON string that will be used as a payload to the signature generation function.

By playing with JSON.parse() and JSON.stringify() locally and comparing it with GO, we did found that it works slightly differently in Javascript.

Here you can see that decoding/encoding using GO, the result string will differ from the source string. GO will automatically escape problematic character.

Quick diving to GO’s sources, and we can see where and why it was done this way. You also can see there is another symbol (PARAGRAPH SEPARATOR) that also will be escaped.

It is logical to assume that the Company’s software works similarly.

But we use NODE.js, and Javascript will do things differently. If the symbols are not escaped, Javascript will keep it as is. If the data was already escaped, Javascript will unescape it and we’ll get a string with invisible characters. Easy to check in the chrome dev-tools console.

We didn’t think we found something new, so by quick googling we found an exact description of the issue, and fixed it by using a simple replace.

P.S.

People still use Microsoft Word for writing text and then copy it to the web forms. In this case, content created by the user caused failures in some flows.

There’s no one to blame in this specific case. Actually, this is a good example of our reality, when working with APIs. Sometimes API will fail, for any reason. You just need to be ready to deal with it.

For me, it was a tiny and “cool” bug to fix that I really enjoyed in these annoying COVID-19 lockdown days :)

The Startup

Get smarter at building your thing. Join The Startup’s +789K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +789K followers.

Alexander Ravikovich

Written by

In GO we trust. Software Engineer.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +789K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store