Using Web-Sockets for Realtime Image updates

Eric Carter
May 6 · 6 min read
Photo by Sven Brandsma on Unsplash

Once upon a time there was a website which rendered an image from the camera of a Raspberry Pi. But it was a bit naff. The user had to click a button to refresh the image, the browser requested an update from the server, then the browser started polling the server until a new image was available, then reloaded the image; meanwhile the Pi polled the server for instructions, then captured and forwarded an image to the server, the server then saved the image and set the flag the browser was polling. Ye Olde design was slow and clunky and needed to be euthanised.

There are a number of technologies available to help. I did not review them all and select the best available option, so I caveat this post by diplomatically stating “better solutions may exist”. This is the story of bashing through the problems and changes to highlight interesting aspects and gotchas.

The application and hence the blog use Javascript, React, Redux, Node and Express (How original?). Some of the gotchas are specific to these libraries but a lot will be relevant to other stacks.

Server Sent Events (SSE)

First I had a look at SSE. It was very easy to demo an API where the server could push text data to the browser. But it only supports textual data and I didn’t want to Base64 encode the image.

Make the server respond to a HTTP request with connection:keep-alive and content-type:text/event-stream headers. Then just write data without closing/ending the response
Client simply creates and subscribes to an Event-source

Web-sockets

A quick google showed that Web-sockets support binary data. So at least I wasn’t wasting my time anymore.

Given I was using Express and Web-sockets I did THIS, followed the first hit https://www.npmjs.com/package/express-ws and followed the instructions. Well, roughly followed the instructions, I was working on an existing project, so I just plumed in a new API among the existing ones.

First apply the Express-ws library to extend Express before adding routes and middlewares
Then add a new route with demo Web-socket behaviour
And simply open and subscribe to the Web-socket in a React component

Thats it. Everything worked first time and we all lived happily ev….

No wait. It didn’t work. Obviously.

Gotcha #1

I have an Nginx proxy directing web and API calls to the relevant servers, but I had not configured it for Web-Sockets.

Gotcha #2

Our existing app applies the Express app to a HTTP server instance. And having not fully read the Express-ws docs I missed the bit that said

When using a custom , you should pass it in here, so that can use it to set up the WebSocket upgrade handlers. If you don't specify a , you will only be able to use it with the server that is created automatically when you call .

Gotcha #3

Express-ws extends Express App and Routes with the .ws method and wraps all the middlewares on that route, for some inexplicable reason the Express-ws authors changed the middleware interface from (request, response, next) to (ws, request, next). This means that you cannot simply re-use existing middlewares such as my token authentication.

Wrap existing middleware to change interface for Express-ws

Gotcha #4

Express-ws updates the request.path appending ‘/.websocket’ the aim of this is to avoid collisions in the path allowing you to provide both http://example.com/api/image and ws://example.com/api/image. But all it does is move the problem to a more obscure location, the collision now happens on /image/:id because ‘.websocket’ looks like an :id wildcard. “Shirley” I suppose we could improve the :id regex to not accept what is probably not a valid ID. But I just moved the API to /api/websocket; more on that later, I promise.

Okay, so far we have a demo proving we can push messages from the server and log them in the browser. Great. How about actually updating an image.

In the old app the Raspberry Pi camera pushed the image to the server which saved it and served it to the browser. But thats not required anymore, we can just forward it directly to the waiting Web-socket.

That was too easy, but google did say Web-sockets support binary data.

Images are often embedded into web pages using the image tag with a remote URL the actual image subsequently loaded by the browser. In my case the image was used as the background of an SVG with a load of D3 magic added on top, but it was still a remote URL.

But now the actual image contents have been pushed to the browser via the Web-Socket, so a remote URL just wont work. Enter the data URL.

Data URL

You can read up on data URLs here https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs

Data URL is a scheme by which you can, among other things, store the image data directly in a really long URL.

Convert the raw jpeg image data from the Web-Socket message into a Data URL, via an arrayBuffer, then pass it to the SVG via React state.

Multiplexing

As promised lets return to the Web-Socket API /api/websocket. At the moment the examples in the blog only allow you to send an image on a dedicated Web-Socket connection. But thats not enough for me, and probably won’t work for you either.

In my app the image alone is not enough information for the browser to do the right thing. It needs context and I would like a timestamp. I also have a requirement to push some other unrelated data from server to browser. It would be pretty foolish of me to open multiple Web-Sockets when one has the required bandwidth, or to create a scheme where the first message sends the context and the next message in a pair sends the image. What I really need it so combine binary image data and control or other data in a single message.

I chose to use BSON, (like a broken record I caveat that other options are available and may be better, I only tried this one)

One big change that may not be obvious is that I have moved the Web-Socket to global state as it will be shared by many components. Here we use Redux to manage global state. I wont discuss how Redux works, except that dispatch(setImage(…)) is how we send the image (indirectly) to the view component. Lines 51:64 illustrate the BSON-Web-Socket stuff.

Gotcha #5

English people generally spell ‘serialise’ with an ‘s’ not a ‘z’

Gotcha #6

BSON decoder needed a different type of array buffer. Notice the Web-Socket type set to ‘blob’ and the change to FileReader

Thats it. It works. Let’s go home. But…

I’m and Engineer not a coder

If the network connection gets interrupted the client may sit waiting for an update that will never come and the server may hold open resources that have been forgotten by the client.

The client only needs the Web-Socket when on particular pages that have dynamically updating content. It is just wasting both client and server resources to keep a Web-Socket open permanently, so the client should close the Web-Socket when it leaves those pages. But it would also be a bit wasteful to constantly close and re-open a Web-Socket when the user bounces around.

The next few ‘features’ are just thoughts.

Sever resources are limited and should be prioritised. For instance one user should not be allowed to open multiple Web-Sockets if that prevents another user opening one. Similarly, if a Web-Socket is not being actively or regularly used its resources should be released for those users with more immediate need.

The client should be able to publish and subscribe to messages without having to worry about if it needs to open a new or reuse an existing Web-Socket. Similarly when un-subscribing from messages, something else should worry about if it is the last subscription and can close the Web-Socket

The End

The new process is conceptually and practically much simpler and more efficient than the polling approach it replaced. A large volume of code was removed as the system no longer needs to: flag polls; lockout user actions; store and serve images on request. The user experience is much slicker; the user is always presented with the most up to date image with no interaction and with no spinners.

The last few days have been a voyage of discovery for me, I wrote the blog because I thought there was enough in it to be interesting and helpful.

Any comments, suggestions, libraries that do this already, then please leave a comment.

thestartupfactory.tech

The best from Design, Tech and Business for all things Startup. You’ll love it…

Eric Carter

Written by

Eric is Head of Engineering at thestartupfactory.tech where he helps startups to build their first releases and to build the teams and processes to scale up.

thestartupfactory.tech

The best from Design, Tech and Business for all things Startup. You’ll love it…