Once upon a time there was a website which rendered an image from the camera of a Raspberry Pi. But it was a bit naff. The user had to click a button to refresh the image, the browser requested an update from the server, then the browser started polling the server until a new image was available, then reloaded the image; meanwhile the Pi polled the server for instructions, then captured and forwarded an image to the server, the server then saved the image and set the flag the browser was polling. Ye Olde design was slow and clunky and needed to be euthanised.
There are a number of technologies available to help. I did not review them all and select the best available option, so I caveat this post by diplomatically stating “better solutions may exist”. This is the story of bashing through the problems and changes to highlight interesting aspects and gotchas.
Server Sent Events (SSE)
First I had a look at SSE. It was very easy to demo an API where the server could push text data to the browser. But it only supports textual data and I didn’t want to Base64 encode the image.
A quick google showed that Web-sockets support binary data. So at least I wasn’t wasting my time anymore.
Given I was using Express and Web-sockets I did THIS, followed the first hit https://www.npmjs.com/package/express-ws and followed the instructions. Well, roughly followed the instructions, I was working on an existing project, so I just plumed in a new API among the existing ones.
Thats it. Everything worked first time and we all lived happily ev….
No wait. It didn’t work. Obviously.
I have an Nginx proxy directing web and API calls to the relevant servers, but I had not configured it for Web-Sockets.
Our existing app applies the Express app to a HTTP server instance. And having not fully read the Express-ws docs I missed the bit that said
When using a custom
http.Server, you should pass it in here, so that
express-wscan use it to set up the WebSocket upgrade handlers. If you don't specify a
server, you will only be able to use it with the server that is created automatically when you call
Express-ws extends Express App and Routes with the .ws method and wraps all the middlewares on that route, for some inexplicable reason the Express-ws authors changed the middleware interface from (request, response, next) to (ws, request, next). This means that you cannot simply re-use existing middlewares such as my token authentication.
Express-ws updates the request.path appending ‘/.websocket’ the aim of this is to avoid collisions in the path allowing you to provide both http://example.com/api/image and ws://example.com/api/image. But all it does is move the problem to a more obscure location, the collision now happens on /image/:id because ‘.websocket’ looks like an :id wildcard. “Shirley” I suppose we could improve the :id regex to not accept what is probably not a valid ID. But I just moved the API to /api/websocket; more on that later, I promise.
Okay, so far we have a demo proving we can push messages from the server and log them in the browser. Great. How about actually updating an image.
In the old app the Raspberry Pi camera pushed the image to the server which saved it and served it to the browser. But thats not required anymore, we can just forward it directly to the waiting Web-socket.
Images are often embedded into web pages using the image tag with a remote URL the actual image subsequently loaded by the browser. In my case the image was used as the background of an SVG with a load of D3 magic added on top, but it was still a remote URL.
But now the actual image contents have been pushed to the browser via the Web-Socket, so a remote URL just wont work. Enter the data URL.
You can read up on data URLs here https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs
Data URL is a scheme by which you can, among other things, store the image data directly in a really long URL.
As promised lets return to the Web-Socket API /api/websocket. At the moment the examples in the blog only allow you to send an image on a dedicated Web-Socket connection. But thats not enough for me, and probably won’t work for you either.
In my app the image alone is not enough information for the browser to do the right thing. It needs context and I would like a timestamp. I also have a requirement to push some other unrelated data from server to browser. It would be pretty foolish of me to open multiple Web-Sockets when one has the required bandwidth, or to create a scheme where the first message sends the context and the next message in a pair sends the image. What I really need it so combine binary image data and control or other data in a single message.
I chose to use BSON, (like a broken record I caveat that other options are available and may be better, I only tried this one)
One big change that may not be obvious is that I have moved the Web-Socket to global state as it will be shared by many components. Here we use Redux to manage global state. I wont discuss how Redux works, except that dispatch(setImage(…)) is how we send the image (indirectly) to the view component. Lines 51:64 illustrate the BSON-Web-Socket stuff.
English people generally spell ‘serialise’ with an ‘s’ not a ‘z’
BSON decoder needed a different type of array buffer. Notice the Web-Socket type set to ‘blob’ and the change to FileReader
Thats it. It works. Let’s go home. But…
I’m and Engineer not a coder
If the network connection gets interrupted the client may sit waiting for an update that will never come and the server may hold open resources that have been forgotten by the client.
The client only needs the Web-Socket when on particular pages that have dynamically updating content. It is just wasting both client and server resources to keep a Web-Socket open permanently, so the client should close the Web-Socket when it leaves those pages. But it would also be a bit wasteful to constantly close and re-open a Web-Socket when the user bounces around.
The next few ‘features’ are just thoughts.
Sever resources are limited and should be prioritised. For instance one user should not be allowed to open multiple Web-Sockets if that prevents another user opening one. Similarly, if a Web-Socket is not being actively or regularly used its resources should be released for those users with more immediate need.
The client should be able to publish and subscribe to messages without having to worry about if it needs to open a new or reuse an existing Web-Socket. Similarly when un-subscribing from messages, something else should worry about if it is the last subscription and can close the Web-Socket
The new process is conceptually and practically much simpler and more efficient than the polling approach it replaced. A large volume of code was removed as the system no longer needs to: flag polls; lockout user actions; store and serve images on request. The user experience is much slicker; the user is always presented with the most up to date image with no interaction and with no spinners.
The last few days have been a voyage of discovery for me, I wrote the blog because I thought there was enough in it to be interesting and helpful.
Any comments, suggestions, libraries that do this already, then please leave a comment.