Towards one-time fee #2
In the last post, I promised some business related insights in the post #2. It turned out that I had to jump into the development (and to, for me, at least, uncharted territory) so this post will be technical again. Sorry, will write it later :-)
The recent firmware and dashboard upgrade showed quite significant drop in the cloud resource utilization. But in fact, the overal decrease in lambda invocations, which was our main resource cost was roughly to one half. That made me to sit behind the drawing desk again and follow the path I was afraid in the corner of my soul. Because we needed to lower the server usage in order of magnitude, al least, the slightly less than a half did not solve that much.
The polling approach
So the path I had in my mind, and was still putting it off, for later optimizations, was complete rework of the way the nodes interact with the console. As you might remember, we had to poll the server each 10 seconds just to have reasonable response in case of user (or better admin) connect to the console. The polling request is common HTTP call which basically asks for following info: “Is there anything new for me?”. As such, there is no direct way for console to “call for her sheeps” other than waiting for them (the nodes) to call home via above mentioned call.
WebSockets to the rescue
As most of the reader would expect, the solution for this problem exist and has its name: WebSockets. WebSockets are relatively mature way, how the modern web applications can communicate with each other in a real time. Imagine any kind of chatting application over web — it would not be possible to do without WebSockets. Instead of polling the server periodically, WebSockets are a kind of “Connect once and keep the connection open”. If a WebSocket connection if established, it can be used for bi-directional message interchnage between the server and its clients, and what’s nice, there is no need for any application knowledge of the remote clients IP addresses, firewall settings etc. Just send a message to the registered connection id and you are done.
If it looks to you even more resource consuming than the polling method, you are wrong. Most of the heavy lifting, including the cipher negotiations, registrations etc., is done once, upon the WebSocket connection setup. From then it is simple a matter of sending short text messages.
Though it was originally developed for an AJAX communication within a browser, it can be used also from other, non-browser, enviromnent. For us, in the hunt for keeping the costs down it was vital to use a managed WebSocket API. The raw costs are higher, but when you consider all the other factors like human power, server scaling, it was vital. It is simply as that: If it will not be feasible to use managed WS API, it is an end of the game. Fortunatelly, AWS recently extended their API Gateway service also for WebSockets API, which can be tightly coupled with AWS Lambda.
The client side (meaning the devices connecting to it) was easy to do, as there is a bunch libraries for that purpose, as WebSockets itself is a standard protocol but on the server side, it was a different story. Amazon offers an API and some wrappers for Python, Node.js, PHP (and maybe others) but certainly not for Ruby, which was a language of my choice. Sure one could implement the server interaction in Python and call it from Ruby, or even maybe circumvent it totally and put the server in a “special client mode” but it would be ugly solution and I doubt it would perform reliably in a long term.
So I had to develop my own Ruby wrapper around the AWS API, which took me at least half of the time from the last post — the rest was to rewrite the client’s firmware to accomodate the change. As I write before, originally the WebSockets was something I had in mind “for later”, but was forced to do it right now. It delayed the project for some 14 days, the bright side is that we do have a real time capability built in, so it will be usefull for other features we plan to introduce later.
So what is the outcome?
We managed to drive the number of invocations of AWS Lambda function roughly ten fold. Provided we are comparing this with already two fold decrease from the last post, we can claim twenty fold utilization decrease in last month. Nice! The WebSocked usage seems to be negligible, at it uses a several magnitudes different unit of measuring the utilization, so the main objective has been done. Sure we have to wait for the total numbers, but as of now, seems promising.
Stay tuned!
