Zero downtime log level change using Unleash
Leveraging Unleash feature toggles to change the log levels at runtime for effective troubleshooting on production without any downtime.
Logging is one of the essential pieces of any application. It provides valuable insight into what our code is doing at any point in time, which can be helpful when trying to find the root cause of an issue.
When faced with a production problem, one of the first things we do is check the logs for any errors and hopefully pinpoint the source of the issue.
When everything is working correctly, the default logging level for most of our systems and applications is INFO. This is great when you have just a few critical operations to monitor. However, it’s not so great if you’re trying to diagnose a problem in the application deployed in production. The developers may have included some basic INFO-level log output within the problematic area, but this may not include enough data to provide a thorough analysis of an erroneous transaction.
We may also have some additional DEBUG log lines (hopefully), but the bigger problem with these is that they may not be visible to us when in the production runtime. 😞
I bet most of you reading this can remember that horrible time when you were debugging an issue and suddenly realized that the log line that gave you that critical piece of information was at DEBUG level and so was unavailable. We may have six different log levels available, but they hardly bring value to the table without granular control over their configuration at runtime.
We at SAFE had an environment variable (.env) to set the log level for all the containers to maintain consistency among them. The problem with having an environment variable is that you need to recreate the containers for the change in the .env file to reflect, leading to downtime.
We as using a popular logger called Winston, and according to its official documentation, It had a way to change the log level at the runtime. Now, all we had to figure out was how to trigger it without any downtime.
We were already using a popular open-source feature management software called Unleash with a custom strategy based on the Base URL of the tenant. If our custom strategy piques your interest, then you can use our npm package too. 😛
You need to add the strategy on the Unleash server as well.
Every feature toggle in Unleash can have something magical called variants. Variants add another dimension of flexibility to feature toggles.
Say, for instance, that you’re testing out a new feature, such as an alternate sign-up form. The feature toggle would expose the form only to a select group of tenants. The variants could decide whether the tenant sees a blue or a green submit button on that form.
Putting 2 + 2 together, we came up with the solution to have a feature toggle in Unleash to override the default log level from the environment variable and add a variant to configure the new updated log level ensuring ZERO downtime. 🚀
Now let us come to the cool part, which is the actual implementation.
Let us create a new feature flag on the Unleash server called log-level-override and add BaseUrl (our custom strategy) as the strategy on Unleash.
Once this is done, we should add the tenant’s base URL to the strategy and validate whether the tenant is able to access the feature flag or not.
Using the script below, you can initialise the connection to Unleash and ensure that the feature toggle is in sync.
Once this is done, We are very close to the finish line 😎. Now go to Unleash and add a new variant to our feature flag to called default which would store our new log level. For our use case, we have kept the type JSON and payload as { "global" : "<log-level>" } signifying the new global log level across the services. If someone wants to have a particular service on a different log level, you can add that info to the payload itself, such as { "global" : "<log-level>", "<service-one>" : "service-one-log-level" }
Once the variant is added, we can leverage Unleash’s getVariant() function to get the variant, hence our new log level. 🎉
We can use the new log level and update the Winston level accordingly.
If you are still following, you might have one question in your mind. All the tenants would receive the same variant (default). Therefore, all of them would share the same log level. What If, I want to have different log levels for different tenants? We have that sorted as well, enter overrides! 😎
You can override the variant using several parameters, but our implementation uses appName i.e. the hostname of the tenant’s Base URL.
Now for each tenant, create more variants with a fixed weight set to 0 (so that only overrides take effect). For each variant, assign the log level you want as a payload.
That’s it! The application should be able to connect to Unleash, get the variant and update the log level with ZERO downtime.
I hope you enjoyed the article. Happy to answer if you have any questions. If you find it useful, please clap and follow us for more!
