“Houston, we have a problem”: 6 Steps to Manage Errors Better
by Joojo Dadzebo Amoquandoh Dontoh
There is absolutely no engineer or developer who always writes error-free code. In software engineering, errors and bugs will always be prevalent and that can definitely be annoying. Errors can appear in any part of the implementation process right up through sandbox environments to production or already shipped products. The main aim of this article is to try and help prevent, mitigate and handle errors properly, by sharing some of the tips and tricks we use at airasia Superapp.
The main areas this article will focus on are:
* Detailed self-explanatory errors
* Logs
* Proper notifications management
* Quick response
* Thorough Root Cause Analysis
* Long-term fixes
Detailed self-explanatory errors
Errors can be difficult and extremely cumbersome to find. The last thing you want after combing through lines of code or log statements is an error message so vague you wished you hadn’t found it. Worse still is when more than one service is involved.
At the very least, you should consider being clear and unambiguous with your error responses and relating status codes.
For example, let’s say a user who does not have the access to a particular resource tries to access it; while trying to access that resource, your service keeps responding with status code of 204 and an error message of no content.
Any new dev or perhaps your future self who may not know that the user does not have permission, may begin to debug for why the resource isn’t being sent back since the response is misleading. A more appropriate response may be a 401 status code with a response message of “not authorized”.
Logs
Logs are an essential aspect of running through the inner workings of application especially during investigations and troubleshooting. Many of us may have found ourselves in circumstances where the production server has been down, forcing us to rummage through the logs for causative issues. This is enough a reason to make sure that logs are clear and run through most functions of your application to provide an adequate stack map of previous actions. Thankfully, there are many well-known packages which already handle most of the heavy lifting for us. Winston is a good example of such loggers. You may also want to consider the various levels of logs throughout the application:
* Info-level logs: These are most likely user-driven actions that happen while the program runs. These logs could also serve as code smells for highlighting potential causes of errors
* Notice-level logs: These logs are to be done when a notable non-fatal error occurs
* Warning-level logs: These logs are usually made to highlight events that may drive the application into a failed state. For example these errors could be made to send alerts when a notable error such as a status 500 happens more than 10 times in 5 mins or when cache storage is reaching its capacity.
* Error-level logs: At this level, anything that is regarded as an error must be logged. This may be an internal error or an error response.
It is also important to add context to your log messages. Error messages without context are basically noise since most of these messages cannot contribute to investigation or troubleshooting efforts. Context can be added to a log message by subtly stating the surrounding actions or processes of the error.
For example, imagine working on a banking application that throws an error:
‘Transaction failed’. In many ways this error is very vague because there are many types of transactions and routines that happen in a banking application. Noting the transaction ID and the type of transaction provides greater meaning to the log. i.e. ‘userOnlinePayment Transaction 394fs948dne7ndue84nd74: failed’.
Proper notifications management
As said earlier, errors are part of the development process and can’t be entirely removed, therefore once they occur at an alarming rate, adequate structuring to inform all relevant engineers is important. This in many ways is similar to warning and notice logs. Various cloud providers such as Google Cloud Platform provide out-of-the-box solutions such as monitoring tools and channels to inform engineers whenever necessary.
Notification overload may however arise from this if not controlled. Therefore attaching an infrastructure to the right application is important. This not only relieves some of the engineers from excessive notifications, but also ensures that these notifications get to the right engineers. Managing the frequency of notifications also reduces desensitization to them.
Quick response
For most companies, server/service downtime means financial loss which is bad for everyone. Therefore reacting quickly in such moments is of utmost importance. All the earlier points lead up to this one because they contribute to troubleshooting efforts. Troubleshooting efforts are important to not only find the errors but to assess the severity of the error and how long is needed to solve it. This information could be the deciding factor between a rollback or a total shutdown. In some cases, a rollback may be good because it allows for a quick dive into the RCA to generate long-term fixes as quickly as possible.
Thorough root cause analysis (RCA) discussion
After a fix has been deployed or a roll-out done, the next course of action is to prevent the error from happening again. RCA discussions are very key to understanding the surrounding concepts of the problem while figuring out that point of failure along with others if there are any. RCAs are better done as a group since the outcomes provide a more diverse perspectives to the problem. It is usually followed by various debugging strategies such as using print statements, commenting blocks of code out, using debugging tools and automated tests. An RCA also means providing long-term solutions such as locating all contributing factors including devs who may need extra training or mentorship.
Long-term fixes
Unlike short-term fixes that handle the immediate problem, long-term fixes are set in place to make sure errors do not recur while strengthening various failure points.
An example of the difference between a short and long term fix is as follows. Let’s say two helper methods call one particular model method to insert data into the DB. However, just one of these helper methods have implemented a mandatory data mutation before the data is to be stored. This creates the problem of data inconsistency in the DB which affects endpoints that are retrieving this information.
The somewhat short-term fix will be to implement the said data mutation in the functions that do not have it to stop the entry of dirty data into the DB. On the other hand, the long term fix will be to move the data mutation logic into the model method to prevent the issue from ever happening and then writing a script to cleanse the already inserted data if the data is frequently used.
In this example the long term fix not only solves the issue but also enforces a proper code structure and industry standards.
These steps help to put most errors to bed and make the whole development process more manageable. Share your thoughts in the comments if you have any! Catch you in the next one 😁😁