Developing Maintainable Software: A Checklist for Developers

7 suggestions for setting up maintainability from Day 1 of application development

In my role as Engineering Manager at Capital One I work to impress the following standards on my teams to ensure that we are delivering maintainable software solutions.

1. Balance Modularization and Re-Usability

Designing a maintainable solution, calls for a modularized solution with reusable components. Targeting highly reusable components and modularization of every single feature will require expert developers, thereby increasing cost. But, these aspects will be beneficial in the long run due to the decreased cost of maintenance and flexibility to make changes. A good design should strive to balance these aspects against the requirements of the product. While most of these aspects can be handled by the product leveraging a good framework, every developer must still take these aspects into consideration while writing code.

2. Incorporate Automated Testing

Every module must include meaningful unit, functional, and regression tests.

• Unit Tests — Unit Testing may be frowned upon as it adds a lot more time to development, but it is the first line of defense when trying to make changes to code, irrespective of whether you wrote that code earlier or not. Good unit test provides an unassailable way to prove that the code works, and hence eliminates the worry of breaking something when you try to make changes. In his article Improve Java Code Coverage and Quality with Unit Tests and JaCoCo, Jon Bodner talks about how unit tests can be leveraged to ensure that code is properly tested.
• Functional Tests — Automating functional tests will help with quick and accurate validation of the requirements. These will also help in validating that any new changes to the software did not break the existing functionality.
• Regression Tests — Not every functionality needs to be tested with every deployment, especially when the software is huge and has a large set of features that require testing. In such a case, it is always good to have a set of slower, more comprehensive tests that you run to “smoke test” and validate that a new build works. This will save testing time while verifying that the changes didn’t break existing functionality in a different piece of the application.

3. Log Meaningful Events

All good software systems must have a good logging scheme, and this logging must be done with a purpose. Every log event must be comprehensive containing meaningful information. In his article Logging Wisdom: How to Log, Emil Stenqvist states that software programs must write log as if it is a journal of its execution: major branching points, processes starting, etc., errors and other unusual events. Logs must contain messages that describe what’s going on, along with the relevant context as key-value pairs. They must include relevant identifiers such as request ID’s, PID’s, user ID’s, etc.

Logs must be written so that they capture the data that is meaningful for the purpose that it is written for. Logging needs must be identified at the time of feature grooming. Few examples of motivations for logging are to:

• Capture Business Metrics
• Capture User navigation for triaging an issue
• Capture application events for monitoring application performance and health

In my career, I have seen logs like ‘I am here’. That is great, but where is the information about:

• Who you are
• Where exactly you are and
• Why you are there

Without these being available, this log is just a waste of drive space.

The motivation for logging must drive the details that go into logging. For example, if a log event is written when a user sees an error message, it is important to log the user ID, date, and time of the error, as well as the details of system state or data that resulted in the user seeing that error message. It is important to be careful not to store sensitive information in logs or encrypt them if they are needed.

4. Display Meaningful User Message

Error messages displayed to the user must help the user understand why they received the error and what steps they can take to resolve the error. In addition, if these errors are caused by system problems, then all relevant data to understand what caused the error must be logged in the application logs. This will help the support teams to quickly identify why the error happened.

Effort must be made to make messages unique so when the user has questions about it, support teams can quickly provide an answer rather than trying to identify which one of the many reasons could have caused the issue.

5. Implement Application and Infrastructure Monitoring

Ensuring that infrastructure and application monitoring is designed and implemented at the time of application development is a key criterion to making good maintainable software. While infrastructure monitoring can be handled by monitoring aspects like memory, CPU utilization, number of instances, etc., application monitoring requires deeper understanding of the application domain and the instrumentation of the application. In his blogpost Logging V. Instrumentation, Peter Bourgon talks about when to use logging versus when to use instrumentation to ultimately increase the system observability.

Application monitoring must focus on proactively identifying the degradation in availability and performance of the application. A downtick or uptick of the monitored parameters must be watched. A highly scalable application may endure a DDOS attack and the attack might go unnoticed in the infrastructure monitoring. Whereas, if the application is being monitored for user traffic, then the DDOS attack will be rendered as a spike in users hitting the platform, which can be leveraged to alert support teams about a possible abnormality.

In addition to tracking the real user transactions, a good monitoring set up will also try to proactively track availability and performance using synthetic transactions. Synthetic transactions are performed by monitoring tools and are useful in understanding the degradation of the systems even when there are no users using the system.

To summarize, a sound application and infrastructure monitoring design will contain:

• Monitoring tools to read logs and perform real time user monitoring, as well as proactive synthetic monitoring of transactions.
• Dashboards that show:
• Trend & thresholds for application performance & availability for the system, and individual components.
• Frequency and count of different error messages. Example: errors displayed to the user or errors when APIs are being tried multiple times, indicating degradation in the API performance.
• Alerts when the thresholds are breached.

6. Maintain Documentation

Good documentation is required for developing and maintaining a good maintainable software solution. Some of the documentation that will be good to have right from the start are:

• Coding Standards — Include standards for naming conventions, styles, requirements for unit tests, functional tests, level of comments in code, etc. Programming languages are starting to adopt standards for this, others have standards defined by large companies. So, adopt these standards rather than re-inventing the wheel.
• Contribution Guidelines — This becomes very important if the project code is updated by more than one team, or even within the team, as not all the contributors are likely to be committers to the project.
• Project Governance — Include how many reviews will be needed to commit the code, version control, who will be committers vs contributors, decisions when there is conflict on design, etc.
• Feature Descriptions — This is one of the key details that gets buried in design drawings and JIRA stories, making it difficult to know what exactly a module is intended for, especially when the developers and product managers change. It is beneficial to maintain high level documentation on the features, making note of some of the key decisions and discussions that went into their design and development. This includes:
• Infrastructure and Architecture Document
• Deployment procedure
• Support FAQs

7. Demand Development Time for Eliminating Tech Debt

Like anything in this world, even the best of intentions to tackle maintainability does not guarantee that you won’t have tech debt, which you will have to tackle retrospectively. It can be a tough sell to ask the product team to support a refactor work, that doesn’t render any new functionality but is aimed at solely making a developer’s life better and making code changes in future less buggy.

A high incident ticket count, the need to change error messaging for the customer’s benefit, or a need to enhance functionality are all times when a development team can easily tackle their tech debt items to increase maintainability. Aside from these opportunities, I suggest that every development team must strive to ask for a 10–20% allocation to work on tech debt items.

Originally published at https://www.capitalone.com.

DISCLOSURE STATEMENT: © 2020 Capital One. Opinions are those of the individual author. Unless noted otherwise in this post, Capital One is not affiliated with, nor endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are property of their respective owners.