Technical Documentation
From the “Software Engineering Cookbook” Series — How to improve your engineering practice with a good technical documentation strategy
This article is from the “Software Engineering Cookbook” Series, aimed at helping growing engineering teams to maintain effectiveness. Today we will cover Technical Documentation, but first let’s hear from one of the newest members of the engineering team at Unikorn Inc., a fictional startup not too far removed from reality.
A junior engineer at Unikorn Inc.
My name is Matilda. I joined Unikorn Inc. just over three months ago after working for about a year and a half at another startup where the engineering practice was not so good. The boss wanted everything done quickly without much care for quality, and then we had to fight fires in production every time there was a release.
I like it here, my teammates are friendly and know their stuff, and we are building a product that many people use every day. The one thing I do not like too much is the process. I feel that sometimes there are too many rules to follow and that makes me code more slowly. I especially do not understand why we need to write so many documents to explain how things work. For me, it is easier and quicker to write and read the code directly.
Anyway, I must go now. I need to fix a bug in the Calico component which is an area I never touched before. I need some time to understand how it works before I can change it. Hopefully I won’t break anything in the process. Wish me luck!
Why we write technical documentation
There are various reasons why up-to-date technical documentation is useful and important, both internal and external to the software engineering team.
Internally, these documents offer an easy way to review the system’s architecture and ensuring new features adhere to it, they help to get new team members up to speed with the codebase more quickly, and they provide a useful reference during maintenance and bug fixing activities.
Externally, they can provide an insight into the system without exposing the full codebase, and thus are useful for different auditing activities such as due diligence before a funding round or an acquisition and to determine compliance with certain industry standards.
However for them to be effective, the technical documents need to be kept current and consistent, ensuring that the present state of the system is correctly documented and there is no contradiction across different documents on how the various components work. This takes time to do properly, ultimately eating into other activities expected from the engineers.
A good documentation strategy should balance these seemingly opposed requirements, striving to produce useful, updated documents while minimizing the upkeep overhead for the engineering team.
Organizing information
The first step in achieving this balance is to determine what kind of information you need to collect about your system, along with the best way to record it. This is somewhat dependent on the nature of the software being written, but normally revolves around these topics:
- The architecture of the software, its components and how they interact with each other, along with third party dependencies
- How each feature is implemented and how it fits in with the system
- Coding reference (e.g. documentation on utility classes, helper functions, APIs, etc.)
This is not an exhaustive list, but a subset specific to the codebase. Other documents that the team might produce also include guidelines and specifications on such topics as configuration and release management, system security, SLAs, etc. which will not be discussed in this article.
To document your code base effectively, there are different types of documents that can be written, almost directly mapping into the topics above: Architecture Documents, Technical Specifications and Code Documentation. Let’s go through each type and see how they can be used, starting from the most common of all: technical specifications.
Technical Specifications
This is the most popular and (in)famous type of technical documents in use in an engineering team.
The purpose of a technical specifications document (or tech spec) is to explain how a feature is implemented, taking on different roles at different stages of the development process:
- Before implementation : It allows people to review the plan to ensure it makes sense, as well as its compliance to the functional specifications.
- During implementation : It provides a contract between the different parts of the feature which are being developed and a single point of reference for the engineers involved in the work.
- After implementation : It serves as a reference for future maintenance and modification of the feature.
As such, it should be a living document that is updated as necessary, most commonly while the feature is under development, if for instance there is any deviation between code and specs due to unforeseen circumstances, and when doing bug fixing further down the line and the change implemented is significant enough.
A tech spec usually defines changes to multiple components within the system in order to satisfy its correlated functional requirements. As such there is a risk that different tech specs, especially if they are from different sprints, end up contradicting each other on how a single component is supposed to function.
There are different ways to mitigate this problem, the easiest and least time consuming being to leave the contradiction as is in the different documents involved and rely instead on component versioning to determine the current “truth” of the system at a given point in time.
For this to work, we need the tech specs to specifically point out the version of all components involved, remembering to increment version numbers when needed.
However, this strategy is not very effective for complex components which would end up being specified in a fragmented fashion all over the documents repository. For this kind of components, it would be better to use an architecture document (see below) to define their function in a centralized location, and update this document when changes are needed.
As usual, common sense and discretion should be applied to determine which strategy is more effective in different parts of the system.
As a side note, enforcing component versioning and explicit reference in the tech specs also encourages the team to think about the impact of each change and how to ensure backward compatibility and related migration where necessary, for instance in public-facing APIs or when modifying a database schema.
Architecture Documents
These documents define the various components in your system and their interaction, as well as setting out guidelines on how to modify them. Their main purpose is to provide a reference when implementing new features on what is already there and how to interact / integrate with it.
The architecture documents should be used when defining new features to ensure the implementation proposed is in compliance with the design of the system. They should be updated accordingly if any architecture change is envisaged, for instance when adding new components or when planning major refactoring.
Often these documents are classified as technical specifications, though in my opinion they are different enough to be treated separately when defining the documentation process. The main differences between them are as follows:
- Scope : The technical specs refer to the implementation of a specific feature, while the architecture documents look at the system and its components as whole units
- Ownership : Whereas different technical specs are usually owned by a feature lead (i.e. an engineer who leads the implementation of the related feature), the architecture document should be owned by the team’s technical authority (either a software architect, or an architecture “committee”) to ensure compliance with internal and external rules and regulations which apply more widely to the system, or even to the whole company
- Permanence : As features change, some technical specs become less relevant and used less over time. The software architecture on the other hand is always alive and evolving, and its documentation needs to be kept current. As such, different rules should apply to software architecture documents (e.g. in terms of obsolescence and regular review) and the process should clearly define them.
Code documentation
The third kind of technical documentation is the one found directly inside the code base in the form of annotations and comments written by the engineers writing or modifying the code.
Annotations are a powerful tool to automatically generate documentation that defines the detailed operation of the code (e.g. API reference). These generally take the form of comments following a certain style defined in an annotation system (e.g. JSDoc for Javascript code). Most modern IDEs (e.g. Visual Studio Code, Android Studio, etc.) can leverage these annotation systems to enable the generation of inline help based on the source code, as well as to provide autocomplete functionality.
This is very useful as it allows the engineers to quickly look up, say, the expected behavior of a helper function and its inputs / outputs without having to interrupt the development flow by opening up the function definition manually.
Ensuring these annotations are kept up-to-date is also reasonably straightforward during code review as it is easy to correlate code and annotation changes, allowing the reviewers to spot inconsistencies before they are committed to the code base.
Simple comments are also a useful documentation tool when written properly, as they can help maintenance further on. The most effective and useful comments are those found in parts of the code which are complex or otherwise difficult to understand, explaining how things work and the reasons behind specific implementation choices.
One of the most challenging aspects of commenting code is determining whether the code being written is complex enough to grant explicit explanation. This is a skill that an engineer can build over time, and can be honed through code review feedback from different members of the team.
Comments and annotations can be used to remove bloat from the tech specs and architecture documents. For instance, it is a valid strategy to keep API definition deliberately vague in these kind of documents to then fill in the gaps using annotations and automatically generated reference.
If this methodology is employed, the recommendation is to review such reference at the end of the sprint as part of the acceptance criteria for release to ensure all required documentation is present.
Choosing the right format
The language and format employed in the technical documentation also greatly affects the usefulness and maintainability of the documents involved.
In general, my suggestion is to keep the writing clear and concise, choosing a straightforward language with short, unequivocal sentences (e.g. avoiding double negatives).
For the technical specs, some resources recommend using bullet points or numbered lists to improve clarity. I never found these particularly practical, as they make the specs more tedious to read and, in the case of numbered lists, more difficult to edit if changes are needed during implementation, but your mileage may vary.
For the architecture documents, the use of diagrams such as sequence diagrams, class diagrams, etc., might cut down on the verbosity of the text and make the document more immediate and easier to consume by both the team and external reviewers, though this might make its modification slower when further revisions are needed.
In all cases, as the engineers are the primary consumers of the technical documentation, it is well worth to ensure that the documents are written with the engineering team in mind as the target audience. This is especially important if you are dealing with a distributed team which might not share the same cultural references, or the same primary language.
Measuring documentation effectiveness
Since there is a cost to writing and maintaining technical documentation, we need to have a system to check the benefit of such investment, normally expected to manifest itself across two aspects: speed and quality.
Unfortunately, it is difficult to directly correlate good documentation with either of these two measures. The best we can do is to look at them “by proxy” and through anecdotal evidence, for example:
- Get an estimate on bug fixing from an expert of a specific component and see how long it takes for a non-expert to fix the actual issue
- See how long it takes for new members of the team to get up to speed with the system, although this can be tricky as the system should be constantly evolving
- Gather feedback (e.g. in 1:1s) from the team on the quality and coverage of the documentation
Also, the documentation strategy can only be effective if the information is easy to find and easy to be updated. It is worth to provide a “cheat sheet” to the team on where to find specific information, how to create documentation when none is available for a specific feature / component, as well as the related review, approval and update procedures. Basically, ensuring your documentation process itself is well documented and that this documentation is readily available can go a long way in making the strategy more successful.
In conclusion, we can say that technical documentation is a necessary evil, especially as the team and the code base grow in size and complexity.
Keeping the process lean and the information sensibly organized can make its maintenance easier for the team and in turn result in documents which are more useful and more up to date.
I hope this article has provided some useful insights to help you implement a successful documentation strategy for your team. Feel free to share your experience and leave your thoughts below to start, or continue, the conversation.
Meanwhile, good luck on your journey and keep on making good things!
