Internal Libraries | Managing Tech in Large Teams
How to effectively spread the technical expertise of your few to the all
I write about my experience and learnings as I continue on my tech development journey.
This piece covers the use of internal-libraries as an essential strategy within the organisation to scale and manage tech reliably, especially when working in large teams.
Let’s start with why!
Tech companies age in dog years. Chasing the first-move, releasing fast and if it happens to break and wreak, lead an army of engineers to somehow stay afloat.
The whole chase is not really a problem if you have a premier league engineering team. But with varying experience and expertise, all the rush will lead to compromises in quality.
The business logics & functions can be verified by the QA team & the clients or the customers. But long term maintainability cannot be judged or verified by the critics. It requires looking inwards and efforts made internally.
So how do you empower your dev team with varied degrees of experience to roll-out reliable code fast? ➡️ By using Internal Libraries in your organisation
Q: What are libraries?
Pre-written pieces of code in forms of framework or APIs, commonly exposed as Functions, Interfaces or Classes. These pieces of code will put a layer of abstraction and will always produce homogenous output.
Libraries have existed since the first programming language, in-a-way every class or public function in your code is a library. But the general understanding is that a piece of reusable code can only be termed as a library if it is used by more than one distinct project and it can be plugged-in without any changes.
Libraries can be composed of hundreds of other libraries like Django, FastAPI, Express.js, SpringMVC. And sometimes could be as tiny as this python library that converts unicode to emoji.
Q: What are internal libraries?
Internal libraries are just the same, only they are maintained & secured to use privately within the organisation. Like a private GitHub code repository. And likewise, these internal-libraries are also supported by the package managers with some auth configs.
Q: How do libraries help devs?
Today, all devs are well aware of the productivity benefit when using widely-adopted and actively-maintained open-source libraries.
But to paint a picture supporting the theme here,
When a dev starts writing a web-service, instead of reinventing the wheel, they just look for a reputed web-server library in the desired programming language.
The web-server library also exposes configurations to meet the requirements which ensures internally that the exposed configs will not affect performance in any way (unless explicitly mentioned as experimental change). The library also provides docs, examples, directory structure guidance, unit-test boilerplate code and a community to refer to as needed.
This is easier and faster. It is also much safer because over the years hundreds of collective & highly experienced minds have contributed to this library and its maintenance to stay performant, efficient and secure.
Using this open web-server library, devs will start writing their HTTP API routes and their business logics underneath. And the day one of the overall product might look like this –
Q: Then why do we need internal libraries on top of these?
The internal libraries will help in the operational challenges described in Q2 above that surface when the requirements and teams will expand in size as the product grows.
Naturally the contributions will start pouring in from multiple ends, and as per Conway’s law — a new component will most likely will get introduced into this product every time a distinct team collaborates on it.
For example, our product after a while might grow to become –
Q: What are these operational challenges?
- How to efficiently and effectively spread the expertise of the particular dev(s) across all the teams and projects?
- How to maintain homogeneity among different services in terms of monitoring, lint, coding style and structure, so development and debugging is intuitive for the devs when cross-pollinating?
- How to protect proprietary algorithms from frequent undesired changes, ie. keep them abstract from direct modifications without code-owners review?
- How to safely expose a feature-management layer in your codebase for the team to do rapid development?
Q: How to efficiently and effectively spread the expertise of the particular dev(s) across all the teams and projects?
Domain level expert dev(s) shall write the internal-libraries for the teams across the organisation to use. Providing docs, example usages, package-manager compatibility, config options, limitations and tests utilities.
Also as the sole owner of this library, address any changes & discussions about or around this library.
For example, the algorithm to generate the short URL in our project can be owned by the distinct domain expert(s) and the devs can just install and use the exposed function to get a new unique short-URL.
This algorithm can undergo continuous improvements or expand into its own system. But for the rest of the system and the devs, this will still be just a library-version-bump with an exposed function; they never need to deal with underlying complexities.
Q: How to maintain homogeneity/familiarity among different services?
Under pressure to develop a feature, a dev might try to find the easiest and quickest way to prepare a new micro-service, the actual HTTP endpoint fulfilling the requirement is the goal. The foundation of the micro-service itself for long-term maintainability is an afterthought, a tech-debt to be addressed later.
And similarly, with multiple projects starting-off in-parallel and in distinct teams — their foundation will vary, so apart from the business logic, the whole project is different now which it doesn’t need to be.
This can be averted by providing the feature-development dev with a quick-start template project as an internal-library to just pick-up and still just focus on writing their feature (ie. the HTTP endpoint).
Often devs just copy-paste one of the latest projects as the base and modify / shape it for the new service accordingly. Although similar, the quick-start template approach should go further and beyond by using internal-libraries itself.
For example,
An internal-library which acts as the wrapper around the web-server library, to add essential components by default like security middlewares, logging middlewares, alerting, graceful request draining, custom metrics, APM integration, config validators, and etc.
Where the middlewares and other parts used in this wrapper could be their own internal libraries based on their complexity,
Sub-example,
the dev(s) who has debugged large system at scale will know best how to optimise logging for filtering and readability, then the dev(s) shall write a library which exposes the functions to write logs where this library internally scans messages to mark sensitive data, add extra essential tags, segregate, propagate ahead, and raise alerts, and etc.
And with all projects being derived from a quickstart template, the learning curve will reduce, devs can easily cross-pollinate to contribute or debug.
And with internal-libraries inside these templates — the latest version of the service is a version-bump away!
Q: How to protect proprietary algorithms from frequent undesired changes?
Enterprise products need the core of the system, the proprietary algorithm, and the complex parts to be extra secure from rapid changes.
An urgent requirement raised by the biggest-client in the organisation might overlap with the vacation days of the code-owner where change is required, the unreviewed rushed change without the code-owner’s review could lead to catastrophic problems.
This is unavoidable, since managers can still overwrite and bypass the code-review criteria.
But if the underlying code is kept at distance, and not be as easy to reach ~ the psychological deterrent can slow down the rush and promote caution.
And this is another way how internal-libraries can help by keeping the system more stable & secure. Check how GitHub allows setting mandatory code reviews by code-owners.
Q: How to safely expose a feature-management layer in your codebase?
Well in the previous example scenario if the code-owner is not reachable at all or the urgent requirement needs an immediate hack for the time being.
Then the abstraction will require deeper focus, and the strict code-review rules further reminded by code-versioning tools will encourage great caution.
When the hacks are unavoidable, at least they will not seep through to the core. Will remain at the upper layers where the surface-area of potential issue can be minimised.
For example, the dev can write a basic `if` condition to overwrite the result of the internal-library for the desired effect but just for the particular client that needs the change urgently.
The checklist on building internal-libraries
Configurations / Customizability
Expected outcomes are document with examples
Sanitises and validates inputs
Internal handlings to ensure unaffected performance irrespective of the config options
Configs that may affect performance are marked as *Experimental
Usability
Compatible with the package-manager of respective programming language eg. npm, pip, nuget, gem etc
The exposed elements should match the general intuition of dev and contextual project
And should just continuously improve over time
Tests
100% unit-test coverage
The example usages contain the test-cases as well
Stress and endurance tested
Actively maintained
At least secure ie. the dependabot / renovate issues are cleared promptly
Community & support
FAQs are documented and kept up-to-date
Mentions best way to reach library owner
Issues template
So the users promptly put the requisite details for registering a bug or improvement
For example- https://github.com/stevemao/github-issue-templates
Build trust
Mentions test coverage
Documents limitations
Active communication on the raised issues
& Bonus
Almost all open-source libraries that we see today were initially started as internal-library in an organisation. Publishing them to the work always helps organisation and the devs to build their technical credibility.
So if internal-libraries are built with this open-source compatibility in mind. Potentially will a good bonus!
Internal-libraries are just one of the many ways to Manage Tech in Large Teams so I hope to add more pieces on this topic next.
