What to Avoid When Creating Internal Company Code Packages

Code packages (or libraries, pick your favorite term) are a great way to share and reuse code. Without them, we would have to re-implement the same functionality over and over again in every system, app or service we build. Most of us rely on them daily to take care of basic, non-unique or app-specific functionality like handling web requests, rendering html, performing authentication and authorization and much, much more. I would even dare to say that in a lot of cases, the amount of packages code running in production surpasses the amount of custom, business logic code.

It’s no wonder, then, that many companies build and maintain their own set of internal packages. It makes a lot of sense — code duplication is usually a bad thing, and you want to minimize it.

But as you probably know, everything comes at a price, and writing and maintaining packages is no different. Throughout my career, I’ve seen many cases where internal packages may have actually hurt productivity, and became a burden which does more harm than good.

Looking back on the internal packages I’ve used throughout the years and analyzing what made them good or bad, I compiled a list of best practices and things to avoid when considering which internal packages to create, and how to maximize their usefulness. This is the first of two blog posts on the subject, and in this one I’m going to discuss which types of packages you want to avoid.

Why would I want to avoid creating internal packages?

As mentioned earlier, there are no free lunches. There is a cost associated with creating and maintaining internal packages. That cost manifests itself in several ways:

1. Using a package (any package, not just internal ones) means introducing a dependency to your codebase. And managing dependencies correctly, especially when you have a lot of them, can be tricky and costly.
2. There is more to creating an internal package than just taking some code and packaging it in a reusable way. You also need to take care of documentation, versioning, release process, backward compatibility, bug fixing, adding new features, managing ownership, publishing changelogs and release notes (the list is not exhaustive). Not to mention that sometimes you will have to update the clients of your package yourself because you need to introduce a breaking change and no one else will have time to do the upgrade in their services. All those things take time, and that is time you could have spent working on your core business, not on infrastructure. You need to consider whether the benefit of reusing this code outweighs the costs of these activities. Over the years, I’ve identified several types of packages which, in my opinion, aren’t worth the cost and should be avoided.

Wrappers over 3rd party SDKs

I’ve seen many examples of this type of package. It usually starts with innocent thoughts like “What if one day we will want to change *current 3rd party* with *some other 3rd party*? It will be much easier to do if we abstract it away using our own wrapper”. Or “Hey, the API of this 3rd party SDK seems complex, we need something much simpler, so we should wrap it”. There are several problems with this approach:

1. The 3rd party you are using has probably dedicated a lot of thought and work into the API of its SDK. There are usually good reasons why things are the way they are. It’s funny to think you know what is the right API better than the engineers who work on this SDK for a living. I’m not suggesting every API of every SDK is perfect, but I would think long hard before deciding that my use case is so different that it requires a different API.
2. By abstracting the SDK behind a wrapper, you limit yourself to only using capabilities and features explicitly exposed by your wrapper. This means you either lose by not being able to use features that the 3rd party SDK has to offer, or you lose by having to constantly expose more and more capabilities from your wrapper, which means more work and changes, and eventually defeats the original purpose of the wrapper.

Another common thought I’ve encountered regarding these types of packages is providing a common configurations for the SDK, to be shared by the entire engineering organization, without having to reconfigure everything each time the SDK is introduced in a new service or app. While I agree this might be useful, I do think this benefit isn’t worth the cost of maintaining a package. Additionally, depending on how the wrapper is implemented, it might make it difficult (or even impossible) to change the configuration in specific places you might actually want to deviate from the standard (which is a similar drawback to what is described in point #2 above).

An approach I’ve seen work in some cases, depending on the type of 3rd party you are integrating with, is to create a service that will be used as a proxy to that 3rd party, and having other apps and services in the system only communicate with the proxy. This approach doesn’t solve the “hiding capabilities” problem, and has other drawbacks like having to operate another service, but at least it doesn’t introduce more package dependencies to the system.

Shared bootstrapping packages

Similar to providing shared configuration for 3rd party SDKs, these packages offer common bootstrapping code that can be reused across services. Things like initializing a db connection, message broker connection, logging and monitoring are all bundled together inside a single package. These types of packages do tend to reduce the amount of bootstrapping code in each service that uses them, however I think they are one of the worst kinds of packages:

1. They tend to bring with them A TON of dependencies, some of which you don’t even really need for your service.
2. They make it extremely hard to have a service that deviates from the standard tech stack or that requires special configuration
3. I’ve seen many cases where they require things from your service which you might not have needed otherwise. Some examples might be requiring some env vars to be defined or having a specific directory structure.

Given the above points, I tend to recommend against having such packages. Instead, I prefer having some sort of scaffolding tool, template, or an example service to look at and copy from when creating a new service. Yes, it might introduce some duplication, but in this case the cost of duplication is negligible compared to the issues that bootstrapping packages cause.

Utils packages

In the context of this blog post, there are two types of utils packages I want to discuss: general utils packages and specific utils packages

General utils

Organizations usually have a small number (mostly even just one) of general utils packages. They are usually called *organization-name*.utils, and are almost always bad. They tend to be a collection of unrelated and random functionalities, which happened to be needed in two different places, and no one had a better idea of where to put it. These packages are usually used in most of the organization’s services, and most of the time have a lot of dependencies. You should avoid these packages as much as you can. If you want to share some functionality between services, either create a specific utils package for it (discussed and explained in the following section) or, god forbid, copy paste the implementation. Yes, you read it right — in some cases it’s better to copy paste code than to introduce a dependency or to add more random things to an already bloated package (sometimes I wish DRY was actually DRYTM: Don’t Repeat Yourself Too Much). Also, in many cases, you can find similar functionality in open sources packages, which are probably better tested and documented.

Specific utils

These packages come in the form of *organization-name*.*functionality*. For example: *organization-name*.networking, *organization-name*.listUtils, *organization-name*.stringUtils etc. I actually think these packages are useful and are one of the cases you should create an internal package for reusability, but there are some things to consider first:

1. If the implemented functionality is general and not domain specific, there are two options: Either an open source package implementing this functionality exists, or, in some rare cases, it doesn’t. If it exists — just use it and save yourself the development and maintenance costs. If a package for this functionality doesn’t exist — it’s best to build it as a public, open-source package. Doing so will result in a higher quality package, as it will most likely be built in a generic way, without making assumptions that are only true for your system.
   An additional benefit of open sourcing the package is you get to give back to the community, and maybe make the lives of other engineers a little bit easier (you yourself probably use packages written by others every day, so it can be nice to give back).
2. If the implemented functionality is domain specific, why is it written in a package and is not part of a specific service? This point is heavily dependent on your architecture, but if you need the same business logic in several places, it’s worth looking into whether something in the system architecture might need to change. Also, make sure that these kind of packages don’t do any side effects like publishing messages and making changes in a DB. It will just be confusing, hard to debug, and make them hard to reuse and maintain.

The above are the top examples that came to mind when thinking about types of packages I would avoid. There are probably more examples. Additionally, there is no one absolute truth. I’m sure there are some examples that would contradict what I wrote, and use cases where these types of package were in fact useful. Your mileage may vary. The important thing for me in this post is to make you think about those packages and their drawbacks, and maybe next time prevent you from doing something automatically just because on the surface it’s considered a good thing.

In the next post, I’ll discuss best practices and tips for making the internal packages you do create useful, easy to use, and easier to maintain.