Go modules in mono-repo
Balancing development flexibility with structured release process
Go modules have come a long way since the August 2018 when the first module capability was launched as part of version 2.11. Traditionally, the recommendation has been one repo = one module. But, I think, over the past year, the Go module tooling along with GOPROXY protocol and infrastructure have matured to enable good support for multiple modules. This is an important implication for developing solutions in a mono-repo. This article is not an introduction to go module or advantage or disadvantage of mono-repo.
The following are the takeaways
- Go tools use version control features like tags, branch and revisions to identify module version and generate psuedo-versions.
- Use module specific branch to create and manage module development along with release specific branches (e.g. alpha, beta) for integration testing. This may require splitting code base across multiple branches at the cost of readability (which can be addressed by approaches identified below)
- Given the out-of-box support for Version Control System within the tools, consider restricted access to your mono-repo and use of GOPROXY protocol to serve artifact for a formal release cycle of packages for external/public consumption. This approach is currently limited by lack of general tooling (e.g. github does not support GOPROXY protocol for package registry).
Modules are how Go manages dependencies.
A module is a collection of packages that are released, versioned, and distributed together. Modules may be downloaded directly from version control repositories or from module proxy servers.
A simple module represented by go.mod file in root directory of module defines the module that is being exported, dependency on the version of go compiler along with other modules that it is dependent on and corresponding version.
Version to tag mapping
Go uses branch, revisions and tags to identify the required version. In the scenario shown here, the following modules and corresponding versions can be defined
1. A go-demo-module v0.1.0 in go.mod file will translate to code being pulled corresponding to tag v0.1.0 but this code will not contain mod1 since that is identified as a different module.
2. An entry go-demo-module/mod1 v0.1.0 in go.mod file will translate to the code being pulled corresponding to tag mod1/v0.1.0.
This mechanism ensures that each module can be used independently within a repository without any ambiguity.
Establishing a standard process for working in a mono-repo helps reduce friction within the team and improve productivity of team. This is one such flow that can be used to streamline working on mono-repo across multiple go modules
It is recommended that a branch (similar to “new-module” in the flow above) be created.
This branch will contain basic files like .gitignore and other template files that would be helpful for defining new modules.
In case an existing repository is being enabled, this can be achieved by creating a new branch from first commit and then reverting the commit.
The new-module can be used to create the various release branches like alpha and beta shown here. These release branches will be used later for defining and building packages for release by defining go.mod files with specific version of modules defined in child directory as described below.
Please ensure that the branch naming convention does not follow a standard that conflicts with go versioning mechanism.
Any new module can be created at any time by branching from new-module. The module would be created in the corresponding directory. In case of standard project layout this may translate to pkg/<module>, cmd/<module> and so on.
Once tagged, these modules can then be used by other modules either within or outside the repo as shown here. In case of external dependency, a more standardized release process may be followed to ensure that all the packages being released have been adequately tested.
The release process ensures that entire development package is well integrated and tested prior to packaging and release.
This, in case of go, can be achieved by defining a virtual module with dependency on other internal modules that should be part of release and performing system integration tests. In go-demo-module, the alpha branch shows one such approach for packaging and testing. This branch contains go.mod which defines the module (i.e. go-demo-module)and associated dependencies (i.e. mod1 v0.10 and mod2 v0.1.0), along with system integration test in alpha_test.go file. The alpha.go has been added to ensure that test can be run.
An alternate way is show in dev branch which establishes a process for module developers to submit a pull request to dev branch. As part of merging pull request, the corresponding module and associated version is added/updated to go.mod as dependency. This provides a more formal release process while also increasing the overall readability of project (since the module code is available as part of branch).
In case a more formal release process (such as above) is implemented, it is important to ensure that external project don’t form dependency on modules directly. This can be achieved by making the project private and releasing tested package as zip file and serve through GOPROXY protocol.
Go modules provide a very flexible way to decompose system developed in go and simplify the development by allowing developers to develop the component independently and then performing integration testing once ready.