Git surgery to retain history

One of my co-workers at NN Group created a Proof of Concept to transform an old Java codebase into a mavenized setup and split it up into smaller pieces. To do so he started with copy-pasting the original files into a new git repository (🤐) and slowly committing changes to get the maven setup working 🎉 . All the while work was continuing in the original codebase and both had diverged from each other… 🤔 Git surgery to the rescue! 🩺

➡️ This article assumes you are comfortable with day-to-day usage of git, have a working understanding of the CLI and how branching/merging works.

Our situation 😓

The original repository containing multiple applications without maven (simplified):

README.md 
JavaSource 
Configuration/application-1 
Configuration/application-2 
Configuration/application-...n

The JavaSource folder contained generic code which could be retrieved using maven.

The target mavenized repository for application-1 (simplified):

README.md 
pom.xml 
mvn/
src/main/configuration/application-1 <-- containing the application-1 Configuration files

Rough steps taken to get it working with maven:

• Contents of JavaSource was packaged and moved to a maven dependency
• Application configurations application-{1..n} were split up into separate repositories, files copied into it without history
• Iterate, tweak until working setup emerged

The plan 🩹🥳

Ideally, you would have changes be based on a branch of your original codebase so you can merge them as a new feature. Since that was not the case we needed to bring in some serious git tools to rewrite history to our liking.

• We needed to remove the files inside /src/main/configuration/application-1 copy-pasted into the mavenized setup to prevent conflicts and confusion
• We needed to extract only the history + changes of the files application-1 in our source repository
• merge the mavenized changeset over the original files so they become complete again

These steps will result in retaining the full file history of all application-1 files. The downside is that the commits where the mavenizing setup was created are not atomic - the source files are absent. This was taken as an acceptable tradeoff.

Put on your scrubs 👩‍⚕️

First things first; make two local checkouts in a working directory so we do not break anything unrelated. Because git is distributed we can do all the following operations on a local copy on your machine and only when you are satisfied push it back to your version control system (Github, Gitlab, ..)

mkdir git-surgery && cd git-surgery 
git clone git@github.com:crunchie84/blogpost-git-surgery-source.git source 
cd source

Let's check our git history:

Extract our application from the source repository

Okay, we need to clean this up so we only have the application-1 part which we need. To do so we can use a very handy command existing in git: git substree split which allows us to extract a folder out of our repository and place only those changes in a separate branch:

git subtree split -P Configuration/application-1 -b rewritten-history-application-1

After executing this command the history now looks as the following:

The newly created branch rewritten-history-application-1 only contains commits (or the part of a commit) which involved the folder Configuration/application-1. It is noteworthy to observe that the new branch does not share a common ancestor with the original master branch because it is a total rewrite of history.

Prepare our new repository

We are going to take the rewritten history of our source repository and use this as the master branch of our 'new' repository (which we are going to mavenize).

# back to the root directory `/git-surgery` 
cd .. mkdir mavenized-solution 
cd mavenized-solution 
git init -b master 
git pull ../source rewritten-history-application-1

We now have pulled the local filesystem based git repository source with specified branch rewritten-history-application-1 into our (empty) master branch and as a result a clean history without any dangling commits:

A side effect of subtree split is that all the files in the extracted folder are placed top level in your commits which we are going to address in a bit:

As a final preparation, we are going to move the files as if they have always lived in src/main/configuration/application-1 to make the history a bit more readable. To do so we can use git filter-branch to rewrite where the files have been all their lives:

# we are in the /git-surgery/mavenized-solution folder 
git filter-branch --force --prune-empty --tree-filter ' dir="src/main/configuration/application-1" if [ ! -e "${dir}" ] then mkdir -p "${dir}" git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files "${dir}" fi'

This command will iterate over the commits and execute the passed script for every one of those. Files will be moved to the correct place before the commit is rewritten.

⚠️ You will get a big warning about the side effects and possible gotchas with filter-branch but for our task, it will suffice:

Now onwards to prepare our mavenized-poc to merge onto our prepared repository!

Cleaning up the mavenized PoC repository

We are going to clone the repository and remove the copy and pasted files of application-1 from the commit history.

cd .. # back to the git-surgery root folder 
git clone git@github.com:crunchie84/blogpost-git-surgery-poc-target.git mavenized-poc

Original history:

We are going to use the community plugin filter-repo which you can install using brew install filter-repo (on MacOS) to ease our syntax what we want to do:

cd mavenized-poc 
git filter-repo --path src/main/configuration/application-1 --invert-paths

We are filtering out all (parts of) commits that have anything to do with the folder containing the copied application-1 source files. The result is a clean history of only the steps to get the mavenized setup working:

The only thing which we need to do is apply the maven setup to our extracted application-1

Merging our cleaned up mavenized setup onto our application repository

We are going to re-use the git pull trick we have used before to get commits from a different repository into ours. But with a twist:

# working dir = /git-surgery/mavenized-poc 
git checkout -b mavenizing-application 
cd ../mavenized-solution 
git pull ../mavenized-poc mavenizing-application --allow-unrelated-histories

First, we make a branch in our clean mavenized setup repository because the name will end up in our commit history and this is an important piece of information. The true magic resides in --allow-unrelated-histories Given that git pull is a shorthand for git fetch && git merge this option allows us to merge unrelated histories. Normally git will always look for a common ancestor when merging but that does not mean it can not merge without!

After invoking the git pull you should be presented with a git merge commit message dialog in which you can add as much extra information that you deem relevant.

Now we can observe in our git commit history what happened:

Two unrelated histories have been merged together retaining the commit history of both. All that is left is to test it out locally. When satisfied you can now (force) push it to your origin as the new repository 🎉

Surgery success, patient dismissed! 🚀

Parting thoughts

• It would have been easier if the proof of concept had been directly created as a branch on the original repository. When we were at this point that was already water under the bridge 😬
• Because of the copy-pasted files it proved to be very difficult to re-attach the new repository directly as a branch onto the original repository
• Taking the original application repository, creating a branch and then copy-pasting the mavenized proof-of-concept codebase over it would also be an (easy) option but then we would lose the history of those changes ⚖️… We opted for a solution that tried to retain both histories

References

• filter-repo community package - https://github.com/newren/git-filter-repo
• merging unrelated git histories — https://developpaper.com/git-pull-merges-two-branches-without-common-ancestor/
• git subtree explained on stack overflow - https://stackoverflow.com/questions/359424/detach-move-subdirectory-into-separate-git-repository/17864475#17864475

Git repositories used as an example in this blogpost

• https://github.com/crunchie84/blogpost-git-surgery-poc-target
• https://github.com/crunchie84/blogpost-git-surgery-source

The complete script explained in this article:

• https://gist.github.com/crunchie84/a070f66a3af57d918e4c345d0ec5ea9b

Originally published at https://markswanderingthoughts.nl on January 21, 2021.