When Atlassian prompted us to switch to git

Published in

Esker-Labs

3 min readJun 11, 2020

The recent changes in Atlassian strategy were the sign we waited for to switch some 1300 projects repositories from Mercurial to Git. In this article we propose to share some pitfalls to avoid.

When it comes to migrating a repository you can simply do each step manually. When you need to migrate several hundreds or thousands of repositories you’d better automate the whole process. So we wrote a script to migrate all our projects to Git (while remaining hosted on Bitbucket).

Migrating a repository

To migrate a repository we tried many scripts found on the internet, however the one that really worked was hg-git (https://bitbucket.org/durin42/hg-git/src/default/README.md).

As I didn’t want my computer to be the migrating slave, I preferred to setup a new Vm.

Installing Python (2.7) and the prerequisites (Mercurial and Dulwich)
Retrieving the sources of hg-git

To save your time, note that we had to manually install the latest version of dulwich (the one proposed by default in pip doesn’t work very well). Please test your installation by migrating manually some hg repositories, you may have some small things to configure.

Algorithm

For all the repositories 
  Clone mercurial repository 
  Modify the hgrc to include the hg-git extension
  Rename the existing hg repository in Bitbucket
  Add hg bookmarks to keep the branches
  Create the new Git repository
  Clone the git repository
  Migrate hg into git with hg-git
  Add gitignore file in every branches
  Push the new git repository
  Clean folders(or keep it :) )

Let’s go through the main functions to help you identify the pitfalls.

Modify the hgrc to include the hg-git extension

In the hgrc file of our mercurial repository we add these few lines to configure hg-git.
Note that we also add an option “branch_bookmark_suffix” that will later allow us to migrate our mercurial branches into git branches.

[extensions]
hggit=PATH_TO_HGGIT_TOOL[git]
branch_bookmark_suffix=_bookmark

Add hg bookmarks to keep branches

The options previously configured allow us to automatically migrate the mercurial branches to git branches. The only thing we have to do is to create bookmarks (with the suffix defined) on heads of all of our branches.

branches = subprocess.check_output(["hg","branches", "-T{branch}$"])
    branches = branches.split("$")    for branch in branches:
        if(branch != ""):
            print("Handling branch : "+branch)
            call(["hg", "update", "-C" , branch])
            branch = branch.replace(' ','_')
            branch = branch.replace('?','_')
            branch = branch.replace('*','_')
            branch = branch.replace('"','_')
            branch = branch.replace('\"','_')
            branch = branch.replace('.','_')
            branch = branch.replace('[','_')
            branch = branch.replace(']','_')
            call(["hg", "bookmark", branch+"_bookmark"])

Note that we take this opportunity to replace all the characters that are not allowed in git branch names (which have unfortunately crashed the migration on our first attempts).

Migrate hg into git with hg-git

This step is simple but is probably the most important one, you just need to position yourself in your local clone of your mercurial repository and do a push in the git one.

os.chdir(REPO_HG_DIR)
    call(["hg", "push", REPO_GIT_DIR])

Add .gitignore file in every branches

In order to avoid future merge problems, we decided to add the .gitignore file in all newly created branches.

call(["git", "config", "user.email", GIT_CONFIG_EMAIL])
call(["git", "config", "user.name", GIT_CONFIG_NAME])for branch in branches:
       
        call(["git", "checkout", branch])
        print("*** Adding GitIgnore File in "+branch+" ***")       if not os.path.isfile(REPO_GIT_DIR+"/.gitignore"):
            shutil.copy2(PATH_TO_GITIGNORE, REPO_GIT_DIR+"/.gitignore")
            call(["git", "add", ".gitignore"])
            call(["git", "commit", "-m 'Adding GitIgnore File'"])
            print("Git Ignore file added")
        else:
            print("Git Ignore found, step skipped for branch :"+branch)

Bitbucket commands

All other sections of the algorithm are simple calls to the REST bitbucket APIs. They are available on the Bitbucket website:
https://developer.atlassian.com/bitbucket/api/2/reference/resource/

In a nutshell

The effective migration of our +1300 repositories lasted a few days. In order to reduce outage we migrated the repositories in several phases.

Repositories that had not been touched for more than 6 months.
The ones that did not have modifications for more than 2 months
Finally the remaining repositories.

It was a really interesting experience and I learned a lot about python (never tried before). This kind of small projects is also interesting on the human part, because you have to synchronize and train over 100 developers.