GEI Generate Script & Inventory Report Commands

Part 2 — Generate Script and Inventory Report

This #2 in a series of articles explaining how to use the GitHub Enterprise Importer (gei) tools to migrate repos from Azure DevOps to GitHub. If you have not already done so you should read this article first to get yourself set up to try the examples. GitHub Enterprise Importer (GEI)

Once you have everything installed, you’ll want to check out what the tool has to offer. Running the tool with help will of course show you the available commands.

gh ado2gh --help

In this article we’ll take a look at Generate Script and Inventory Report commands.

Generate Script

According to the help: “Generates a migration script. This provides you the ability to review the steps that this tool will take, and optionally modify the script if desired before running it. Note: Expects ADO_PAT env variable or — ado-pat option to be set.”

Run the generate script command with help to see its options.

gh ado2gh generate-script --help
You are running the latest version of the ado2gh CLI [v0.31]
Description:
  Generates a migration script. This provides you the ability to review the steps that this tool will take, and optionally modify the script if desired before running it.
  Note: Expects ADO_PAT env variable or --ado-pat option to be set.

Usage:
  gh-ado2gh generate-script [options]

Options:
  --github-org <github-org> (REQUIRED)
  --ado-org <ado-org>
  --ado-team-project <ado-team-project>
  --output <output>                      [default: ./migrate.ps1]
  --sequential                           Waits for each migration to finish before moving on to the next one.
  --ado-pat <ado-pat>
  --verbose
  --download-migration-logs              Downloads the migration log for each repository migration.
  --create-teams                         Includes create-team scripts that creates admins and maintainers teams and adds them to repos.
  --link-idp-groups                      Adds --idp-group to the end of create teams scripts that links the created team to an idP group.
  --lock-ado-repos                       Includes lock-ado-repo scripts that lock repos before migrating them.
  --disable-ado-repos                    Includes disable-ado-repo scripts that disable repos after migrating them.
  --integrate-boards                     Includes configure-autolink and integrate-boards scripts that configure Azure Boards integrations.
  --rewire-pipelines                     Includes share-service-connection and rewire-pipeline scripts that rewire Azure Pipelines to point to GitHub repos.
  --all                                  Includes all script generation options.
  --repo-list <repo-list>                Path to a csv file that contains a list of repos to generate a script for. The CSV file should be generated using the inventory-report command.
  -?, -h, --help                         Show help and usage information

A couple of things to understand about the script that this command generates.

1. There are 3 arguments that define the scope of what gets scripted.

gh ado2gh generate-script `
--github-org "MyShuttle" `      # The GitHub Org you're migrating to
--ado-org "dlloyd" `            # The Azure DevOps Org you're migrating from 
--ado-team-project "My Shuttle" # The AzDO Team Project you're migrating from
# Leaving out the --ado-team-project will generate a script for all the
# team projects under the azure devops org specified 

Try the above command with your orgs to see what you get. Remember all this does is generate a script, nothing to worry about at this point.

2. This command above will create a PowerShell script called migrate.ps1, unless you supply the output argument and a filename. Like this:

gh ado2gh generate-script `
--github-org "MyShuttle" `        
--ado-org "dlloyd" `          
--ado-team-project "My Shuttle" `
--output "./MyShuttle.ps1"

3. By default, the generated script (as seen below) will be written to run migrations asynchronously, meaning it will start each migration getting a migration id from each call. Then you can use the wait-for-migration command to know when the migration is complete so you can carry on with another part of your overall script.

The generated script would look something like this:

function Exec {
    param (
        [scriptblock]$ScriptBlock
    )
    & @ScriptBlock
    if ($lastexitcode -ne 0) {
        exit $lastexitcode
    }
}

function ExecAndGetMigrationID {
    param (
        [scriptblock]$ScriptBlock
    )
    $MigrationID = Exec $ScriptBlock | ForEach-Object {
        Write-Host $_
        $_
    } | Select-String -Pattern "\(ID: (.+)\)" | ForEach-Object { $_.matches.groups[1] }
    return $MigrationID
}

function ExecBatch {
    param (
        [scriptblock[]]$ScriptBlocks
    )
    $Global:LastBatchFailures = 0
    foreach ($ScriptBlock in $ScriptBlocks)
    {
        & @ScriptBlock
        if ($lastexitcode -ne 0) {
            $Global:LastBatchFailures++
        }
    }
}

$Succeeded = 0
$Failed = 0
$RepoMigrations = [ordered]@{}

# =========== Queueing migration for Organization: dlloyd ===========

# === Queueing repo migrations for Team Project: dlloyd/My Shuttle ===

$MigrationID = ExecAndGetMigrationID { gh ado2gh migrate-repo --ado-org "dlloyd" --ado-team-project "My Shuttle" --ado-repo "MyShuttle" --github-org "MyShuttle" --github-repo "My-Shuttle-MyShuttle" }
$RepoMigrations["dlloyd/My-Shuttle-MyShuttle"] = $MigrationID

# =========== Waiting for all migrations to finish for Organization: dlloyd ===========

# === Waiting for repo migration to finish for Team Project: My Shuttle and Repo: MyShuttle. Will then complete the below post migration steps. ===
$CanExecuteBatch = $true
if ($null -ne $RepoMigrations["dlloyd/My-Shuttle-MyShuttle"]) {
    gh ado2gh wait-for-migration --migration-id $RepoMigrations["dlloyd/My-Shuttle-MyShuttle"]
    $CanExecuteBatch = ($lastexitcode -eq 0)
}
if ($CanExecuteBatch) {
    $Succeeded++
} else {
    $Failed++
}

Write-Host =============== Summary ===============
Write-Host Total number of successful migrations: $Succeeded
Write-Host Total number of failed migrations: $Failed

if ($Failed -ne 0) {
    exit 1
}

Alternatively, the migrate-repo command can be written to wait for the migration to complete before running the next command by just adding a wait argument. More about that in another article.

However if you add the sequential argument to the generate script command, it will add the wait argument to the script it generates.

Like in this example.

gh ado2gh generate-script `
    --github-org "MyShuttle" `
    --ado-org "dlloyd" `
    --ado-team-project "My Shuttle" `
    --output "./MyShuttle.ps1"`
    --sequential

function Exec {
    param (
        [scriptblock]$ScriptBlock
    )
    & @ScriptBlock
    if ($lastexitcode -ne 0) {
        exit $lastexitcode
    }
}
# =========== Organization: dlloyd ===========

# === Team Project: dlloyd/My Shuttle ===

Exec { gh ado2gh migrate-repo --ado-org "dlloyd" --ado-team-project "My Shuttle" --ado-repo "MyShuttle" --github-org "MyShuttle" --github-repo "My-Shuttle-MyShuttle" --wait }

This is a much simpler script. And allows the ability to run other scripts that you may need for your repos once the migration is complete. This is my preference actually. I’ll talk more about that when I cover the migrate-repo command.

The greatest thing about generate-script is it gives you a great starting point.

Adding the -all argument will script out other things you can do, try it out and see what you get.

There is also an argument called repo-list that will read in a .csv file of Repos to generate a migration script for.

Inventory-Report

According to the help: “Generates several CSV files containing lists of ADO orgs, team projects, repos, and pipelines. Useful for planning large migrations. Note: Expects ADO_PAT env variable or — ado-pat option to be set.”

Inventory Report only needs one argument, your ADO Org and it will generate four .csv files that inventory what you have that can be migrated.

Very useful as input if you want to build out a script that iterates over all the team projects and all their repos and pipelines. Or even just to get a view of what you have to migrate.

Try this command. (Replacing it with your Azure DevOps Organization of course)

gh ado2gh inventory-report --ado-org "dlloyd" 

It should have generated four files with the following information.

• orgs.csv

Contents: Name, url, owner, teamproject-count, repo-count, pipeline-count, is-pat-org-admin, pr-count

• team-projects.csv

Contents: org, teamproject, url,repo-count, pipeline-count, pr-count

• repos.csv

Contents: org, teamproject, repo, url,last-push-date, pipeline-count, compressed-repo-size-in-bytes, most-active-contributor, pr-count, commits-past-year

• pipelines.csv

Contents: org, teamproject, repo, pipeline, url

My take on these 2 commands.

They are great to get an inventory of what you need to do and generate a great starting point towards your final migration scripts.

Very useful during the investigation stage of any migration.