Mastering Git submodules

Are they the right tool for the job?

Submodules or subtrees?

Check out our amazing Git-related video courses! Some of them are even free!

Submodule fundamentals

A plethora of traps

iOS sometimes cunningly autocorrects submodules as “sobmodules.” Someone got bitten, it seems…

The dangers we face

  • Every time you add a submodule, change its remote’s URL, or change the referenced commit for it, you demand a manual update by every collaborator.
  • Forgetting this explicit update can result in silent regressions of the submodule’s referenced commit.
  • Commands such as status and diff display precious little info about submodules by default.
  • Because lifecycles are separate, updating a submodule inside its container project requires two commits and two pushes.
  • Submodule heads are generally detached, so any local update requires various preparatory actions to avoid creating a lost commit.
  • Removing a submodule requires several commands and tweaks, some of which are manual and unassisted.

Submodules, step by step

  • main acts as the container repo, local to the first collaborator,
  • plugin acts as the central maintenance repo for the module, and
  • remotes contains the filesystem remotes for the two previous repos.

Adding a submodule

.
├── README.md
├── lib
│ └── index.js
└── plugin-config.json
main (master u=) $ git submodule add ../plugin vendor/plugins/demo
Cloning into 'vendor/plugins/demo'…
done.
main (master + u=) $
main (master + u=) $ cat .git/config

[submodule "vendor/plugins/demo"]
url = ../remotes/plugin
main (master + u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: .gitmodules
new file: vendor/plugins/demo
main (master + u=) $ cat .gitmodules
[submodule "vendor/plugins/demo"]
path = vendor/plugins/demo
url = ../plugin

└── vendor
└── plugins
└── demo
├── .git
├── README.md
├── lib
│ └── index.js
└── plugin-config.json
git config --global status.submoduleSummary true
main (master + u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: .gitmodules
new file: vendor/plugins/demo
Submodule changes to be committed:* vendor/plugins/demo 0000000...fe64799 (3):
> Fix repo name for main project companion demo repo
main (master + u=) $ cd vendor/plugins/demo
demo (master u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
demo (master u=) $ cat .git
gitdir: ../../../.git/modules/vendor/plugins/demo
demo (master u=) $ cd -
main (master + u=) $ git commit -m "Ajout submodule plugin demo"
main (master u+1) $ git push

Grabbing a repo that uses submodules

main (master u=) $ cd ..
git-subs $ git clone remotes/main colleague
Cloning into 'colleague'...
done.
git-subs $ cd colleague
colleague (master u=) $
vendor
└── plugins
└── demo
colleague (master u=) $ git submodule init
Submodule 'vendor/plugins/demo' (/tmp/git-subs/remotes/plugin) registered for path 'vendor/plugins/demo'
colleague (master u=) $ git submodule update
Cloning into 'vendor/plugins/demo'...
done.
Submodule path 'vendor/plugins/demo': checked out 'fe6479991d214f4d95ac2ae959d7252a866e01a3'
colleague (master u=) $ git submodule update --init
colleague (master u=) $ cd -
git-subs $ rm -fr colleague
git-subs $ git clone --recursive remotes/main colleague
Cloning into 'colleague'...
done.
Submodule 'vendor/plugins/demo' (/tmp/git-subs/remotes/plugin) registered for path 'vendor/plugins/demo'
Cloning into 'vendor/plugins/demo'...
done.
Submodule path 'vendor/plugins/demo': checked out 'fe6479991d214f4d95ac2ae959d7252a866e01a3'
git-subs $ cd colleague/vendor/plugins/demo
demo ((master)) $
demo ((fe64799...)) $
demo ((master)) $ git status
HEAD detached at fe64799
nothing to commit, working directory clean

Getting an update from the submodule’s remote

colleague (master u=) $ cd ../plugin
plugin (master u=) $ git log --oneline
fe64799 Fix repo name for main project companion demo repo
89d24ad Main files (incl. subdir) for plugin, to populate its tree.
cc88751 Initial commit
plugin (master u=) $ date > fake-work
plugin (master % u=) $ git add fake-work
plugin (master + u=) $ git commit -m "Pseudo-commit #1"
[master e6f5bb6] Pseudo-commit #1
1 file changed, 1 insertion(+)
create mode 100644 fake-work
plugin (master u+1) $ date >> fake-work
plugin (master * u+1) $ git commit -am "Pseudo-commit #2"

plugin (master u+2) $ git push
plugin (master u=) $ cd ../main
main (master u=) $
main (master u=) $ cd vendor/plugins/demo
demo (master u=) $ git fetch
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 6 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From /private/tmp/git-subs/main/../remotes/plugin
fe64799..0e90143 master -> origin/master
demo (master u-2) $ git log --oneline origin/master -10
0e90143 Pseudo-commit #2
e6f5bb6 Pseudo-commit #1
fe64799 Fix repo name for main project companion demo repo
89d24ad Main files (incl. subdir) for plugin, to populate its tree.
cc88751 Initial commit
demo (master u-2) $ git checkout -q 0e90143
demo ((remotes/origin/HEAD)) $ cd -
main (master * u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: vendor/plugins/demo (new commits)Submodules changed but not updated:* vendor/plugins/demo fe64799...0e90143 (2):
> Pseudo-commit #2
> Pseudo-commit #1
no changes added to commit (use "git add" and/or "git commit -a")
main (master * u=) $ git diff
diff --git i/vendor/plugins/demo w/vendor/plugins/demo
index fe64799..0e90143 160000
--- i/vendor/plugins/demo
+++ w/vendor/plugins/demo
@@ -1 +1 @@
-Subproject commit fe6479991d214f4d95ac2ae959d7252a866e01a3 +Subproject commit 0e9014309fe6c663e806c9f91297a592ee04cb6c
main (master * u=) $ git diff --submodule=log
Submodule vendor/plugins/demo fe64799..0e90143:
> Pseudo-commit #2
> Pseudo-commit #1
git config --global diff.submodule logmain (master * u=) $ git diff
Submodule vendor/plugins/demo fe64799..0e90143:
> Pseudo-commit #2
> Pseudo-commit #1
main (master * u=) $ git commit -am "Setting submodule on PC2"
main (master u+1) $ git push

Pulling a submodule-using repo

colleague (master u=) $ git pull
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From /tmp/git-subs/remotes/main
c995ed0..ac96c22 master -> origin/master
Fetching submodule vendor/plugins/demo
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 6 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From /tmp/git-subs/remotes/plugin
fe64799..0e90143 master -> origin/master
Successfully rebased and updated refs/heads/master.
colleague (master * u=) $
colleague (master * u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: vendor/plugins/demo (new commits)Submodules changed but not updated:* vendor/plugins/demo 0e90143...fe64799 (2):
< Pseudo-commit #2
< Pseudo-commit #1
no changes added to commit (use "git add" and/or "git commit -a")
colleague (master * u=) $ git submodule update
Submodule path 'vendor/plugins/demo': checked out '0e9014309fe6c663e806c9f91297a592ee04cb6c'
git config --global alias.spull '!git pull && git submodule sync --recursive && git submodule update --init --recursive'
git config --global alias.spull '__git_spull() { git pull "$@" && git submodule sync --recursive && git submodule update --init --recursive; }; __git_spull'
#! /bin/bash
git pull "$@" &&
git submodule sync --recursive &&
git submodule update --init --recursive
chmod +x git-spull

Updating a submodule in-place in the container

colleague (master u=) $ cd vendor/plugins/demo
demo ((remotes/origin/HEAD)) $ git checkout master
Previous HEAD position was 0e90143... Pseudo-commit #2
Switched to branch 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
demo (master u-2) $ git pull --rebase
First, rewinding head to replay your work on top of it...
Fast-forwarded master to 0e9014309fe6c663e806c9f91297a592ee04cb6c.
demo (master u=) $
colleague (master u=) $ git submodule update --remote --rebase -- vendor/plugins/demo
colleague (master u=) $ cd vendor/plugins/demo
demo (master u=) $
demo (master u=) $ date >> fake-work
demo (master * u=) $ git commit -am "Pseudo-commit #3"
[master 12e3a52] Pseudo-commit #3
1 file changed, 1 insertion(+)
demo (master u+1) $ cd ../../..
colleague (master * u=) git commit -am "Using PC3 on the submodule"
[master ad9da82] Using PC3 on the submodule
1 file changed, 1 insertion(+), 1 deletion(-)
colleague (master u+1) $
colleague (master u+1) $ git push
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 355 bytes | 0 bytes/s, done.
Total 4 (delta 1), reused 0 (delta 0)
To /tmp/git-subs/remotes/main
766cd47..ad9da82 master -> master
colleague (master u=) $ cd ../main
main (master u=) $ git pull
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From ../remotes/main
766cd47..ad9da82 master -> origin/master
Fetching submodule vendor/plugins/demo
Successfully rebased and updated refs/heads/master.
main (master * u=) $
main (master * u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: vendor/plugins/demo (new commits)Submodules changed but not updated:* vendor/plugins/demo 12e3a52...0e90143:
Warn: vendor/plugins/demo doesn't contain commit 12e3a529698c519b2fab790630f71bd531c45727
no changes added to commit (use "git add" and/or "git commit -a")
main (master * u=) $ git submodule update
fatal: reference is not a tree: 12e3a529698c519b2fab790630f71bd531c45727
Unable to checkout '12e3a529698c519b2fab790630f71bd531c45727' in submodule path 'vendor/plugins/demo'
main (master * u=) $ cd ../colleague/vendor/plugins/demo
demo (master u+1) $ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 329 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To /tmp/git-subs/remotes/plugin
0e90143..12e3a52 master -> master
demo (master u=) $ cd -
main (master * u=) $ git submodule update
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /private/tmp/git-subs/main/../remotes/plugin
0e90143..12e3a52 master -> origin/master
Submodule path 'vendor/plugins/demo': checked out '12e3a529698c519b2fab790630f71bd531c45727'
git config --global alias.spush 'push --recurse-submodules=on-demand'

Removing a submodule

  • You just want to clear the working directory (perhaps before archiving the container’s WD) but want to retain the possibility of restoring it later (so it has to remain in .gitmodules and .git/modules);
  • You wish to definitively remove it from the current branch.

Temporarily removing a submodule

main (master u=) $ git submodule deinit vendor/plugins/demo
Cleared directory 'vendor/plugins/demo'
Submodule 'vendor/plugins/demo' (../remotes/plugin) unregistered for path 'vendor/plugins/demo'
main (master u=) $
main (master u=) $ git submodule update --init
Submodule 'vendor/plugins/demo' (../remotes/plugin) registered for path 'vendor/plugins/demo'
Submodule path 'vendor/plugins/demo': checked out '12e3a529698c519b2fab790630f71bd531c45727'
main (master u=) $

Permanently removing a submodule

main (master u=) $ git rm vendor/plugins/demo
rm 'vendor/plugins/demo'
main (master + u=) $ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
modified: .gitmodules
deleted: vendor/plugins/demo
fatal: Not a git repository: 'vendor/plugins/demo/.git'
Submodule changes to be committed:
* vendor/plugins/demo 12e3a52...0000000:
main (master + u=) $ git ci -m "Removed the demo submodule"
[master 31cb27d] Removed the demo submodule
2 files changed, 4 deletions(-)
delete mode 160000 vendor/plugins/demo
main (master u+1) $ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
nothing to commit, working directory clean
git submodule deinit path/to/module # ensure local config cleanup
git rm path/to/module # clean WD and .gitmodules

Best practice recap (TL;DR)

Configuration settings

  • diff.submodule = log (so you get clearer container diffs when referenced submodule commits changed).
  • fetch.recurseSubmodules = on-demand (so you are confident new referenced commits for known submodules get fetched with container updates).
  • status.submoduleSummary = true (so git status gets useful again when a referenced submodule commit changed).

Adding or cloning

  • Initial add: git submodule add <url> <path>
  • Initial container clone: git clone — recursive <url> [<path>]

Grabbing updates inside a submodule

  1. cd path/to/module
  2. git fetch
  3. git checkout -q <commit-sha1>
  4. cd -
  5. git commit -am “Updated submodule X to: blah blah”

Grabbing container updates

  1. git pull
  2. git submodule sync — recursive
  3. git submodule update — init — recursive

Updating a submodule inside container code

  1. git submodule update — remote — rebase — path/to/module
  2. cd path/to/module
  3. Local work, testing, eventually staging
  4. git commit -am “Update to central submodule: blah blah”
  5. git push
  6. cd -
  7. git commit -am “Updated submodule X to: blah blah”

Permanently removing a submodule (1.7.8+)

  1. git submodule deinit path/to/module
  2. git rm path/to/module
  3. git commit -am “Removed submodule X”

Leftovers

Git commands

  • git submodule foreach lets you run arbitrary commands on all known (initialized) submodules, recursively or not; commands have access to various environment variables that state the submodule’s path, its referenced commit and the container working directory’s root path. Useful for custom scripting.
  • git submodule status is a specific status display for submodules, recursive on request. It tells us what the referenced commits are, whether working directories stray from that, whether submodules are initialized yet or not, and even merge conflicts, if any. Faster than manually checking through your working directories.
  • git submodule summary lists history ranges between the latest referenced commits and the ones currently checked out. This is what git status and git log display when submodule logs are enabled.
  • git mv on a 1.7.8+ submodule directory (one with a gitfile) does the right thing: it changes the relative path inside the gitfile, updates the core.worktree reference in the submodule’s repo inside .git/modules, and updates and stages .gitmodules.

CLI options

  • git diff — ignore-submodules, just like git status — ignore-submodules, remove any submodule-related information. Country-productive IMHO.

Configuration settings

  • diff.ignoreSubmodules permanently zaps out submodule info from diffs. A rather bad idea, if you ask me.

Want to learn more?

--

--

--

I make cool stuff and teach others to (Git, Rails, JS/CoffeeScript/Node/etc).

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Christophe Porteneuve

Christophe Porteneuve

I make cool stuff and teach others to (Git, Rails, JS/CoffeeScript/Node/etc).

More from Medium

What is Git?

An overview of Git, including what it is and how to use it.

GIT Branching

Multiple git configs (profiles) on one computer