Mar 29, 2022
Backup and restore Gitlab in docker
Gitlab hosts everything about the code including the docs and the pipeline data, etc. It’s crucial to back it up. You can also use restore to migrate the Gitlab to another server. This post will show you how to backup and restore the Gitlab-CE docker version.
Backup Gitlab in docker
Locate backup path
gitlab_rails[‘backup_path’] is commented in the Gitlab configuration file gitlab.rb, its value is the default backup path which is at /var/opt/gitlab/backups.
# From Gitlab dockerroot@gitlab:/etc/gitlab# cat /etc/gitlab/gitlab.rb | grep backup_path
# gitlab_rails['manage_backup_path'] = true
# gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
Create the backup
You don’t need to stop anything before creating the backup.
# From host outside of the Gitlab docker[root@gitlab ~]:~$ docker exec -it gitlab gitlab-rake gitlab:backup:create
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
* win/flaskapi ... [DONE]
* win/flaskapi.wiki ... [SKIPPED]
* xiang/flaskapi ... [DONE]
* xiang/flaskapi.wiki ... [SKIPPED]
done
Dumping uploads ...
done
Dumping builds ...
done
Dumping artifacts ...
done
Dumping pages ...
done
Dumping lfs objects ...
done
Dumping container registry images ...
[DISABLED]
Creating backup archive: 1537738648_2018_09_23_10.8.3_gitlab_backup.tar ... done
Uploading backup archive to remote storage ... skipped
Deleting tmp directories ... done
done
done
done
done
done
done
done
Deleting old backups ... skipping
[root@gitlab ~]:~$
The backup uses the Linux commands tar and gzip. This works fine in most cases, but can cause problems when data is rapidly changing. When data changes while tar is reading it, the error file changed as we read it may occur, and will cause the backup process to fail. In such case, you add the copy strategy to your backup command like docker exec -it gitlab gitlab-rake gitlab:backup:create STRATEGY=copy.
Verify the backup
# From Gitlab dockerroot@gitlab:/etc/gitlab# ls -lart /var/opt/gitlab/backups
total 644
drwxr-xr-x 19 root root 4096 Sep 22 23:52 ..
-rw------- 1 git git 215040 Sep 23 21:37 1537738648_2018_09_23_10.8.3_gitlab_backup.tar
-rw------- 1 git git 215040 Sep 23 21:38 1537738690_2018_09_23_10.8.3_gitlab_backup.tar
drwx------ 2 git root 4096 Sep 23 21:38 .
Backup configuration and secret files
Yes, the configuration and secret files are not backed up during the previous backup procedure. This is because the previous one encrypts some Gitlab data by using the secret key in the configuration and secret files. If you save them to the same place, you’re just defeating the encryption.
So please also backup /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json and save them to a secure place from other Gitlab backup data.
Upload backups to any remote storage
Restore Gitlab
You can only restore the Gitlab backup to the same Gitlab version and type. And you also need to have a working Gitlab instance.
Stop some Gitlab services
# From Gitlab dockergitlab-ctl reconfigure
gitlab-ctl start
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-ctl status
ls -lart /var/opt/gitlab/backups
Copy the backup files to the [‘backup_path’] in the docker container
# From host outside of the Gitlab docker[root@gitlab ~]# docker cp /home/Mohamed.Emam/MyBackup/1611340190_2021_01_22_13.5.1_gitlab_backup.tar gitlab:/var/opt/gitlab/backups/
Start the restore
The backup file must can be found in the backup path, which is defined in the configuration file /etc/gitlab/gitlab.rb by the key gitlab_rails[‘backup_path’].
# From host outside of the Gitlab docker[root@gitlab ~]:~$ docker exec -it gitlab gitlab-rake gitlab:backup:restore --trace
** Invoke gitlab:backup:restore (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:backup:restore
Unpacking backup ... done
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
(...)
COPY 0
setval
--------
1
(1 row)COPY 0
setval
--------
1
(1 row)
(...)
ALTER TABLE
ALTER TABLE
(...)
CREATE INDEX
(...)
ALTER TABLE
ALTER TABLE
(...)
WARNING: no privileges were granted for "public"
GRANT
[DONE]
done
** Invoke gitlab:backup:repo:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:repo:restore
Restoring repositories ...
* win/flaskapi ... [DONE]
* xiang/flaskapi ... [DONE]
Put GitLab hooks in repositories dirs [DONE]
done
** Invoke gitlab:backup:uploads:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:uploads:restore
Restoring uploads ...
done
** Invoke gitlab:backup:builds:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:builds:restore
Restoring builds ...
done
** Invoke gitlab:backup:artifacts:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:artifacts:restore
Restoring artifacts ...
done
** Invoke gitlab:backup:pages:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:pages:restore
Restoring pages ...
done
** Invoke gitlab:backup:lfs:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:lfs:restore
Restoring lfs objects ...
done
** Invoke gitlab:shell:setup (first_time)
** Invoke gitlab_environment
** Execute gitlab:shell:setup
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? yes** Invoke cache:clear (first_time)
** Invoke cache:clear:redis (first_time)
** Invoke environment
** Execute cache:clear:redis
** Execute cache:clear
Deleting tmp directories ... done
done
done
done
done
done
done
done
[root@gitlab ~]:~$
We can also add the param BACKUP to specify the backup file if there’s more than one backup tar file in the backup path. The value of the BACKUP is the backup file timestamp, for example : docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=1611340190_2021_01_22_13.5.1 — trace.
Restart Gitlab with a sanity check
Restart the Gitlab services by gitlab-ctl restart:
# From Gitlab dockerroot@gitlab:/# gitlab-ctl restart
ok: run: alertmanager: (pid 2789) 1s
ok: run: gitaly: (pid 2797) 0s
ok: run: gitlab-monitor: (pid 2806) 0s
ok: run: gitlab-workhorse: (pid 2811) 1s
ok: run: logrotate: (pid 2827) 0s
ok: run: nginx: (pid 2834) 1s
ok: run: node-exporter: (pid 2839) 0s
ok: run: postgres-exporter: (pid 2845) 1s
ok: run: postgresql: (pid 2855) 0s
ok: run: prometheus: (pid 2864) 0s
ok: run: redis: (pid 2873) 1s
ok: run: redis-exporter: (pid 2877) 0s
ok: run: sidekiq: (pid 2957) 0s
ok: run: sshd: (pid 2960) 0s
ok: run: unicorn: (pid 2968) 1s
Launch the Gitlab sanity check by gitlab-rake gitlab: check SANITIZE=true:
root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...GitLab Shell version >= 7.1.2 ? ... OK (7.1.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/2 ... ok
2/3 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: FAILED: Failed to connect to internal API
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.Checking GitLab Shell ... FinishedChecking Sidekiq ...Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.Checking Sidekiq ... FinishedReply by email is disabled in config/gitlab.yml
Checking LDAP ...LDAP is disabled in config/gitlab.ymlChecking LDAP ... FinishedChecking GitLab ...Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yesyes
Database contains orphaned GroupMembers? ... nono
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/2 ... yesyes
2/3 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.7)
Git version >= 2.9.5 ? ... yes (2.16.4)yes (2.3.7)
Git version >= 2.9.5 ? ... yes (2.16.4)
Git user has default SSH configuration? ... yes
Active users: ... 2Checking GitLab ... Finishedroot@gitlab:/#
Verify the Gitlab container health by docker ps:
# From Ubuntu host outside of the Gitlab dockerxiang@ubuntu1804:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
2ce7236d3756 gitlab/gitlab-ce:10.8.3-ce.0 "/assets/wrapper" 2 weeks ago Up 15 minutes (healthy)
0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
Automate the Backup Process
This is a Bash script to automate the Backup Process can be configured as a cron job
#/bin/sh#script to automate the gitlab backup and save it in folder /Mohamed.Emam/gitlab-backupdocker exec -it 2ce7236d3756 gitlab-rake gitlab:backup:createdocker cp gitlab:/var/opt/gitlab/backups/ /home/Mohamed.Emam/gitlab-backupdocker cp gitlab:/etc/gitlab/gitlab.rb /home/Mohamed.Emam/gitlab-backupdocker cp gitlab:/etc/gitlab/gitlab-secrets.json /home/Mohamed.Emam/gitlab-backup
Finally, I hope this tutorial be useful and for any questions feel free to comment, or you can contact me directly on Mohamed.ElEmam.Hussin@gmail.com.
