Solve the git fetch origin hangs issue again

In my earlier article, I mentioned to set the timeout to solve the git fetch origin hang issue, the command I mentioned is:

GIT_SSH_COMMAND="ssh -o ConnectTimeout=10 -o ConnectionAttempts=1" git fetch origin

But on the build server, I find even use this command, the command still hangs, so I am thinking about another solution to solve the problem.

Solution

My solution is saving the pid and another information called phase on the pid file, the job not only check the process exists, but also check the phase, if the phase is 1, then it means the previous command encounter the hang issue again, so the second job will kill the process and continue running.

Solution description

Save the pid together with phase, such as when application start, I will save current pid and phase 0 on the pid file, the content will like below:

pid,0

And before I run command get fetch origin , I will update the phase to 1 on the pid file, so the content will be:

pid,1

After that, I will update the phase accordingly, so the content of the pid file will be:

pid,2
pid,3
...

When the next task start, I will check the old pid file, and also check the phase, if the phase is 1, it means that the command was hang, so I will kill the process by the pid, and run the second job, otherwise, the second job will return and wait next try.

After updated the build script, I hope that the issue should be fixed, I will continue monitoring the status of the build server.