Deepak Kumar Mishra, Support Engineer, EMEA TFS Team, brings this knowledge experience while working with one of his customers and this important piece of information might help a lot of users who are using Git repos in tfs and face issue while performing a push operation.
Issue Description:
A Customer is trying to push an 18GB Git repository in TFS 2015 Update 3. It worked locally to a local server - but we cannot get it to push to our global TFS server. While pushing the repository via SSH, we get the following errors in a Linux machine:
[xxxx@servername]$ grep tfs .git/config
url = ssh://tfs. servername:22/tfs/collection/teamproject/_git/repo
[xxxx@servername]$ git push
Counting objects: 95405, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (24474/24474), done.
Writing objects: 100% (95405/95405), 18.15 GiB | 8.34 MiB/s, done.
Total 95405 (delta 68425), reused 95401 (delta 68421)
remote: Analyzing objects... (95405/95405) (3030021 ms)
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Cause:
This is a bug in TFS related to SSh which will be fixed in upcoming release.
Our SSH framework times out the connection if there is no communication for over 5 minutes. If pushing a massive repo, the server could do over 5 minutes of work ‘silently’, leading to a connection drop.
The workaround is to make the initial push over https and then switch to SSH.
Troubleshooting and Resolution:
1.The size of the repository was big and so we tried to reduce the size first by using the following commands.
git gc --prune=now --aggressive
git repack -adf
then push
time git gc --prune=now --aggressive; time git repack -adf ; time git push
The command failed with the following errors:
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
2. Then we tried to do a push with Http first and decided to do the next clone and push via SSH. This gave the following error regarding certificate as HTTPS was configured in this case.
fatal: unable to access 'https://tfs.servername/tfs/collection/teamproject/_git/repo/ ': Peer certificate cannot be authenticated with known CA certificates
3. So we informed customer that they were using using a self-signed certificate to allow HTTPS in TFS on-premises They need to permanently add the HTTPS certificate as “trusted” on the machine and ignore the self-sign check for git by using the flag git -c http.sslVerify=false
4. Then We checked and found that the push operation worked as expected by using Https. [[xxxx@servername]$ git -c http.sslVerify=false push Counting objects: 95405, done. Delta compression using up to 4 threads. Compressing objects: 100% (24499/24499), done. Writing objects: 100% (95405/95405), 18.54 GiB | 4.79 MiB/s, done. Total 95405 (delta 68399), reused 95401 (delta 68395) remote: Analyzing objects... (95405/95405) (3894107 ms) remote: Storing packfile... done (2695635 ms) remote: Storing index... done (936 ms) To 'https://tfs.servername/tfs/collection/teamproject/_git/repo/ * [new branch] master -> master
5. Finally we were able to do the next clone and then push by using SSH
Written by: Deepak Kumar Mishra, Support Engineer
Reviewed by: Nitish Nagpal, Support Escalation Engineer