git cannot fetch: fatal: fetch-pack: unable to fork off sideband demultiplexer

Thank you for all the analysis in the above-mentioned GitHub issues. I've linked the "Ubuntu Docker Images" project, to make the corresponding team aware of this issue.

Thanks Lukas!

Just mind the fact that the issue might not be with the Docker image per se, or it might be a combination of multiple werid behaviours.

As per https://github.com/argoproj/argo-cd/issues/9888#issuecomment-1271726968, a weird thing I've noticed is that git is not linked anymore with libpthread starting from Ubuntu 22.04:

ArgoCD v2.11 uses ubuntu:21.04 as a base image
ArgoCD v2.12 uses ubuntu:22.04 as a base image

---

<email address hidden> $ git --version
15:08:35.903602 git.c:444 trace: built-in: git version
git version 2.30.2

<email address hidden> $ ldd $(which git)
 linux-vdso.so.1 (0x00007ffd4d59f000)
 libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f6918811000)
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f69187f5000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f69187d3000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f69185e7000)
 /lib64/ld-linux-x86-64.so.2 (0x00007f6918c27000)

<email address hidden> $ dpkg-query -l libc6 git gpg gpg-agent
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-=====================================================
ii git 1:2.30.2-1ubuntu1 amd64 fast, scalable, distributed revision control system
ii gpg 2.2.20-1ubuntu3 amd64 GNU Privacy Guard -- minimalist public key operations
ii gpg-agent 2.2.20-1ubuntu3 amd64 GNU privacy guard - cryptographic agent
ii libc6:amd64 2.33-0ubuntu5 amd64 GNU C Library: Shared libraries

---

<email address hidden> $ git --version
15:12:14.779580 git.c:455 trace: built-in: git version
git version 2.34.1

<email address hidden> $ ldd $(which git)
 linux-vdso.so.1 (0x00007ffeb5bae000)
 libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f2fbe15a000)
 libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2fbe13e000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2fbdf16000)
 /lib64/ld-linux-x86-64.so.2 (0x00007f2fbe598000)

<email address hidden> $ dpkg-query -l libc6 git gpg gpg-agent
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-===================-============-=====================================================
ii git 1:2.34.1-1ubuntu1.4 amd64 fast, scalable, distributed revision control system
ii gpg 2.2.27-3ubuntu2.1 amd64 GNU Privacy Guard -- minimalist public key operations
ii gpg-agent 2.2.27-3ubuntu2.1 amd64 GNU privacy guard - cryptographic agent
ii libc6:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: Shared libraries

It's probably not directly related to git itself, but rather the containerization.

I tried to reproduce the bug based on this comment (https://github.com/argoproj/argo-cd/issues/9888#issuecomment-1271819234) in a regular x86 host but everything worked as expected:

*21.04*:

```
root@6d56a04fa8c1:~# git clone <email address hidden>:lucaskanashiro/pcs.git
Cloning into 'pcs'...
Enter passphrase for key '/root/.ssh/id_rsa':
remote: Enumerating objects: 43789, done.
remote: Counting objects: 100% (1314/1314), done.
remote: Compressing objects: 100% (379/379), done.
remote: Total 43789 (delta 939), reused 1183 (delta 933), pack-reused 42475
Receiving objects: 100% (43789/43789), 20.58 MiB | 10.64 MiB/s, done.
Resolving deltas: 100% (35429/35429), done.
root@6d56a04fa8c1:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.04
DISTRIB_CODENAME=hirsute
DISTRIB_DESCRIPTION="Ubuntu 21.04"
root@6d56a04fa8c1:~# git --version
git version 2.30.2
```

*22.04*:

```
root@06bdbdc4c967:~# git clone <email address hidden>:lucaskanashiro/pcs.git
Cloning into 'pcs'...
Enter passphrase for key '/root/.ssh/id_rsa':
remote: Enumerating objects: 43789, done.
remote: Counting objects: 100% (1314/1314), done.
remote: Compressing objects: 100% (379/379), done.
remote: Total 43789 (delta 939), reused 1183 (delta 933), pack-reused 42475
Receiving objects: 100% (43789/43789), 20.58 MiB | 10.42 MiB/s, done.
Resolving deltas: 100% (35429/35429), done.
root@06bdbdc4c967:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
root@06bdbdc4c967:~# git --version
git version 2.34.1
```

So this seems to be related to the VMware Tanzu infrastructure indeed.

Thanks Lu[ck]as for the checks.
I've pinged VMware to hopefully reach someone with a Tanzu-responsibility-Hat to have a look at this.

Just for the record, I'm not a VMware expert myself and probably mistaken the correct name of the product.

It should be VMware TKGi (VMware Tanzu Kubernetes Grid Integrated Edition, formerly known as VMware Enterprise PKS).

We all seem to be able to reproduce the issue on this platform only, AWS and local machine are working fine.

See for example this [1] comment from a Swiss Post guy (they have a similar solution to ours, I guess?):

[1]: https://github.com/argoproj/argo-cd/issues/9809#issuecomment-1256354948

We are using:
TKGi version: 1.12.4-build.14

Nodes:
Version: v1.21.9+vmware.1
OS: Ubuntu 16.04.7 LTS
Kernel: 4.15.0-167-generic
Container Runtime: docker://20.10.9

Swiss Post reported the same issue with:

TKGi version: 1.13.4-build.15

Nodes:
Version: v1.22.6+vmware.1
OS: Ubuntu 16.04.7 LTS
Kernel: 4.15.0-176-generic
Container Runtime: docker://20.10.9