Got some more pcaps. I can also reproduce by sending a 503 after a 200, without aborting during/before body transfer (*).
============================ server.py logs ============================ 13:29:55 [3666956] client said: b'GET /ubuntu/dists/jammy/InRelease HTTP/1.1\r\nHost: junk.devs.nu:3851\r\nCache-Control: max-age=0\r\nAccept: text/*\r\nUser-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive\r\n\r\n' 13:29:55 [3666956] we say: 200 and serve local InRelease 13:29:55 [3666956] client said: b'GET /ubuntu/dists/jammy-updates/InRelease HTTP/1.1\r\nHost: junk.devs.nu:3851\r\nCache-Control: max-age=0\r\nAccept: text/*\r\nUser-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive\r\n\r\n' 13:29:55 [3666956] we say: 503
============================ 503 request/response (valid length, no early FIN) ============================ GET /ubuntu/dists/jammy-updates/InRelease HTTP/1.1 Host: junk.devs.nu:3851 Cache-Control: max-age=0 Accept: text/* User-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive
HTTP/1.1 503 OK date: Fri, 17 Feb 2023 15:11:49 GMT content-type: text/plain content-length: 19
something is broken
============================
I present to you:
- https://junk.devs.nu/2023/lp2003851/not-reproduced-port-34634.txt - https://junk.devs.nu/2023/lp2003851/not-reproduced-port-34634.pcap - https://junk.devs.nu/2023/lp2003851/reproduced-port-58014.txt - https://junk.devs.nu/2023/lp2003851/reproduced-port-58014.pcap
In the not-reproduced case, the error is acknowledged and the download is retried. Then it fails quickly.
In the reproduced case, the error is not acknowledged, and the result is a hanging apt-get. Logs ending with:
Hit:3 http://repo.zabbix.com/zabbix/6.2/ubuntu jammy InRelease 0% [Working]
============================ (*) observed in the wild with these req/resp: ============================ GET /ubuntu/dists/jammy-backports/InRelease HTTP/1.1 Host: apt.osso.nl Cache-Control: max-age=0 Accept: text/* If-Modified-Since: Tue, 21 Feb 2023 19:32:00 GMT User-Agent: Debian APT-HTTP/1.3 (2.4.8) non-interactive
HTTP/1.1 200 OK date: Wed, 22 Feb 2023 02:36:08 GMT content-type: octet/stream content-length: 106807 last-modified: Wed, 22 Feb 2023 01:49:00 GMT x-original-source: http://nl.archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease (+ valid body)
GET /ubuntu-security/dists/jammy-security/InRelease HTTP/1.1 Host: apt.osso.nl Cache-Control: max-age=0 Accept: text/* If-Modified-Since: Tue, 21 Feb 2023 23:11:00 GMT User-Agent: Debian APT-HTTP/1.3 (2.4.8) non-interactive
HTTP/1.1 503 Connection timeout date: Wed, 22 Feb 2023 02:36:08 GMT content-type: text/html content-length: 486 (+ valid body)
============================ extra contemplations: ============================ We've only recently added the /ubuntu-security/ (extra path, on the same server). Maybe that is the reason why this is biting us right now.
Got some more pcaps. I can also reproduce by sending a 503 after a 200, without aborting during/before body transfer (*).
======= ======= ======= ======= ======= ======= ======= dists/jammy/ InRelease HTTP/1.1\r\nHost: junk.devs. nu:3851\ r\nCache- Control: max-age= 0\r\nAccept: text/*\ r\nUser- Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive \r\n\r\ n' dists/jammy- updates/ InRelease HTTP/1.1\r\nHost: junk.devs. nu:3851\ r\nCache- Control: max-age= 0\r\nAccept: text/*\ r\nUser- Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive \r\n\r\ n'
server.py logs
=======
13:29:55 [3666956] client said: b'GET /ubuntu/
13:29:55 [3666956] we say: 200 and serve local InRelease
13:29:55 [3666956] client said: b'GET /ubuntu/
13:29:55 [3666956] we say: 503
======= ======= ======= ======= ======= ======= ======= dists/jammy- updates/ InRelease HTTP/1.1
503 request/response (valid length, no early FIN)
=======
GET /ubuntu/
Host: junk.devs.nu:3851
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive
HTTP/1.1 503 OK
date: Fri, 17 Feb 2023 15:11:49 GMT
content-type: text/plain
content-length: 19
something is broken
======= ======= ======= =======
I present to you:
- https:/ /junk.devs. nu/2023/ lp2003851/ not-reproduced- port-34634. txt /junk.devs. nu/2023/ lp2003851/ not-reproduced- port-34634. pcap /junk.devs. nu/2023/ lp2003851/ reproduced- port-58014. txt /junk.devs. nu/2023/ lp2003851/ reproduced- port-58014. pcap
- https:/
- https:/
- https:/
In the not-reproduced case, the error is acknowledged and the download is retried. Then it fails quickly.
In the reproduced case, the error is not acknowledged, and the result is a hanging apt-get. Logs ending with:
Hit:3 http:// repo.zabbix. com/zabbix/ 6.2/ubuntu jammy InRelease
0% [Working]
======= ======= ======= ======= ======= ======= ======= dists/jammy- backports/ InRelease HTTP/1.1
(*) observed in the wild with these req/resp:
=======
GET /ubuntu/
Host: apt.osso.nl
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Tue, 21 Feb 2023 19:32:00 GMT
User-Agent: Debian APT-HTTP/1.3 (2.4.8) non-interactive
HTTP/1.1 200 OK nl.archive. ubuntu. com/ubuntu/ dists/jammy- backports/ InRelease
date: Wed, 22 Feb 2023 02:36:08 GMT
content-type: octet/stream
content-length: 106807
last-modified: Wed, 22 Feb 2023 01:49:00 GMT
x-original-source: http://
(+ valid body)
GET /ubuntu- security/ dists/jammy- security/ InRelease HTTP/1.1
Host: apt.osso.nl
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Tue, 21 Feb 2023 23:11:00 GMT
User-Agent: Debian APT-HTTP/1.3 (2.4.8) non-interactive
HTTP/1.1 503 Connection timeout
date: Wed, 22 Feb 2023 02:36:08 GMT
content-type: text/html
content-length: 486
(+ valid body)
======= ======= ======= ======= ======= ======= =======
extra contemplations:
=======
We've only recently added the /ubuntu-security/ (extra path, on the same server). Maybe that is the reason why this is biting us right now.