Bug #1863232 “daemon rotates socket on restart” : Bugs : mod-wsgi package : Ubuntu

Revision history for this message

Dariusz Gadomski (dgadomski) wrote on 2020-02-14:

#1

Since it is already present upstream it's fixed for focal and eoan.

Changed in mod-wsgi (Ubuntu):
status:	New → Fix Released
Changed in mod-wsgi (Ubuntu Bionic):
assignee:	nobody → Dariusz Gadomski (dgadomski)
status:	New → In Progress
Changed in mod-wsgi (Ubuntu Xenial):
assignee:	nobody → Dariusz Gadomski (dgadomski)
status:	New → In Progress

Dariusz Gadomski (dgadomski) on 2020-02-14

tags:

added: sts

Revision history for this message

Dariusz Gadomski (dgadomski) wrote on 2020-02-14:

#2

xenial_mod-wsgi_4.3.0-1.1ubuntu1.debdiff Edit (6.5 KiB, text/plain)

SRU proposal for Xenial.

description:

updated

Revision history for this message

Dariusz Gadomski (dgadomski) wrote on 2020-02-14:

#3

bionic_mod-wsgi_4.5.17-1ubuntu1.debdiff Edit (6.3 KiB, text/plain)

SRU proposal for bionic.

Mauricio Faria de Oliveira (mfo) on 2020-02-14

tags:

added: sts-sponsor-mfo

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#4

Hi Dariusz,

I reviewed the debdiffs, they look good overall.

I just had some minor changes to reflect review
practices/comments that I have been thru myself:

- Nice catch on updating the Maintainers field.

- Changelog order of patch file/description:
I changed to file first, description later,
as I've seen as more used/standard practice.

- DEP3 headers are present in the .patch; good.

I updated from 'Origin: upstream' to 'backport'
because the patch has a removal of (unneeded)
release notes file.

Even being uneeded, there were changes to the
upstream patch, so it's no longer clean apply/
cherry pick, thus the change to 'backport',
per Debian DEP-3 spec [1]:

"backport" (in the case of an upstream patch that had to be modified to apply on the current version)

I added a '[backport]' section to the .patch
file describing that. (and this other change:)

I also noted that on Xenial there are changes
actually needed to src/server/__init__.py, so
arguably indeed a "true" backport this time.

Oh, and there's a digit missing in Bug-Ubuntu
number, just added that/goes to right URL now.

- Xenial fix on .patch file:

Still on Xenial there's an extra '+'/plus sign
in the commented line added to __init__.py, so
I fixed that one:

+++#WSGISocketRotation Off

Bionic had that right:

++#WSGISocketRotation Off

- Version numbers look good/upgrade path is OK:

- - Versions 4.3.0-1.1ubuntu1 and 4.5.17-1ubuntu1
never existed in package's publishing history [2].

- - Upgrade path is OK in the same release.

$ grep -m2 urgency= x/dput/lp1863232_xenial_mod-wsgi.debdiff
+mod-wsgi (4.3.0-1.1ubuntu1) xenial; urgency=medium
mod-wsgi (4.3.0-1.1build1) xenial; urgency=medium

$ dpkg --compare-versions 4.3.0-1.1build1 lt 4.3.0-1.1ubuntu1 ; echo $?
0

$ grep -m2 urgency= b/dput/lp1863232_bionic_mod-wsgi.debdiff
+mod-wsgi (4.5.17-1ubuntu1) bionic; urgency=medium
mod-wsgi (4.5.17-1) unstable; urgency=medium

$ dpkg --compare-versions 4.5.17-1 lt 4.5.17-1ubuntu1; echo $?
0

- - Upgrade path is OK across releases (t/x/b/e)

$ dpkg --compare-versions 3.4-4ubuntu2.1.14.04.2 lt 4.3.0-1.1ubuntu1 ; echo $?
0

$ dpkg --compare-versions 4.3.0-1.1ubuntu1 lt 4.5.17-1ubuntu1; echo $?
0

$ dpkg --compare-versions 4.5.17-1ubuntu1 lt 4.6.5-1 ; echo $?
0

That's it!

cheers,
Mauricio

[1] https://dep-team.pages.debian.net/deps/dep3/
[2] https://launchpad.net/ubuntu/+source/mod-wsgi/+publishinghistory

Hi Dariusz,

I reviewed the debdiffs, they look good overall.

I just had some minor changes to reflect review
practices/comments that I have been thru myself:

- Nice catch on updating the Maintainers field.

- Changelog order of patch file/description:
  I changed to file first, description later,
  as I've seen as more used/standard practice.

- DEP3 headers are present in the .patch; good.

I updated from 'Origin: upstream' to 'backport'
because the patch has a removal of (unneeded)
release notes file.

Even being uneeded, there were changes to the
upstream patch, so it's no longer clean apply/
cherry pick, thus the change to 'backport',
per Debian DEP-3 spec [1]:

"backport" (in the case of an upstream patch that had to be modified to apply on the current version)

I added a '[backport]' section to the .patch
file describing that. (and this other change:)

I also noted that on Xenial there are changes
actually needed to src/server/__init__.py, so
arguably indeed a "true" backport this time.

Oh, and there's a digit missing in Bug-Ubuntu
number, just added that/goes to right URL now.

- Xenial fix on .patch file:

Still on Xenial there's an extra '+'/plus sign
in the commented line added to __init__.py, so
I fixed that one:

+++#WSGISocketRotation Off

Bionic had that right:

++#WSGISocketRotation Off

- Version numbers look good/upgrade path is OK:

- - Versions 4.3.0-1.1ubuntu1 and 4.5.17-1ubuntu1
    never existed in package's publishing history [2].

- - Upgrade path is OK in the same release.

$ grep -m2 urgency= x/dput/lp1863232_xenial_mod-wsgi.debdiff 
+mod-wsgi (4.3.0-1.1ubuntu1) xenial; urgency=medium
 mod-wsgi (4.3.0-1.1build1) xenial; urgency=medium

$ dpkg --compare-versions 4.3.0-1.1build1 lt 4.3.0-1.1ubuntu1 ; echo $?
0

$ grep -m2 urgency= b/dput/lp1863232_bionic_mod-wsgi.debdiff 
+mod-wsgi (4.5.17-1ubuntu1) bionic; urgency=medium
 mod-wsgi (4.5.17-1) unstable; urgency=medium

$ dpkg --compare-versions 4.5.17-1 lt 4.5.17-1ubuntu1; echo $?
0

- - Upgrade path is OK across releases (t/x/b/e)

$ dpkg --compare-versions 3.4-4ubuntu2.1.14.04.2 lt 4.3.0-1.1ubuntu1 ; echo $?
0

$ dpkg --compare-versions 4.3.0-1.1ubuntu1 lt 4.5.17-1ubuntu1; echo $?
0

$ dpkg --compare-versions 4.5.17-1ubuntu1 lt 4.6.5-1 ; echo $?
0

That's it!

cheers,
Mauricio

[1] https://dep-team.pages.debian.net/deps/dep3/
[2] https://launchpad.net/ubuntu/+source/mod-wsgi/+publishinghistory

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#5

Attaching the updated debdiffs for reference.

They build successfully on a PPA for all architectures,
and have been used on steps to reproduce/verify (below.)

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#6

lp1863232_bionic_mod-wsgi.debdiff Edit (6.5 KiB, text/plain)

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#7

lp1863232_xenial_mod-wsgi.debdiff Edit (6.8 KiB, text/plain)

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#8

Dariusz,

As part of my learning homework to be able to
technically review the patch in this proposal,

I came across three things that may be useful
for the bug description/SRU template, so I'll
just add them, if you don't mind.

1) Understanding the scenario where the bug
happens (mod-wsgi running in daemon mode/not
embedded mode, only the former uses sockets;
and reload/not-restart apache2 between HTTP
requests in the same keep-alive connection)
[1, 2].

2) Detailed steps to reproduce/verify the bug.

3) Confirming that the default behavior with
the patch remains as in the previous version
(of course, as it should/is supposed to, but
it's worth mentioning in bug description :-)

With all those points now understood/verified,
I'll proceed with the upload for SRU to B/X.

Thanks!

cheers,
Mauricio

[1] https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
[2] https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#9

Download full text (10.2 KiB)

Steps to reproduce/verify on Xenial:
===

References:
- https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
- https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process
- https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
- https://www.howtoforge.com/tutorial/how-to-run-python-scripts-with-apache-and-mod_wsgi-on-ubuntu-18-04/

Create a container with apache2/mod-wsgi in daemon mode,
w/ keep-alive timeout long enough, and wsgi hello-world:
---

$ lxc launch ubuntu:xenial lp1863232x
$ lxc exec lp1863232x -- su - ubuntu

$ sudo apt update
$ sudo apt install -y apache2 libapache2-mod-wsgi

$ sudo sed -i '/^KeepAliveTimeout/ s/ .*/ 15/' /etc/apache2/apache2.conf

$ cat <<EOF | sudo tee /etc/apache2/conf-available/wsgi.conf
WSGIScriptAlias /hello-world /var/www/html/hello-world.py
WSGIDaemonProcess 127.0.0.1 processes=2 threads=16 display-name=%{GROUP}
WSGIProcessGroup 127.0.0.1
EOF

$ sudo a2enconf wsgi
$ sudo systemctl reload apache2

$ cat <<EOF | sudo tee /var/www/html/hello-world.py
def application(environ, start_response):
status = '200 OK'
output = b'Hello World!\n'

response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]

start_response(status, response_headers)
return [output]
EOF

$ curl 127.0.0.1/hello-world
Hello World!

Check WSGI socket filename changes when
reloading apache2 (aka graceful restart):
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.0.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.1.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.2.1.sock

Create script and shell one-liner
to send two HTTP requests in the
same connection (keep-alive used)
---

$ cat <<EOF >http-request
GET /hello-world HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive

EOF

One connection/One request:

$ (cat http-request; sleep 1) | telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Mon, 17 Feb 2020 23:33:19 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 13
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain

Hello World!
Connection closed by foreign host.

One connection/Two requests (added timestamps to check timeout values)

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:34:42 == Trying 127.0.0.1...
23:34:42 == Connected to 127.0.0.1.
23:34:42 == Escape character is '^]'.
23:34:42 == HTTP/1.1 200 OK
23:34:42 == Date: Mon, 17 Feb 2020 23:34:42 GMT
23:34:42 == Server: Apache/2.4.18 (Ubuntu)
23:34:42 == Content-Length: 13
23:34:42 == Keep-Alive: timeout=5, max=100
23:34:42 == Connection: Keep-Alive
23:34:42 == Content-Type: text/plain
23:34:42 ==
23:34:42 == Hello World!
23:34:43 == HTTP/1.1 200 OK
23:34:43 == Date: Mon, 17 Feb 2020 23:34:43 GMT
23:34:43 == Server: Apache/2.4.18 (Ubuntu)...

Steps to reproduce/verify on Xenial:
===

References:
- https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
- https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process
- https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
- https://www.howtoforge.com/tutorial/how-to-run-python-scripts-with-apache-and-mod_wsgi-on-ubuntu-18-04/

Create a container with apache2/mod-wsgi in daemon mode,
w/ keep-alive timeout long enough, and wsgi hello-world:
---

$ lxc launch ubuntu:xenial lp1863232x
$ lxc exec lp1863232x -- su - ubuntu

$ sudo apt update
$ sudo apt install -y apache2 libapache2-mod-wsgi

$ sudo sed -i '/^KeepAliveTimeout/ s/ .*/ 15/' /etc/apache2/apache2.conf

$ cat <<EOF | sudo tee /etc/apache2/conf-available/wsgi.conf
WSGIScriptAlias /hello-world /var/www/html/hello-world.py
WSGIDaemonProcess 127.0.0.1 processes=2 threads=16 display-name=%{GROUP}
WSGIProcessGroup 127.0.0.1
EOF

$ sudo a2enconf wsgi
$ sudo systemctl reload apache2

$ cat <<EOF | sudo tee /var/www/html/hello-world.py
def application(environ, start_response):
    status = '200 OK'
    output = b'Hello World!\n'

response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]

start_response(status, response_headers)
    return [output]
EOF

$ curl 127.0.0.1/hello-world
Hello World!

Check WSGI socket filename changes when
reloading apache2 (aka graceful restart):
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.0.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.1.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.2.1.sock

Create script and shell one-liner
to send two HTTP requests in the
same connection (keep-alive used)
---

$ cat <<EOF >http-request
GET /hello-world HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive

EOF

One connection/One request:

$ (cat http-request; sleep 1) | telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Mon, 17 Feb 2020 23:33:19 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 13
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain

Hello World!
Connection closed by foreign host.

One connection/Two requests (added timestamps to check timeout values)

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:34:42 == Trying 127.0.0.1...
23:34:42 == Connected to 127.0.0.1.
23:34:42 == Escape character is '^]'.
23:34:42 == HTTP/1.1 200 OK
23:34:42 == Date: Mon, 17 Feb 2020 23:34:42 GMT
23:34:42 == Server: Apache/2.4.18 (Ubuntu)
23:34:42 == Content-Length: 13
23:34:42 == Keep-Alive: timeout=5, max=100
23:34:42 == Connection: Keep-Alive
23:34:42 == Content-Type: text/plain
23:34:42 == 
23:34:42 == Hello World!
23:34:43 == HTTP/1.1 200 OK
23:34:43 == Date: Mon, 17 Feb 2020 23:34:43 GMT
23:34:43 == Server: Apache/2.4.18 (Ubuntu)
23:34:43 == Content-Length: 13
23:34:43 == Keep-Alive: timeout=5, max=99
23:34:43 == Connection: Keep-Alive
23:34:43 == Content-Type: text/plain
23:34:43 == 
23:34:43 == Hello World!
23:34:58 == Connection closed by foreign host.

Reproduce the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 503,
depending on apache2 MPM module:
mpm_event/mpm_worker/mpm_prefork)
---

$ lsb_release -cs
xenial

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.3.0-1.1build1

For reference on socket filename:

$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6244.0.1.sock

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:48:29 == Trying 127.0.0.1...
23:48:29 == Connected to 127.0.0.1.
23:48:29 == Escape character is '^]'.
23:48:29 == HTTP/1.1 200 OK
23:48:29 == Date: Mon, 17 Feb 2020 23:48:29 GMT
23:48:29 == Server: Apache/2.4.18 (Ubuntu)
23:48:29 == Content-Length: 13
23:48:29 == Keep-Alive: timeout=15, max=100
23:48:29 == Connection: Keep-Alive
23:48:29 == Content-Type: text/plain
23:48:29 == 
23:48:29 == Hello World!
23:48:35 == HTTP/1.1 503 Service Unavailable
23:48:35 == Date: Mon, 17 Feb 2020 23:48:35 GMT
23:48:35 == Server: Apache/2.4.18 (Ubuntu)
23:48:35 == Content-Length: 374
23:48:35 == Connection: close
23:48:35 == Content-Type: text/html; charset=iso-8859-1
23:48:35 == 
23:48:35 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
23:48:35 == <html><head>
23:48:35 == <title>503 Service Unavailable</title>
23:48:35 == </head><body>
23:48:35 == <h1>Service Unavailable</h1>
23:48:35 == <p>The server is temporarily unable to service your
23:48:35 == request due to maintenance downtime or capacity
23:48:35 == problems. Please try again later.</p>
23:48:35 == <hr>
23:48:35 == <address>Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port 80</address>
23:48:35 == </body></html>
23:48:35 == Connection closed by foreign host.

Notice the error log mention socket file name not found:

$ sudo tail /var/log/apache2/error.log 
...
[Mon Feb 17 23:47:43.366910 2020] [mpm_event:notice] [pid 5892:tid 140660633114496] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Feb 17 23:47:43.422167 2020] [mpm_event:notice] [pid 5892:tid 140660633114496] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations
[Mon Feb 17 23:47:43.422207 2020] [core:notice] [pid 5892:tid 140660633114496] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 17 23:47:46.384026 2020] [wsgi:error] [pid 5898:tid 140660447377152] (2)No such file or directory: [client 127.0.0.1:33536] mod_wsgi (pid=5898): Unable to connect to WSGI daemon process '127.0.0.1' on '/var/run/apache2/wsgi.5892.0.1.sock'.

The socket file name has changed:

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6244.1.1.sock

With the fix/patch for Xenial:
---

$ sudo add-apt-repository ppa:mfo/lp1863232v2
$ sudo apt update

$ sudo apt install -y libapache2-mod-wsgi
$ sudo systemctl restart apache2

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.3.0-1.1ubuntu1

Same behavior continues by default:
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.0.1.sock

$ ps 4149
  PID TTY      STAT   TIME COMMAND
 4149 ?        Ss     0:00 /usr/sbin/apache2 -k start

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.1.1.sock

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.2.1.sock

Manually changing the setting:
---

$ echo 'WSGISocketRotation Off' | sudo tee -a /etc/apache2/conf-enabled/wsgi.conf 
$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock

Notice name pattern change as in patch. (u33 = UID 33 = www-data:)

$ id -un 33
www-data

And socket file name remains constant:

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock

Check the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 500 now)
---

The second HTTP request (after reload) no longer hits Error 503,
but now Error 500, and the log file no longer mentions file not
found, but a different message.

The Error 500 is observed in/consistent with Eoan as well, thus
not a missing patch or something -- this is an improvement.

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:51:50 == Trying 127.0.0.1...
23:51:50 == Connected to 127.0.0.1.
23:51:50 == Escape character is '^]'.
23:51:50 == HTTP/1.1 200 OK
23:51:50 == Date: Mon, 17 Feb 2020 23:51:50 GMT
23:51:50 == Server: Apache/2.4.18 (Ubuntu)
23:51:50 == Content-Length: 13
23:51:50 == Keep-Alive: timeout=15, max=100
23:51:50 == Connection: Keep-Alive
23:51:50 == Content-Type: text/plain
23:51:50 == 
23:51:50 == Hello World!
23:51:56 == HTTP/1.1 500 Internal Server Error
23:51:56 == Date: Mon, 17 Feb 2020 23:51:56 GMT
23:51:56 == Server: Apache/2.4.18 (Ubuntu)
23:51:56 == Content-Length: 607
23:51:56 == Connection: close
23:51:56 == Content-Type: text/html; charset=iso-8859-1
23:51:56 == 
23:51:56 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
23:51:56 == <html><head>
23:51:56 == <title>500 Internal Server Error</title>
23:51:56 == </head><body>
23:51:56 == <h1>Internal Server Error</h1>
23:51:56 == <p>The server encountered an internal error or
23:51:56 == misconfiguration and was unable to complete
23:51:56 == your request.</p>
23:51:56 == <p>Please contact the server administrator at
23:51:56 == webmaster@localhost to inform them of the time this error occurred,
23:51:56 == and the actions you performed just before this error.</p>
23:51:56 == <p>More information about this error may be available
23:51:56 == in the server error log.</p>
23:51:56 == <hr>
23:51:56 == <address>Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port 80</address>
23:51:56 == </body></html>
23:51:56 == Connection closed by foreign host.

$ sudo tail /var/log/apache2/error.log 
...
[Mon Feb 17 23:51:51.260775 2020] [mpm_event:notice] [pid 6554:tid 140543840806784] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Feb 17 23:51:51.326786 2020] [mpm_event:notice] [pid 6554:tid 140543840806784] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations
[Mon Feb 17 23:51:51.326873 2020] [core:notice] [pid 6554:tid 140543840806784] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 17 23:51:56.277943 2020] [wsgi:alert] [pid 7039:tid 140543732401920] mod_wsgi (pid=7039): Request origin could not be validated.
[Mon Feb 17 23:51:56.278181 2020] [wsgi:error] [pid 6907:tid 140543690430208] [client 127.0.0.1:33556] Truncated or oversized response headers received from daemon process '127.0.0.1': /var/www/html/hello-world.py

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#10

Download full text (8.0 KiB)

Steps to reproduce/verify on Bionic:
===

Similarly to steps described for Xenial.
Skipping the identical steps.

$ lxc launch ubuntu:bionic lp1863232b
$ lxc exec lp1863232b -- su - ubuntu

One connection/Two requests (added timestamps to check timeout values)
---

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
00:03:46 == Trying 127.0.0.1...
00:03:46 == Connected to 127.0.0.1.
00:03:46 == Escape character is '^]'.
00:03:46 == HTTP/1.1 200 OK
00:03:46 == Date: Tue, 18 Feb 2020 00:03:46 GMT
00:03:46 == Server: Apache/2.4.29 (Ubuntu)
00:03:46 == Content-Length: 13
00:03:46 == Vary: Accept-Encoding
00:03:46 == Keep-Alive: timeout=15, max=100
00:03:46 == Connection: Keep-Alive
00:03:46 == Content-Type: text/plain
00:03:46 ==
00:03:46 == Hello World!
00:03:47 == HTTP/1.1 200 OK
00:03:47 == Date: Tue, 18 Feb 2020 00:03:47 GMT
00:03:47 == Server: Apache/2.4.29 (Ubuntu)
00:03:47 == Content-Length: 13
00:03:47 == Vary: Accept-Encoding
00:03:47 == Keep-Alive: timeout=15, max=99
00:03:47 == Connection: Keep-Alive
00:03:47 == Content-Type: text/plain
00:03:47 ==
00:03:47 == Hello World!
00:03:56 == Connection closed by foreign host.

Reproduce the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 503,
depending on apache2 MPM module.
mpm_event just closes connection,
mpm_worker/mpm_prefork hit 503)
---

$ lsb_release -cs
bionic

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.5.17-1

$ sudo a2dismod mpm_event
$ sudo a2enmod mpm_worker
$ sudo systemctl restart apache2

For reference on socket filename:

$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4982.0.1.sock

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
14:46:04 == Trying 127.0.0.1...
14:46:04 == Connected to 127.0.0.1.
14:46:04 == Escape character is '^]'.
14:46:04 == HTTP/1.1 200 OK
14:46:04 == Date: Tue, 18 Feb 2020 14:46:04 GMT
14:46:04 == Server: Apache/2.4.29 (Ubuntu)
14:46:04 == Content-Length: 13
14:46:04 == Vary: Accept-Encoding
14:46:04 == Keep-Alive: timeout=15, max=100
14:46:04 == Connection: Keep-Alive
14:46:04 == Content-Type: text/plain
14:46:04 ==
14:46:04 == Hello World!
14:46:10 == HTTP/1.1 503 Service Unavailable
14:46:10 == Date: Tue, 18 Feb 2020 14:46:10 GMT
14:46:10 == Server: Apache/2.4.29 (Ubuntu)
14:46:10 == Content-Length: 374
14:46:10 == Connection: close
14:46:10 == Content-Type: text/html; charset=iso-8859-1
14:46:10 ==
14:46:10 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
14:46:10 == <html><head>
14:46:10 == <title>503 Service Unavailable</title>
14:46:10 == </head><body>
14:46:10 == <h1>Service Unavailable</h1>
14:46:10 == <p>The server is temporarily unable to service your
14:46:10 == request due to maintenance downtime or capacity
14:46:10 == problems. Please try again later.</p>
14:46:10 == <hr>
14:46:10 == <address>Apache/2.4.29 (Ubuntu) Server at 127.0.0.1 Port 80</address>
14:46:11 == </body></html>
14:46:11 ...

Steps to reproduce/verify on Bionic:
===

Similarly to steps described for Xenial.
Skipping the identical steps.

$ lxc launch ubuntu:bionic lp1863232b
$ lxc exec lp1863232b -- su - ubuntu

One connection/Two requests (added timestamps to check timeout values)
---

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
00:03:46 == Trying 127.0.0.1...
00:03:46 == Connected to 127.0.0.1.
00:03:46 == Escape character is '^]'.
00:03:46 == HTTP/1.1 200 OK
00:03:46 == Date: Tue, 18 Feb 2020 00:03:46 GMT
00:03:46 == Server: Apache/2.4.29 (Ubuntu)
00:03:46 == Content-Length: 13
00:03:46 == Vary: Accept-Encoding
00:03:46 == Keep-Alive: timeout=15, max=100
00:03:46 == Connection: Keep-Alive
00:03:46 == Content-Type: text/plain
00:03:46 == 
00:03:46 == Hello World!
00:03:47 == HTTP/1.1 200 OK
00:03:47 == Date: Tue, 18 Feb 2020 00:03:47 GMT
00:03:47 == Server: Apache/2.4.29 (Ubuntu)
00:03:47 == Content-Length: 13
00:03:47 == Vary: Accept-Encoding
00:03:47 == Keep-Alive: timeout=15, max=99
00:03:47 == Connection: Keep-Alive
00:03:47 == Content-Type: text/plain
00:03:47 == 
00:03:47 == Hello World!
00:03:56 == Connection closed by foreign host.

Reproduce the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 503,
depending on apache2 MPM module.
mpm_event just closes connection,
mpm_worker/mpm_prefork hit 503)
---

$ lsb_release -cs
bionic

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.5.17-1

$ sudo a2dismod mpm_event
$ sudo a2enmod mpm_worker
$ sudo systemctl restart apache2

For reference on socket filename:

$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4982.0.1.sock

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
14:46:04 == Trying 127.0.0.1...
14:46:04 == Connected to 127.0.0.1.
14:46:04 == Escape character is '^]'.
14:46:04 == HTTP/1.1 200 OK
14:46:04 == Date: Tue, 18 Feb 2020 14:46:04 GMT
14:46:04 == Server: Apache/2.4.29 (Ubuntu)
14:46:04 == Content-Length: 13
14:46:04 == Vary: Accept-Encoding
14:46:04 == Keep-Alive: timeout=15, max=100
14:46:04 == Connection: Keep-Alive
14:46:04 == Content-Type: text/plain
14:46:04 == 
14:46:04 == Hello World!
14:46:10 == HTTP/1.1 503 Service Unavailable
14:46:10 == Date: Tue, 18 Feb 2020 14:46:10 GMT
14:46:10 == Server: Apache/2.4.29 (Ubuntu)
14:46:10 == Content-Length: 374
14:46:10 == Connection: close
14:46:10 == Content-Type: text/html; charset=iso-8859-1
14:46:10 == 
14:46:10 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
14:46:10 == <html><head>
14:46:10 == <title>503 Service Unavailable</title>
14:46:10 == </head><body>
14:46:10 == <h1>Service Unavailable</h1>
14:46:10 == <p>The server is temporarily unable to service your
14:46:10 == request due to maintenance downtime or capacity
14:46:10 == problems. Please try again later.</p>
14:46:10 == <hr>
14:46:10 == <address>Apache/2.4.29 (Ubuntu) Server at 127.0.0.1 Port 80</address>
14:46:11 == </body></html>
14:46:11 == Connection closed by foreign host.

$ sudo tail /var/log/apache2/error.log 
...
[Tue Feb 18 14:46:05.923939 2020] [mpm_worker:notice] [pid 4982:tid 139825593654208] AH00297: SIGUSR1 received.  Doing graceful restart
[Tue Feb 18 14:46:05.981362 2020] [mpm_worker:notice] [pid 4982:tid 139825593654208] AH00292: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/2.7 configured -- resuming normal operations
[Tue Feb 18 14:46:05.981409 2020] [core:notice] [pid 4982:tid 139825593654208] AH00094: Command line: '/usr/sbin/apache2'
[Tue Feb 18 14:46:10.941713 2020] [wsgi:error] [pid 4986:tid 139825361426176] (2)No such file or directory: [client 127.0.0.1:37514] mod_wsgi (pid=4986): Unable to connect to WSGI daemon process '127.0.0.1' on '/var/run/apache2/wsgi.4982.0.1.sock' as user with uid=33.

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4982.1.1.sock

With the fix/patch for Bionic:
---

$ sudo add-apt-repository ppa:mfo/lp1863232v2
$ sudo apt update

$ sudo apt install -y libapache2-mod-wsgi
$ sudo systemctl restart apache2

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.5.17-1ubuntu1

Same behavior by default:
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6043.0.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6043.1.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6043.2.1.sock

Now, manually changing it:
---

$ echo 'WSGISocketRotation Off' | sudo tee -a /etc/apache2/conf-enabled/wsgi.conf 
$ sudo systemctl restart apache2

Notice name pattern change as in patch. (u33 = UID 33 = www-data:)
And socket file name remains constant:

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6403.u33.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6403.u33.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6403.u33.1.sock

$ id -un 33
www-data

Check the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 500 now)
---

The second HTTP request (after reload) no longer hits Error 503,
but now Error 500, and the log file no longer mentions file not
found, but a different message.

The Error 500 is observed in/consistent with Eoan as well, thus
not a missing patch or something -- this is an improvement.

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo "$(date +'%T') == $line"; done
14:49:57 == Trying 127.0.0.1...
14:49:57 == Connected to 127.0.0.1.
14:49:57 == Escape character is '^]'.
14:49:57 == HTTP/1.1 200 OK
14:49:57 == Date: Tue, 18 Feb 2020 14:49:57 GMT
14:49:57 == Server: Apache/2.4.29 (Ubuntu)
14:49:57 == Content-Length: 13
14:49:57 == Vary: Accept-Encoding
14:49:57 == Keep-Alive: timeout=15, max=100
14:49:57 == Connection: Keep-Alive
14:49:57 == Content-Type: text/plain
14:49:57 == 
14:49:57 == Hello World!
14:50:03 == HTTP/1.1 500 Internal Server Error
14:50:03 == Date: Tue, 18 Feb 2020 14:50:03 GMT
14:50:03 == Server: Apache/2.4.29 (Ubuntu)
14:50:03 == Content-Length: 607
14:50:03 == Connection: close
14:50:03 == Content-Type: text/html; charset=iso-8859-1
14:50:03 == 
14:50:03 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
14:50:03 == <html><head>
14:50:03 == <title>500 Internal Server Error</title>
14:50:03 == </head><body>
14:50:03 == <h1>Internal Server Error</h1>
14:50:03 == <p>The server encountered an internal error or
14:50:03 == misconfiguration and was unable to complete
14:50:03 == your request.</p>
14:50:03 == <p>Please contact the server administrator at
14:50:03 == webmaster@localhost to inform them of the time this error occurred,
14:50:03 == and the actions you performed just before this error.</p>
14:50:03 == <p>More information about this error may be available
14:50:03 == in the server error log.</p>
14:50:03 == <hr>
14:50:03 == <address>Apache/2.4.29 (Ubuntu) Server at 127.0.0.1 Port 80</address>
14:50:03 == </body></html>
14:50:03 == Connection closed by foreign host.

$ sudo tail /var/log/apache2/error.log 
...
[Tue Feb 18 14:49:58.563491 2020] [mpm_worker:notice] [pid 6403:tid 139801328274368] AH00297: SIGUSR1 received.  Doing graceful restart
[Tue Feb 18 14:49:58.619982 2020] [mpm_worker:notice] [pid 6403:tid 139801328274368] AH00292: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/2.7 configured -- resuming normal operations
[Tue Feb 18 14:49:58.620021 2020] [core:notice] [pid 6403:tid 139801328274368] AH00094: Command line: '/usr/sbin/apache2'
[Tue Feb 18 14:50:03.581491 2020] [wsgi:alert] [pid 6735:tid 139801212405504] mod_wsgi (pid=6735): Request origin could not be validated.
[Tue Feb 18 14:50:03.581691 2020] [wsgi:error] [pid 6614:tid 139801220822784] [client 127.0.0.1:37590] Truncated or oversized response headers received from daemon process '127.0.0.1': /var/www/html/hello-world.py

Mauricio Faria de Oliveira (mfo) on 2020-02-18

description:	updated
description:	updated

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2020-02-18:

#11

Uploaded changes to B/X.

There are no other mod-wsgi uploads in the unapproved queue nor waiting in -proposed/pending SRU.

mod-wsgi (source)
4.5.17-1ubuntu1 main python medium ubuntu-server Proposed 56 seconds ago

mod-wsgi (source)
4.3.0-1.1ubuntu1 main python medium ubuntu-server Proposed 53 seconds ago

Changed in mod-wsgi (Ubuntu Xenial):
importance:	Undecided → Medium
Changed in mod-wsgi (Ubuntu Bionic):
importance:	Undecided → Medium
Changed in mod-wsgi (Ubuntu):
importance:	Low → Undecided
description:	updated

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2020-02-20: Please test proposed package

#12

Hello Dariusz, or anyone else affected,

Accepted mod-wsgi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mod-wsgi/4.5.17-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mod-wsgi (Ubuntu Bionic):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-bionic

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2020-02-20:

#13

Hello Dariusz, or anyone else affected,

Accepted mod-wsgi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mod-wsgi/4.3.0-1.1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in mod-wsgi (Ubuntu Xenial):
status:	In Progress → Fix Committed
tags:	added: verification-needed-xenial

Revision history for this message

Dariusz Gadomski (dgadomski) wrote on 2020-02-20:

#14

Bionic verification:
After installing libapache2-mod-wsgi 4.5.17-1ubuntu1 and setting
WSGISocketRotation Off
in /etc/apache2/mods-available/wsgi.conf I confirm there is no socket rotation after reload:

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4963.u33.1.sock
$ sudo systemctl reload apache2.service
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4963.u33.1.sock

tags:

added: verification-done-bionic
removed: verification-needed-bionic

Revision history for this message

Dariusz Gadomski (dgadomski) wrote on 2020-02-20:

#15

Xenial verification:
Similarly to bionic with Xenial version of libapache2-mod-wsgi 4.3.0-1.1ubuntu1 and setting
WSGISocketRotation Off
in /etc/apache2/conf-enabled/wsgi.conf there's no socket rotation:
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.3170.u33.1.sock
$ sudo systemctl reload apache2.service
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.3170.u33.1.sock
$ apt-cache policy libapache2-mod-wsgi | grep Installed
Installed: 4.3.0-1.1ubuntu1

tags:

added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial

Revision history for this message

Ubuntu SRU Bot (ubuntu-sru-bot) wrote on 2020-02-20: Autopkgtest regression report (mod-wsgi/4.5.17-1ubuntu1)

#16

All autopkgtests for the newly accepted mod-wsgi (4.5.17-1ubuntu1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

nova/2:17.0.12-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#mod-wsgi

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2020-03-02: Update Released

#17

The verification of the Stable Release Update for mod-wsgi has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2020-03-02:

#18

This bug was fixed in the package mod-wsgi - 4.5.17-1ubuntu1

---------------
mod-wsgi (4.5.17-1ubuntu1) bionic; urgency=medium

* d/p/allow-disabling-daemon-socket-rotation-on-restart.patch:
Add option for disabling daemon socket rotation on restarts (LP: #1863232)

-- Dariusz Gadomski <email address hidden> Fri, 14 Feb 2020 12:08:41 +0100

Changed in mod-wsgi (Ubuntu Bionic):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2020-03-02:

#19

This bug was fixed in the package mod-wsgi - 4.3.0-1.1ubuntu1

---------------
mod-wsgi (4.3.0-1.1ubuntu1) xenial; urgency=medium

* d/p/allow-disabling-daemon-socket-rotation-on-restart.patch:
Add option for disabling daemon socket rotation on restarts (LP: #1863232)

-- Dariusz Gadomski <email address hidden> Fri, 14 Feb 2020 12:36:00 +0100

Changed in mod-wsgi (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Mauricio Faria de Oliveira (mfo) on 2020-09-22

tags:

removed: sts-sponsor-mfo

Ubuntu
mod-wsgi package

daemon rotates socket on restart

Bug Description

Other bug subscribers

Patches

Remote bug watches

	Status	Importance	Assigned to
mod-wsgi (Ubuntu)	Fix Released	Undecided	Unassigned
Xenial	Fix Released	Medium	Dariusz Gadomski
Bionic	Fix Released	Medium	Dariusz Gadomski

Ubuntumod-wsgi package

daemon rotates socket on restart

Bug Description

Other bug subscribers

Patches

Remote bug watches

Ubuntu
mod-wsgi package