package nginx-core 1.18.0-6ubuntu14.1 failed to install/upgrade during focal->jammy upgrade

Status changed to 'Confirmed' because the bug affects multiple users.

Just to confirm, you *do* have libnginx-mod-stream-geoip and libnginx-mod-stream both installed, correct?

Right, i attached the console output to this comment:

root@host:/etc/nginx/modules-enabled$ dpkg -l|grep nginx
ii libnginx-mod-http-geoip2 1.18.0-6ubuntu14.1 amd64 GeoIP2 HTTP module for Nginx
ii libnginx-mod-http-image-filter 1.18.0-6ubuntu14.1 amd64 HTTP image filter module for Nginx
ii libnginx-mod-http-xslt-filter 1.18.0-6ubuntu14.1 amd64 XSLT Transformation module for Nginx
ii libnginx-mod-mail 1.18.0-6ubuntu14.1 amd64 Mail module for Nginx
ii libnginx-mod-stream 1.18.0-6ubuntu14.1 amd64 Stream module for Nginx
ii libnginx-mod-stream-geoip2 1.18.0-6ubuntu14.1 amd64 GeoIP2 Stream module for Nginx
ii nginx 1.18.0-6ubuntu14.1 amd64 small, powerful, scalable web/proxy server
ii nginx-common 1.18.0-6ubuntu14.1 all small, powerful, scalable web/proxy server - common files
ii nginx-core 1.18.0-6ubuntu14.1 amd64 nginx web/proxy server (standard version)
ii python3-certbot-nginx 1.21.0-1 all Nginx plugin for Certbot
root@host:/etc/nginx/modules-enabled$ apt reinstall libnginx-mod-stream-geoip2 libnginx-mod-stream libnginx-mod-http-geoip2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 3 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/94.9 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 87499 files and directories currently installed.)
Preparing to unpack .../libnginx-mod-http-geoip2_1.18.0-6ubuntu14.1_amd64.deb ...
Unpacking libnginx-mod-http-geoip2 (1.18.0-6ubuntu14.1) over (1.18.0-6ubuntu14.1) ...
Preparing to unpack .../libnginx-mod-stream_1.18.0-6ubuntu14.1_amd64.deb ...
Unpacking libnginx-mod-stream (1.18.0-6ubuntu14.1) over (1.18.0-6ubuntu14.1) ...
Preparing to unpack .../libnginx-mod-stream-geoip2_1.18.0-6ubuntu14.1_amd64.deb ...
Unpacking libnginx-mod-stream-geoip2 (1.18.0-6ubuntu14.1) over (1.18.0-6ubuntu14.1) ...
Setting up libnginx-mod-stream (1.18.0-6ubuntu14.1) ...
Setting up libnginx-mod-http-geoip2 (1.18.0-6ubuntu14.1) ...
Setting up libnginx-mod-stream-geoip2 (1.18.0-6ubuntu14.1) ...
Processing triggers for nginx-core (1.18.0-6ubuntu14.1) ...
Triggering nginx reload ...
root@host:/etc/nginx/modules-enabled$ cat 70-mod-stream-geoip2.conf
load_module modules/ngx_stream_geoip2_module.so;
root@host:/etc/nginx/modules-enabled$ nginx -T
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_stream_geoip2_module.so" failed (/usr/share/nginx/modules/ngx_stream_geoip2_module.so: undefined symbol: ngx_stream_core_module) in /etc/nginx/modules-enabled/70-mod-stream-geoip2.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed
root@host:/etc/nginx/modules-enabled$ file /usr/share/nginx/modules/ngx_stream_geoip2_m...

It's geoip2, albeit i installed also the normal geoip module:

root@host:~$ apt reinstall libnginx-mod-stream-geoip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 9,134 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://mirror.hetzner.com/ubuntu/packages jammy-updates/universe amd64 libnginx-mod-stream-geoip amd64 1.18.0-6ubuntu14.1 [9,134 B]
Fetched 9,134 B in 0s (286 kB/s)
(Reading database ... 87518 files and directories currently installed.)
Preparing to unpack .../libnginx-mod-stream-geoip_1.18.0-6ubuntu14.1_amd64.deb ...
Unpacking libnginx-mod-stream-geoip (1.18.0-6ubuntu14.1) over (1.18.0-6ubuntu14.1) ...
Setting up libnginx-mod-stream-geoip (1.18.0-6ubuntu14.1) ...
Processing triggers for nginx-core (1.18.0-6ubuntu14.1) ...
Triggering nginx reload ...
root@host:~$ nginx -T
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_stream_geoip_module.so" failed (/usr/share/nginx/modules/ngx_stream_geoip_module.so: undefined symbol: ngx_stream_add_variable) in /etc/nginx/modules-enabled/70-mod-stream-geoip.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed
root@host:~$

Same error.

Can you provide a minimal reproducible example of an nginx configuration we can use to test with? This way we can do some digging and testing.

root@host:/etc/nginx# cat nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
 worker_connections 768;
 # multi_accept on;
}

http {

 ##
 # Basic Settings
 ##

 sendfile on;
 tcp_nopush on;
 types_hash_max_size 2048;
 # server_tokens off;

 # server_names_hash_bucket_size 64;
 # server_name_in_redirect off;

 include /etc/nginx/mime.types;
 default_type application/octet-stream;

 ##
 # SSL Settings
 ##

 ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
 ssl_prefer_server_ciphers on;

 ##
 # Logging Settings
 ##

 access_log /var/log/nginx/access.log;
 error_log /var/log/nginx/error.log;

 ##
 # Gzip Settings
 ##

 gzip on;

 # gzip_vary on;
 # gzip_proxied any;
 # gzip_comp_level 6;
 # gzip_buffers 16 8k;
 # gzip_http_version 1.1;
 # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

 ##
 # Virtual Host Configs
 ##

 include /etc/nginx/conf.d/*.conf;
 include /etc/nginx/sites-enabled/*;
}

#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
root@nz2:/etc/nginx# ls -R *
fastcgi.conf fastcgi_params koi-utf koi-win mime.types nginx.conf nginx.conf_old proxy_params scgi_params uwsgi_params win-utf

conf.d:

modules-available:

modules-enabled:
50-mod-http-geoip2.conf 70-mod-stream-geoip2.conf

sites-available:

sites-enabled:

snippets:
fastcgi-php.conf snakeoil.conf
root@host:/etc/nginx# cat modules-enabled/*
load_module modules/ngx_http_geoip2_module.so;
load_module modules/ngx_stream_geoip2_module.so;
root@host:/etc/nginx#

Curiously on a different server (also vserver but intel cpu) it works (used the same config file)...
md5sums are matching and as follows:
root@host:/etc/nginx/modules-enabled# md5sum $(which nginx)
8ae236b8cfaa5ba5f471ab7fba65700d /usr/sbin/nginx
root@host:/etc/nginx/modules-enabled# md5sum /usr/share/nginx/modules/ngx_stream_geoip2_module.so
9205822e0db2d913277e45fa1e4cc648 /usr/share/nginx/modules/ngx_stream_geoip2_module.so
root@host:/etc/nginx/modules-enabled#

Cannot reproduce on a fresh Focal instance.

Are you doing anything special with this one server? Custom libraries, custom builds, mix-and-match PPAs, different GCC versions, etc? This doesn't sound like a bug in the package, but more a problem with that one server because it can't be reproduced on a fresh Focal installation.

Hi everyone,

I am having the same issue as the one describe here. Two days ago, I upgraded my Ubuntu 20.04 to 22.04 and noticed that Nginx is not working anymore

Would you like to have any attachment with my comment?

Many thanks for your response

Hi Arnaud, thanks for the offer to help. I'd suggest by looking at Thomas' questions in comment #8 that need answered. For capturing your own error data, I'd suggest getting /var/log/apt/history.log and /var/log/apt/term.log, your nginx config customizations and nginx error logs (if they show anything useful), and relevant journalctl output.

Arnaud also make double sure you have the exact same version numbers as Steffen, same distro release (22.04), etc. Sometimes different bugs can have similar error messages. If in any doubt, please file a new bug; it's easier to dupe your bug to this one than to address two potentially distinct issues in a single bug report.

Like Thomas, I was unable to reproduce this using a stock config, including with libnginx-mod-stream-geoip installed. I also reviewed your config (from comment #7) against mine, and they're pretty much identical, however I have more items in /etc/nginx/modules-enabled/ than you do.

When I removed the links in /etc/nginx/modules-enabled/ to match yours, I did reproduce it:

triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo nginx -T > /dev/null
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo mv /tmp/70-mod-stream-geoip.conf .
triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo nginx -T > /dev/null
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
triage-jammy+22.04:/etc/nginx/modules-enabled$ ls
50-mod-http-echo.conf@ 50-mod-http-image-filter.conf@ 50-mod-mail.conf@ 70-mod-stream-geoip.conf@
50-mod-http-geoip2.conf@ 50-mod-http-xslt-filter.conf@ 50-mod-stream.conf@ 70-mod-stream-geoip2.conf@
triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo mkdir /tmp/foo
triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo mv 50-mod-http-echo.conf 50-mod-http-image-filter.conf 50-mod-http-xslt-filter.conf 50-mod-mail.conf 50-mod-stream.conf 70-mod-stream-geoip.conf /tmp/foo/
triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo nginx -T > /dev/null
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_stream_geoip2_module.so" failed (/usr/share/nginx/modules/ngx_stream_geoip2_module.so: undefined symbol: ngx_stream_core_module) in /etc/nginx/modules-enabled/70-mod-stream-geoip2.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

Moving the links back one by one and re-testing, it appears the wayward one was:

triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo mv /tmp/foo/50-mod-stream.conf .
triage-jammy+22.04:/etc/nginx/modules-enabled$ sudo nginx -T > /dev/null
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Do you think potentially you may have manually removed 50-mod-stream.conf, or did somehow it get removed as part of the upgrade process?

I just encountered the same issue. I originally had 18.04 installed and first upgraded to 20.04 without problems. A day later I upgraded to 22.04 which gave me the nginx error described above.
Looking into the /etc/nginx/modules-enabled folder there were only two symbolic links:

total 0
lrwxrwxrwx 1 root root 55 Mar 14 13:41 50-mod-http-geoip2.conf -> /usr/share/nginx/modules-available/mod-http-geoip2.conf
lrwxrwxrwx 1 root root 57 Mar 14 13:42 70-mod-stream-geoip2.conf -> /usr/share/nginx/modules-available/mod-stream-geoip2.conf

Date and time of those links match the time when the upgrade to 22.04 was performed. I don't remember having touched this folder manually (but I am not 100% sure).

Using the information from the previous comment I was able to fix the problem by creating a link for an additional module:

user@host:/etc/nginx/modules-enabled$ sudo ln -s /usr/share/nginx/modules-available/mod-stream.conf 50-mod-stream.conf

After that nginx started without further problems.