Ubuntu
mysql-5.7 package

mysqld eats more than 16 GB of memory on startup

Bug #1839527 reported by Alexander E. Patrakov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
mysql-5.7 (Ubuntu)
New
Undecided
Unassigned

Bug Description

(this bug has been reproduced in an LXC container running on Arch Linux, that's why the strange kernel version)

On Ubuntu 18.04, systemd runs mysqld in two steps. First:

ExecStartPre=/usr/share/mysql/mysql-systemd-start pre

In this script, one of the steps is: mysqld --verbose --help --innodb-read-only

Then:

ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid

It's the first mysqld (with --help) which is a problem. The problem happens because, due to PermissionsStartOnly=true, systemd runs ExecStartPre commands with insane limits. In particular:

Aug 08 18:19:23 mysql57 bash[13131]: core file size (blocks, -c) unlimited
Aug 08 18:19:23 mysql57 bash[13131]: data seg size (kbytes, -d) unlimited
Aug 08 18:19:23 mysql57 bash[13131]: scheduling priority (-e) 0
Aug 08 18:19:23 mysql57 bash[13131]: file size (blocks, -f) unlimited
Aug 08 18:19:23 mysql57 bash[13131]: pending signals (-i) 63701
Aug 08 18:19:23 mysql57 bash[13131]: max locked memory (kbytes, -l) 16384
Aug 08 18:19:23 mysql57 bash[13131]: max memory size (kbytes, -m) unlimited
Aug 08 18:19:23 mysql57 bash[13131]: open files (-n) 1073741816 <--- see this?
Aug 08 18:19:23 mysql57 bash[13131]: pipe size (512 bytes, -p) 8
Aug 08 18:19:23 mysql57 bash[13131]: POSIX message queues (bytes, -q) 819200
Aug 08 18:19:23 mysql57 bash[13131]: real-time priority (-r) 0
Aug 08 18:19:23 mysql57 bash[13131]: stack size (kbytes, -s) 8192
Aug 08 18:19:23 mysql57 bash[13131]: cpu time (seconds, -t) unlimited
Aug 08 18:19:23 mysql57 bash[13131]: max user processes (-u) 63701
Aug 08 18:19:23 mysql57 bash[13131]: virtual memory (kbytes, -v) unlimited
Aug 08 18:19:23 mysql57 bash[13131]: file locks (-x) unlimited

When running with such a high "ulimit -n" (again, this is something which systemd does by default for ExecStartPre), "mysqld --help" consumes unimaginable amounts of memory and even causes swapping and page allocation failures on my PC with 16 GB of RAM.

Workaround in /etc/systemd/system/mysql.service.d/override.conf:

[Service]
ExecStartPre=
ExecStartPre=/bin/sh -c 'ulimit -n 1024 ; /usr/share/mysql/mysql-systemd-start pre'

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: mysql-server-5.7 5.7.27-0ubuntu0.18.04.1
Uname: Linux 5.2.5-arch1-1-ARCH x86_64
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
Date: Thu Aug 8 18:29:38 2019
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
 [mysqldump]
 quick
 quote-names
 max_allowed_packet = 16M
MySQLConf.etc.mysql.mysql.conf.d.mysqld_safe_syslog.cnf:
 [mysqld_safe]
 syslog
MySQLVarLibDirListing: False
ProcEnviron:
 LANG=en_US.UTF-8
 TERM=xterm-256color
 SHELL=/bin/bash
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, no user)
SourcePackage: mysql-5.7
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.mysql.mysql.conf.d.mysqld.cnf: [modified]
mtime.conffile..etc.mysql.mysql.conf.d.mysqld.cnf: 2019-08-08T17:51:27.927907

Revision history for this message
Alexander E. Patrakov (patrakov-gmail) wrote :
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Since the LimitNOFILE=5000 isn't applied, the process gets whatever default the host has. In this case, Arch's kernel, since this is an LXC container.

Aug 08 18:19:23 mysql57 bash[13131]: open files (-n) 1073741816 <--- see this?

You probably have the same value on the host. I've seen another bug recently that involved arch linux and high limits: https://github.com/kubernetes-sigs/kind/pull/760

There it is claimed that Arch's default is 1073741816, which matches your number above. A bug was filed against mariadb, which has the same behavior you described with mysql here:

https://jira.mariadb.org/browse/MDEV-18360

The same code is in mysql 5.7

All that being said, this works fine when the host is Ubuntu, because the limits are set to lower values there by default. One could still argue that mysql is running on Ubuntu, just with an "odd" non-ubuntu kernel, and that it shouldn't allocate memory like that.

tags: added: server-triage-discuss
Revision history for this message
Alexander E. Patrakov (patrakov-gmail) wrote :

Well, it is not just "Arch Linux default". What happens here is that recent systemd upstream (and therefore Arch Linux, too) sets fs.nr_open=1073741816 by default. The commit that introduces this is https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812. LXC just inherits it.

See also the discussion here: https://github.com/systemd/systemd/issues/11510

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I filed this bug upstream: https://bugs.mysql.com/bug.php?id=96525

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Still, something is happening in ubuntu where we still get the conservative 1024*1024 value, and we do have systemd 240 in eoan. I removed the 5000 setting from the mysql service file, issued daemon-reload, and I still get 1024*1024.

In any case, by forcing a larger limit, I was able to reproduce the issue.

UPDATE: ah, there is a change in the debian/ubuntu packaging:
systemd (240-2) unstable; urgency=medium
...
  * Don't bump fs.nr_open in PID 1.
    In v240, systemd bumped fs.nr_open in PID 1 to the highest possible
    value. Processes that are spawned directly by systemd, will have
    RLIMIT_NOFILE be set to 512K (hard).
...

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Upstream bug was verified by them.

Andreas Hasenack (ahasenack)
tags: removed: server-triage-discuss
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.