Disappearing /var/run/mysqld causes mysqld to fail after reboot

Bug #1435823 reported by Luca D'Isanto
110
This bug affects 20 people
Affects Status Importance Assigned to Milestone
mysql-5.6 (Ubuntu)
Fix Released
High
Unassigned
mysql-5.7 (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

This bug is no longer a duplicate of bug 1421303
(I open thi bug because Robie Basak say to open new bug)

with mysql-server version: 5.6.23-1~exp1~ubuntu4 mysql.service fail to start after reboot:

● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: activating (start-post) since mar 2015-03-24 08:41:00 CET; 25s ago
  Process: 951 ExecStart=/usr/bin/mysqld_safe (code=exited, status=0/SUCCESS)
  Process: 943 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 951 (code=exited, status=0/SUCCESS); : 952 (mysql-systemd-s)
   CGroup: /system.slice/mysql.service
           └─control
             ├─ 952 /bin/bash /usr/share/mysql/mysql-systemd-start post
             └─2601 sleep 1

mar 24 08:41:00 luca-k53sd systemd[1]: Starting MySQL Community Server...
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: 150324 08:41:00 mysqld_safe Can't log to error log and syslog at the same time. Remove all --log-err...e effect.
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: 150324 08:41:00 mysqld_safe Logging to '/var/log/mysql/error.log'.
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: mkdir: impossibile creare la directory "/var/run/mysqld": Permesso negato
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: chown: impossibile accedere a "/var/run/mysqld": File o directory non esistente
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: chmod: impossibile accedere a "/var/run/mysqld": File o directory non esistente
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: 150324 08:41:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Hint: Some lines were ellipsized, use -l to show in full.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: mysql-server 5.6.23-1~exp1~ubuntu4
ProcVersionSignature: Ubuntu 3.19.0-10.10-generic 3.19.2
Uname: Linux 3.19.0-10-generic x86_64
ApportVersion: 2.16.2-0ubuntu4
Architecture: amd64
CurrentDesktop: KDE
Date: Tue Mar 24 13:05:23 2015
InstallationDate: Installed on 2015-03-18 (6 days ago)
InstallationMedia: Kubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20150318)
PackageArchitecture: all
SourcePackage: mysql-5.6
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Luca D'Isanto (lukadisanto) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in mysql-5.6 (Ubuntu):
status: New → Confirmed
Revision history for this message
Bruno (bruno666-666) wrote :

Create a newfile /etc/tmpfiles.d/mysql.conf:

# systemd tmpfile settings for mysql
# See tmpfiles.d(5) for details

d /var/run/mysqld 0755 mysql mysql -

After reboot, mysql should start normally.

Revision history for this message
Robie Basak (racb) wrote :

@Luca

Does Bruno's fix work for you? I'd like to know if so, because that suggests that we have a problem with /var/run/mysqld not recreated after boot which is certainly a bug. I need to figure out steps to reproduce in order to fix it.

Revision history for this message
Luca D'Isanto (lukadisanto) wrote :

Hi,
With Bruno's fix works

Thank you!!

Robie Basak (racb)
summary: - mysql.service fail to start (mysql-server version:
- 5.6.23-1~exp1~ubuntu4)
+ Disappearing /var/run/mysqld causes mysqld to fail after reboot
Revision history for this message
Bruno (bruno666-666) wrote :

@Luca: prego;)

I guess that /run/mysqld is not created after boot because in /lib/systemd/mysql.service, mysqld_safe script (which seems to be responsible for its creation) is invoked with user/group mysql/mysql and /run is only writable by root.

Robie Basak (racb)
Changed in mysql-5.6 (Ubuntu):
importance: Undecided → High
tags: added: mysql-5.6-transition
Revision history for this message
Bruce Pieterse (octoquad) wrote :

Comment #3 works for me as well.

Revision history for this message
Rocko (rockorequin) wrote :

Comment #3 fixed it for me, too.

Revision history for this message
dave nelson (davidnelson) wrote :

Comment #3 fixed it for me, too.

Revision history for this message
atomiko (lex-atomiko) wrote :

Comment #3 works perfect, thank you!

Robie Basak (racb)
Changed in mysql-5.6 (Ubuntu):
milestone: none → ubuntu-15.04
Robie Basak (racb)
Changed in mysql-5.6 (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.2 KiB)

This bug was fixed in the package mysql-5.6 - 5.6.23-1~exp1~ubuntu5

---------------
mysql-5.6 (5.6.23-1~exp1~ubuntu5) vivid; urgency=medium

  * Ubuntu upload from Debian VCS commit 804459b.

mysql-5.6 (5.6.23-1~exp1) UNRELEASED; urgency=medium

  [ Robie Basak ]
  * New upstream release.

  [ Norvald H. Ryeng ]
  * Remove unused build options.
  * Add libedit dependency to mysql-client-core-5.6.
  * Remove chmods not needed after fix for upstream bug #68517.
  * Lintian fixes.

  [ Bjoern Boschman ]
  * Removed hardening-wrapper from build-deps.
  * Added python dep for mysql-testsuite-5.6.
  * Added dpkg-dev dep.
  * Cleanup a bit more.

  [ Robie Basak ]
  * Add dep8 smoke test.
  * Re-enable libmysqld-pic, libmysqld-dev, libmysqlclient18 and
    libmysqlclient-dev (renamed accordingly) to replace binary packages
    provided by src:mysql-5.5.
  * Add mysql-server, mysql-client and mysql-testsuite to be switched
    over from src:mysql-5.5.
  * Import mysql-common from src:mysql-5.5 for 5.5 -> 5.6 switchover.
  * Cherry-pick relevant parts of Ubuntu delta from mysql-5.5:
    - d/mysql-server-5.5.postrm: restoring postrm debhelper token to get
      proper behaviors from dh_installinit and dh_apparmor.
    - Add Apport hook.

  [ James Page ]
  * d/tests/upstream: Add main.ctype_uca to skip-test list as it requires
    writable /usr.
  * d/p/fix-mysqlhotcopy-test-failure.patch: Add return code 255 to the
    list of allowed return codes.
  * d/tests/control: Re-enable upstream DEP-8 tests.

  [ Otto Kekäläinen ]
  * Added parallel build support via DEB_BUILD_OPTIONS

  [ Otto Kekäläinen ]
  * Remove makeflags from cmake command and output MAKEFLAGS for debugging
    purposes just before make is run

  [ Akhil Mohan ]
  * Provisioned dep8 test case upstream separated to run as non-root

  [ Norvald H. Ryeng ]
  * Simplify and stabilize tests
  * Remove USE_STATIC_MYSQLD
  * Add test plan to README.Maintainer
  * Add --force rationale to test plan

  [ Akhil Mohan ]
  * Make builds parallel Use MAKEFLAGS for parallel builds

  [ Otto Kekäläinen ]
  * Move mysql_plugin to server package as it controls the server plugins
  * Wrap-and-sort
  * Remove legacy 5.0->5.1 transition code that refers a internal-use-only
    -directory that does not even exist anymore.
  * Removed all unnecessary .dirs as most of the directories are generated
    automatically
  * Don't emit extra output at init invocation Many users have complained that
    they're database is corrupt because they tried to start the database and it
    refuses to do so, and at the same time emitting a message with the word
    "corrupt". this is just plain wrong and confusing, so I'm removing it.

  [ Akhil Mohan ]
  * added systemd service profile and script

  [ Norvald H. Ryeng ]
  * Remove special build options for taocrypt

  [ Robie Basak ]
  * Drop unused variable definition
  * Create /etc/mysql manually in debian/rules
  * Replace conf.d/.keepme with an empty directory

  [ Akhil Mohan ]
  * Standards version moved up to 3.9.6 in d/control
  * Added Systemd service profile and script

  [ Robie Basak ]
  * wrap-and-sort for systemd change
  * Drop unnecessary binary target
  ...

Read more...

Changed in mysql-5.6 (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
al (e-webmaszer-t) wrote :

Hi,

I am not an expert,so I do'nt know how to apply the Bruno's fix

When I start mysql I get:

Sep 11 11:08:52 serv mysql[25802]: Checking for tables which need an up...e
Sep 11 11:08:52 serv mysql[25802]: not closed cleanly..
Hint: Some lines were ellipsized, use -l to show in full.

I understand I have to

Create a newfile /etc/tmpfiles.d/mysql.conf:

What does it mean " d /var/run/mysqld 0755 mysql mysql - " ? Delete?

Thank You to all mysql experts that can solve this question.

Yours Sincerely,
Alex

Revision history for this message
Bruce Pieterse (octoquad) wrote :

Hi al,

As per tempfile(1) (man tempfile) that line creates a new directory at /var/run/mysqld with 0755 permissions and sets the owner and group to mysql.

Revision history for this message
Robie Basak (racb) wrote :

@kiaikim

Why did you add a task for mysql-5.7? This bug was fixed in 5.6.25-2 which predates mysql-5.7.

Changed in mysql-5.7 (Ubuntu):
status: New → Invalid
Revision history for this message
kiaikim (kiaikim) wrote :

@racb
I'm sorry. I think I got webpages confused. I had the same problem with 5.7.17 on Ubuntu 16.04. I looked all over and tried whatever I could find. Must have thought I tried #3, but I did chown of /var/run/mysqld/ with mysql service running, which also worked. If you'd like I can start over, see if the problem persists and try #3. But probably better to see if others have this problem.

Revision history for this message
kiaikim (kiaikim) wrote :

Actually, I created /var/run/mysqld/, and then did chown.

Revision history for this message
Robie Basak (racb) wrote :

No problem. Feel free to file a separate bug against mysql-5.7 and link to it from here. If you can, please find steps that a developer can use to reproduce the problem. If you can't, then feel free to file a bug but please set the status to Incomplete. Others affected can then find and communicate in the bug with you to try and figure out steps to reproduce, or a technical explanation of what is happening.

Note though that the community doesn't generally follow bugs, so to find others to get help you are more likely to get help in other places. See http://www.ubuntu.com/support/community for community support options.

Revision history for this message
Orin Fink (orin-fink) wrote :

This issue seems to be relevant also in the most recent update from apt for Ubuntu 14.04. (Note i am running percona version) I just ran the update this morning and am now seeing this same issue. I've tried to implement the tmfiles.d solution but that does not seem to take effect. I _can_ manually create the /var/run/mysqld/ folder and chown same as kiakim, but this manual step is a bit wonky.

When will the 'fix' be available? Or, is this now an issue for the percona team to address?

Revision history for this message
Orin Fink (orin-fink) wrote :

This may or may not be relevant... but i did notice in the release notes for the 5.6.35 version of mysql (the version now being used by our ubuntu 14.04 update)... it looks like work has been done on the mysqld_safe script and use of chown is removed. Is it possible the necessary security lockdown re-introduced the bug.

https://dev.mysql.com/doc/relnotes/mysql/5.6/en/news-5-6-35.html

Revision history for this message
Szalai Kálmán (ulyszk) wrote :

Same issue here, workaround by tmpfiles.d.

Revision history for this message
Robie Basak (racb) wrote :

> When will the 'fix' be available?

This bug has already been fixed. If the issue has somehow recurred, or occurred in a different package, then please file a new bug with a full report (packages and versions used, steps to reproduce, etc) and link to this one. If you do not do this, it will not appear in our reporting and thus will never get fixed.

Revision history for this message
kiaikim (kiaikim) wrote :

I'm not the only one having this problem with 5.7. See the first comment under http://stackoverflow.com/a/42640731/2668783

Revision history for this message
Robie Basak (racb) wrote :

> I'm not the only one having this problem with 5.7.

That's fine, but as I said, please file a separate bug with a full report.

Revision history for this message
gene (geneyang) wrote :

This bug is appear AGAIN !!!

# systemd-tmpfiles --create
# ls -d /var/run/mysqld
/var/run/mysqld

# ls -dF /var/run/mysqld
/var/run/mysqld/

# service mysql start
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.

# ls -dF /var/run/mysqld
ls: cannot access '/var/run/mysqld': No such file or directory

# systemctl status mysql.service
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-04-08 13:20:58 CST; 1min 10s ago
  Process: 15705 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=1/FAILURE)

Apr 08 13:20:58 localhost systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 08 13:20:58 localhost systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Apr 08 13:20:58 localhost systemd[1]: Stopped MySQL Community Server.
Apr 08 13:20:58 localhost systemd[1]: mysql.service: Start request repeated too quickly.
Apr 08 13:20:58 localhost systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 08 13:20:58 localhost systemd[1]: Failed to start MySQL Community Server.

# mysqld --version
mysqld Ver 5.7.29-0ubuntu0.18.04.1 for Linux on x86_64 ((Ubuntu))

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

Revision history for this message
Paride Legovini (paride) wrote :

Hello gene,

I doubt the problem you are observing is related to the issue described in this report. In this bug description the failure mode is:

mar 24 08:41:00 luca-k53sd mysqld_safe[951]: mkdir: impossibile creare la directory "/var/run/mysqld": Permesso negato
mar 24 08:41:00 luca-k53sd mysqld_safe[951]: chown: impossibile accedere a "/var/run/mysqld": File o directory non esistente

where the errors are "Permission denied" and "No such file or directory". This doesn't happen in your case, what happens is that systemd can't start the service and cleans up the tmp directory after giving up.

Anyway if you are still convinced you are hitting this problem again please file a separate report for mysql 5.7, as this one is specifically for version 5.6. See Robie's comments above. Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.