Juju Charms Collection
ceilometer package

install hook fails on systemd due to apt.systemd.daily execution

Bug #1656651 reported by Mario Splivalo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Ceilometer Charm
Fix Released
Medium
Mario Splivalo
OpenStack Ceilometer Charm 17.02
ceilometer (Juju Charms Collection)
Invalid
Medium
Mario Splivalo

Bug Description

When ceilometer charm tries to install dependencies for ceilometer, on systemd distros (xenial and above), this happens:

unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 dpkg-query: package 'python-apt' is not installed and no information is available
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 Use dpkg --info (= dpkg-deb --info) to examine archive files,
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 and dpkg --contents (= dpkg-deb --contents) to list their contents.
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 dpkg-query: package 'python-netaddr' is not installed and no information is available
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 Use dpkg --info (= dpkg-deb --info) to examine archive files,
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 and dpkg --contents (= dpkg-deb --contents) to list their contents.
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 dpkg-query: package 'python-netifaces' is not installed and no information is available
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 Use dpkg --info (= dpkg-deb --info) to examine archive files,
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 and dpkg --contents (= dpkg-deb --contents) to list their contents.
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 dpkg-query: package 'python-pip' is not installed and no information is available
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 Use dpkg --info (= dpkg-deb --info) to examine archive files,
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 and dpkg --contents (= dpkg-deb --contents) to list their contents.
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 dpkg-query: package 'python-yaml' is not installed and no information is available
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 Use dpkg --info (= dpkg-deb --info) to examine archive files,
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 and dpkg --contents (= dpkg-deb --contents) to list their contents.
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 /var/lib/juju/agents/unit-ceilometer-0/charm/hooks/install: /var/lib/juju/agents/unit-ceilometer-0/charm/hooks/install.real: /usr/bin/python: bad interpreter: No such file or directory
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 INFO unit.ceilometer/0.install logger.go:40 /var/lib/juju/agents/unit-ceilometer-0/charm/hooks/install: line 20: /var/lib/juju/agents/unit-ceilometer-0/charm/hooks/install.real: Success
unit-ceilometer-0[1764]: 2017-01-15 13:17:47 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 126

This is due to systemd blocking apt right after the container start, due to systemd initialization.

Fix should be implemented similar to this manner:

http://bazaar.launchpad.net/~mariosplivalo/charms/trusty/mongodb/lp1513094/revision/97

Tags: sts
Mario Splivalo (mariosplivalo)
Changed in ceilometer (Juju Charms Collection):
assignee: nobody → Mario Splivalo (mariosplivalo)
Mario Splivalo (mariosplivalo)
Changed in ceilometer (Juju Charms Collection):
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/420639

Revision history for this message
James Page (james-page) wrote : Re: Install hook fails on systemd due to dpkg lock

What is systemd doing on start with dpkg locks?

The challenge here is that this will potentially impact every openstack charm with the current python2 shim script (which is all of them I think).

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

When I 'strategically' place the "ps -ef | grep apt" inside install hook's bash script, one can see that it is apt.systemd.daily that is holding the lock, preventing hook's apt to run:

http://paste.ubuntu.com/23809930/

Revision history for this message
James Page (james-page) wrote :

Marking as Medium, as this is seen most frequently on the LXD local provider.

summary: - Install hook fails on systemd due to dpkg lock
+ install hook fails on systemd due to apt.systemd.daily execution
Changed in ceilometer (Juju Charms Collection):
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ceilometer (master)

Reviewed: https://review.openstack.org/420639
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=8e3b47678ef8a638b4846e32e0731016bf329cb5
Submitter: Jenkins
Branch: master

commit 8e3b47678ef8a638b4846e32e0731016bf329cb5
Author: Mario Splivalo <email address hidden>
Date: Mon Jan 16 11:26:27 2017 +0100

    Fix race condition with systemd's apt initialization

    When LXC container start systemd runs apt.systemd.daily,
    which locks dpkg database.

    This change implements a 'wait for lock release', so that
    install hook doesn't fail when deploying inside LXC.

    Change-Id: I71a38ec8ea208f6c9ef1ae3dabda63bd26fa20fa
    Closes-Bug: 1656651

Changed in ceilometer (Juju Charms Collection):
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-ceilometer:
assignee: nobody → Mario Splivalo (mariosplivalo)
importance: Undecided → Medium
status: New → Fix Committed
Changed in ceilometer (Juju Charms Collection):
status: Fix Committed → Invalid
James Page (james-page)
Changed in charm-ceilometer:
milestone: none → 17.02
James Page (james-page)
Changed in charm-ceilometer:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.