Deployment of MOS 9.1 with StackLight - InfluxDB/Grafana Fuel Plugin is failed

Bug #1652640 reported by Sergii Rizvan
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
StackLight
Fix Released
High
Simon Pasquier
0.10
Fix Released
Undecided
Unassigned

Bug Description

Summary:
systemd package was recently added to trusty-updates repository. That's why post-install script from influxdb package tries to use systemclt to enable influxdb service. That leads to deployment error.

Env description:
 - compute
 - cinder+controller
 - elasticsearch_kibana+influxdb_grafana

Steps to reproduce:
1. Get Fuel 9.1
2. Configure
 - compute
 - cinder+controller
 - elasticsearch_kibana+influxdb_grafana (stacklight plugin from the oficial mirantis catalog)
3. Deploy environment
4. Deployment ends up with the following error: http://paste.openstack.org/show/593380/

This issue occurs because post-install script tries to use systemcl to enable influxdb service. Here is the parts of the post-install script which lead to the issue:

...
function install_systemd {
  cp -f $SCRIPT_DIR/influxdb.service /lib/systemd/system/influxdb.service
  systemctl enable influxdb
}
...

...
elif [[ -f /etc/debian_version ]]; then
  # Debian/Ubuntu logic
  which systemctl &>/dev/null
  if [[ $? -eq 0 ]]; then
      install_systemd
  else
      # Assuming sysv
      install_init
      install_update_rcd
  fi
...

P.S. As I've figured out from here https://github.com/openstack/fuel-plugin-influxdb-grafana/blob/master/pre_build_hook#L14

pre_build_hook just downloads influxdb deb package from https://s3.amazonaws.com/influxdb/influxdb_0.11.1-1_amd64.deb
So we need to patch that package and place it somewhere else.

[1] https://lists.ubuntu.com/archives/trusty-changes/2016-December/023368.htm

Workaround #1:
=============

Add systemd-shim to "additional packages" in fuel interface before deploying the environment.

Workaround #2:
=============

1. Provision the environment.
2. Install systemd-shim on the influxdb-grafana nodes.
3. Deploy the environment.

Changed in lma-toolchain:
status: New → Confirmed
Revision history for this message
Javier Diaz Jr (javierdiazcharles) wrote :

I believe I am hitting the same issue with Stacklight in MOS 8 w/MU3. Ping me directly via Slack if you need access to lab where deployment is failing. See paste to match above:

http://paste.openstack.org/show/593479/

tags: added: customer-found
Revision history for this message
Ruslan Khozinov (rkhozinov) wrote :

That issue isn’t related to the https://bugs.launchpad.net/fuel/+bug/1652597

Sergii Rizvan (srizvan)
tags: added: blocker-for-qa
Changed in lma-toolchain:
assignee: nobody → Swann Croiset (swann-w)
assignee: Swann Croiset (swann-w) → guillaume thouvenin (guillaume-thouvenin)
Revision history for this message
guillaume thouvenin (guillaume-thouvenin) wrote :

I'm not sure to understand the issue. What you are suggesting is that the version of InfluxDB that is downloaded by the pre_build_hook failed to be installed because now systemd is used? So you mean that InfluxDB 0.11.1 cannot be installed if systemd is detected?

Revision history for this message
Simon Pasquier (simon-pasquier) wrote :

IIUC this is the same cause as for https://bugs.launchpad.net/fuel/+bug/1648732. Because systemd package was recently added to trusty-updates repository, it breaks packages that erroneously detect systemd as the init system and try to run 'systemctl enable ...' in the post-inst script. One way to fix it is to install the systemd-shim package in this situation.

Revision history for this message
Simon Pasquier (simon-pasquier) wrote :

Ideally systemd-shim should only be installed on Trusty.

Changed in lma-toolchain:
importance: Undecided → Critical
milestone: none → 1.0.0
Revision history for this message
Sergii Rizvan (srizvan) wrote :

systemd-shim package isn't present in the list of initial packages by default. That's why in default configuration systemd package is going to be installed. This leads to the issue described above.
Fix should be relatively easy: we need just to remove systemctl call from postinst script. For example, like this: https://review.fuel-infra.org/#/c/29463/

Revision history for this message
Simon Pasquier (simon-pasquier) wrote :

Please re-read my comment: systemd-shim will be available since it will be included in the plugin... I don't see a problem here. Also we don't have the bandwidth/expertise to rebuild the upstream package so your proposal is a no-go for us.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-influxdb-grafana (master)

Fix proposed to branch: master
Review: https://review.openstack.org/416276

Changed in lma-toolchain:
assignee: guillaume thouvenin (guillaume-thouvenin) → Simon Pasquier (simon-pasquier)
status: Confirmed → In Progress
Revision history for this message
Laurent (laurent-tupin) wrote :

I confirm that fix works on our environment with a MOS9.1.

Thanks for your job

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-influxdb-grafana (master)

Reviewed: https://review.openstack.org/416276
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=5f57e8dbe81e49ee51e1b022345965a503de6785
Submitter: Jenkins
Branch: master

commit 5f57e8dbe81e49ee51e1b022345965a503de6785
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 3 16:42:32 2017 +0100

    Fix InfluxDB install when systemd is present

    The influxdb package fails to install on Trusty machines because
    while systemd is now installed, it isn't running as the init system.
    This change installs the systemd-shim package to get rid of the issue.

    Change-Id: I571f923c877e723f74a39691e69df8957d21e95d
    Closes-Bug: #1652640

Changed in lma-toolchain:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-influxdb-grafana (stable/1.0)

Fix proposed to branch: stable/1.0
Review: https://review.openstack.org/419945

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-influxdb-grafana (stable/1.0)

Reviewed: https://review.openstack.org/419945
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=4d47c04caaf05fb1f66b6352fb4d07cb5f8cbe17
Submitter: Jenkins
Branch: stable/1.0

commit 4d47c04caaf05fb1f66b6352fb4d07cb5f8cbe17
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 3 16:42:32 2017 +0100

    Fix InfluxDB install when systemd is present

    The influxdb package fails to install on Trusty machines because
    while systemd is now installed, it isn't running as the init system.
    This change installs the systemd-shim package to get rid of the issue.

    Change-Id: I571f923c877e723f74a39691e69df8957d21e95d
    Closes-Bug: #1652640
    (cherry picked from commit 5f57e8dbe81e49ee51e1b022345965a503de6785)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-influxdb-grafana (stable/0.10)

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/420591

Revision history for this message
Ruslan Khozinov (rkhozinov) wrote :

I've builded the fuel-plugin-influxdb-grafana plugin from master

d201f5cf5620787cad2493d276c75c4dbb4d1ecd Prevent co-installation with the Contrail plugin
387ce3ee4c53f76ee854b25ec4d1bbad7d68f9fa Enforce strings in the Hiera YAML file
e8b18384e5dc763d80ece5abd5f80f7d79d73617 Don't rstrip() literal strings from Hiera
5f57e8dbe81e49ee51e1b022345965a503de6785 Fix InfluxDB install when systemd is present
768d468bf8010039fabe1203b27b50f7c763bbae Fix HAproxy dashboard
9a747fb3625e295113b83de344341c48e6c23ce3 Merge "Enforce string type for passwords in hiera data"
c6193a22165d2d69aac447da2ee7150306254054 Enforce string type for passwords in hiera data
6c21061d85787171e37e4d80b809205b5f55a2f7 [Docs] Edits
0e67121b7c9ed9217bd75ca3908763dd8bd7d748 Merge "Fix task dependency with the LMA collector plugin"
5f68b8579acfa6304d4d048dd0361d317dd98534 Update documentation for 1.0

And the redeployed previously failed env with 0.10.2 version.

So an env with the 1.0.0 version of the fuel-plugin-influxdb-grafana plugin has been successfully deployed.

Revision history for this message
Ruslan Khozinov (rkhozinov) wrote :

I've checked kibana/grafana services. They work properly.

Changed in lma-toolchain:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-influxdb-grafana (stable/0.10)

Reviewed: https://review.openstack.org/420591
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=acce42de965e424eaa9f660e14786ee11a46e463
Submitter: Jenkins
Branch: stable/0.10

commit acce42de965e424eaa9f660e14786ee11a46e463
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 3 16:42:32 2017 +0100

    Fix InfluxDB install when systemd is present

    The influxdb package fails to install on Trusty machines because
    while systemd is now installed, it isn't running as the init system.
    This change installs the systemd-shim package to get rid of the issue.

    Change-Id: I571f923c877e723f74a39691e69df8957d21e95d
    Closes-Bug: #1652640
    (cherry picked from commit 5f57e8dbe81e49ee51e1b022345965a503de6785)

description: updated
Revision history for this message
Simon Pasquier (simon-pasquier) wrote :

Lowering to High because there's a workaround.

Changed in lma-toolchain:
importance: Critical → High
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.