wrong order of execution of cloud-init modules. 'write_files' is executed before 'runcmd'

Bug #1543063 reported by Alexander Gordeev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Mikhail Zhnichkov
Mitaka
Won't Fix
High
Sergey Slipushenko

Bug Description

it's expected that 'write_files' will be executed at the very end of cloud-init and it must be likely the last module to be executed by cloud-init. At least, it shall be executed before 'runcmd'

cloud-init modules will be executed by the order they were defined in cloud.cfg:
Eg. http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/config/cloud.cfg

having 'write_files' processed before 'runcmd' could indicate a node to be completely provisioned while it's not true. 'runcmd' could be executed and interfere with the first tasks of granular deployment.

https://github.com/openstack/fuel-agent/blob/master/cloud-init-templates/cloud_config_ubuntu.jinja2#L91-L103

in other words, this is the root cause of https://bugs.launchpad.net/fuel/+bug/1536924

provided fix for #1536924 only mitigates the issue.

Revision history for this message
Alexander Gordeev (a-gordeev) wrote :
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

2016-02-05 15:27:33,779 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/nocloud/sem/config_write_files - wb: [420] 19 bytes
2016-02-05 15:27:33,780 - helpers.py[DEBUG]: Running config-write-files using lock (<FileLock using file '/var/lib/cloud/instances/nocloud/sem/config_write_files'>)
2016-02-05 15:27:33,780 - cc_write_files.py[WARNING]: Undecodable permissions None, assuming 420
2016-02-05 15:27:33,781 - util.py[DEBUG]: Writing to /etc/nailgun-agent/config.yaml - wb: [420] 36 bytes
2016-02-05 15:27:33,782 - util.py[DEBUG]: Changing the ownership of /etc/nailgun-agent/config.yaml to 0:0
2016-02-05 15:27:33,782 - cc_write_files.py[WARNING]: Undecodable permissions None, assuming 420
2016-02-05 15:27:33,782 - util.py[DEBUG]: Writing to /etc/nailgun_systemtype - wb: [420] 6 bytes
2016-02-05 15:27:33,783 - util.py[DEBUG]: Changing the ownership of /etc/nailgun_systemtype to 0:0

Feb 5 15:27:40 node-5 [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instances/nocloud/sem/config_runcmd - wb: [420] 20 bytes
Feb 5 15:27:40 node-5 [CLOUDINIT] helpers.py[DEBUG]: Running config-runcmd using lock (<FileLock using file '/var/lib/cloud/instances/nocloud/sem/config_runcmd'>)
Feb 5 15:27:40 node-5 [CLOUDINIT] util.py[DEBUG]: Shellified 5 commands.
Feb 5 15:27:40 node-5 [CLOUDINIT] util.py[DEBUG]: Writing to /var/lib/cloud/instances/nocloud/scripts/runcmd - wb: [448] 178 bytes

Feb 5 15:27:54 node-5 [CLOUDINIT] util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/runcmd'] with allowed return codes [0] (shell=False, capture=False)

Changed in fuel:
milestone: none → 9.0
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

I think it's high priority. It didn't break provisioning/deployment so far, but it probably could.

Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

we need to include our customized cloud.cfg. It could be packaged and included into cloud-init package. Otherwise, fuel-agent can inject this config during the provisioning stage.

description: updated
Ilya Kutukov (ikutukov)
Changed in fuel:
importance: Undecided → Medium
importance: Medium → High
assignee: nobody → Fuel Python Team (fuel-python)
status: New → Confirmed
Dmitry Pyzhov (dpyzhov)
tags: added: area-python tech-debt
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Sergey Slipushenko (sslypushenko)
Changed in fuel:
assignee: Sergey Slipushenko (sslypushenko) → Alexander Saprykin (cutwater)
assignee: Alexander Saprykin (cutwater) → Sergey Slipushenko (sslypushenko)
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Technical debt, doesn't affect real users. Removing from Mitaka release.

Changed in fuel:
assignee: Sergey Slipushenko (sslypushenko) → Fuel Python (Deprecated) (fuel-python)
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python (Deprecated) (fuel-python) → Fuel Sustaining (fuel-sustaining-team)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Mikhail Zhnichkov (mzhnichkov)
Revision history for this message
Mikhail Zhnichkov (mzhnichkov) wrote :

I tried to change an order in templates without any results. But, changing of order in configuration file '/etc/cloud/cloud.cfg' make module 'write-files' to start after 'runcmd'.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/368076

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/368076
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=52c4cfd9372a24c5fbad4277f96972973a684f96
Submitter: Jenkins
Branch: master

commit 52c4cfd9372a24c5fbad4277f96972973a684f96
Author: Mikhail <email address hidden>
Date: Fri Sep 9 18:10:09 2016 +0300

    Change an order of executing cloud-init modules

    Module 'write-files' should be executed after module 'runcmd'

    Change-Id: I93360afb05ab739fe3f44b187db41054673caa0e
    Closes-Bug: #1543063

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-agent 10.0.0rc1

This issue was fixed in the openstack/fuel-agent 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-agent 10.0.0

This issue was fixed in the openstack/fuel-agent 10.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.