Fuel for OpenStack

[upgrade] Rsync docker container after Fuel upgrade uses old puppet manifests while generating xinetd configuration

Bug #1382531 reported by Artem Panchenko on 2014-10-17

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Committed	Low	Matthew Mosesohn	Fuel for OpenStack 6.0
5.1.x	Fix Committed	High	Matthew Mosesohn	Fuel for OpenStack 5.1.1
6.0.x	Fix Committed	Low	Matthew Mosesohn	Fuel for OpenStack 6.0

Bug Description

We have a bug in Fuel 5.0.x https://bugs.launchpad.net/fuel/+bug/1322577 which is fixed in 5.1. But after Fuel upgrade from 5.0.1 to 5.1 the issue isn't gone because new 'rsync' container still has old configuration for xinetd:

[root@fuel-lab-cz5558 ~]# dockerctl shell rsync cat /etc/xinetd.d/rsync
service rsync
{
        disable = no
        socket_type = stream
        wait = no
        user = root
        server = /usr/bin/rsync
        server_args = --daemon
        log_on_failure += USERID
}

This happens becuase the folder with puppet manifests inside 'rsync' container is mounted from host system and '/etc/puppet/modules' directory contains old (5.0.1 in my case) files:

[root@fuel-lab-cz5558 ~]# dockerctl list -l | grep rsync
rsync fuel/rsync_5.1 Running fuel-core-5.1-rsync

[root@fuel-lab-cz5558 ~]# dockerctl shell rsync grep puppet /proc/mounts
/dev/mapper/os-root /etc/puppet ext4 ro,relatime,barrier=1,stripe=128,data=ordered 0 0

[root@fuel-lab-cz5558 ~]# ll /etc/puppet/
total 28
drwxr-xr-x 4 root root 4096 вер 17 18:29 2014.1.1-5.0.2
drwxr-xr-x 4 root root 4096 вер 17 18:28 2014.1.1-5.1
-rw-r--r--. 1 root root 4133 січ 6 2014 auth.conf
-rw-r--r--. 1 root root 0 жов 16 08:41 hiera.yaml
drwxr-xr-x. 2 root root 4096 жов 16 08:41 manifests
drwxr-xr-x. 62 root root 4096 жов 16 08:41 modules
-rw-r--r--. 1 root root 853 січ 6 2014 puppet.conf

This issue can be workarounded in the following way:

[root@fuel-lab-cz5558 ~]# dockerctl shell rsync sed -e 's/puppet\/modules/puppet\/2014.1.1-5.1\/modules/' -e 's/apply/apply --modulepath \/etc\/puppet\/2014.1.1-5.1\/modules/' -i /usr/local/bin/start.sh
[root@fuel-lab-cz5558 ~]# dockerctl restart rsync

Tags:

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2014-10-17:

This only affects 5.1 because 6.0 uses host manifests directly and no longer relies on the puppet container.
This should be fixed in the fuel-upgrade scripts because all containers read /etc/puppet/modules and /etc/puppet/manifests directly. It's possible there was a bug in the upgrade here and the symlink wasn't set up right.

no longer affects:

fuel/6.0.x

Nastya Urlapova (aurlapova) on 2014-10-17

Changed in fuel:
milestone:	none → 6.0

Bogdan Dobrelya (bogdando) on 2014-10-23

Changed in fuel:
assignee:	nobody → Fuel Library Team (fuel-library)
status:	New → Triaged
milestone:	6.0 → none

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-10-23:

According to #1, This should be fixed in the fuel-upgrade scripts

Changed in fuel:
milestone:	none → 5.1.1
no longer affects:	fuel/5.1.x
Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Fuel Python Team (fuel-python)

Dmitry Pyzhov (dpyzhov) on 2014-11-07

Changed in fuel:
status:	Triaged → Confirmed

Revision history for this message

Evgeniy L (rustyrobot) wrote on 2014-11-07:

>> This should be fixed in the fuel-upgrade scripts because all containers read /etc/puppet/modules and /etc/puppet/manifests directly. It's possible there was a bug in the upgrade here and the symlink wasn't set up right.

We should not rewrite `/etc/puppet/modules` and `/etc/puppet/manifests` with some symlinks, because this manifests are used during the deployment of previous versions of environments.

What we can do is to make symlink to /etc/puppet/latest from the latest version of manifests, and fix start.sh in the container to run manifests and modules from this directory.

Revision history for this message

Evgeniy L (rustyrobot) wrote on 2014-11-07:

The right way to solve the problem is not to mix openstack manifests and manifests for fuel master node.
We should keep separate directories for them, it make sense because the release cycle for openstack release and master can be different.
And we can have problems if we don't make this separation.

Revision history for this message

Evgeniy L (rustyrobot) wrote on 2014-11-08:

Another way to solve the problem is to fix Dockerfile for rsync [1], not to copy manifests and modules in /etc/puppet, use another directory and fix start.sh to run manifests and modules from this new directory, in this case manifests/modules will not be overridden by manifests/modules from the host system. And container will always have original puppet scripts without this "symlink magic" which we will have to handle somehow in case of rollbacks.

[1] https://github.com/stackforge/fuel-main/blob/master/docker/rsync/Dockerfile

Changed in fuel:
assignee:	Fuel Python Team (fuel-python) → Fuel Library Team (fuel-library)

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2014-11-18:

/etc/puppet/modules is already a symlink. Why do you want to make this more complicated? It's not even clear how the /etc/puppet directory should look like before, during, and after host upgrade.

We can move to host puppet manifests for 5.1.1 if it can be passed as an exception and if it helps, but proposing /etc/puppet/modules-HOST_ONLY directory doesn't seem to gain any benefits. All it does is add extra lines of upgrade logic outside of fuel_upgrade which already manages this task.

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Evgeniy L (rustyrobot)

Revision history for this message

Evgeniy L (rustyrobot) wrote on 2014-11-18:

http://irclog.perlgeek.de/fuel-dev/2014-11-18#i_9679058

Changed in fuel:
assignee:	Evgeniy L (rustyrobot) → Fuel Library Team (fuel-library)

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-11-18:

+1 to Evgeniy. The container MUST be self-sufficient as much as possible. That's from docker best practices and it makes sence, because "less external factors" means "less points of failures".

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-18: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/135295

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-20: Fix proposed to fuel-library (stable/5.1)

#10

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/135947

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-21: Fix merged to fuel-library (stable/5.1)

#11

Reviewed: https://review.openstack.org/135947
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=34dd49e31791da777faf088d313a23edd44124b3
Submitter: Jenkins
Branch: stable/5.1

commit 34dd49e31791da777faf088d313a23edd44124b3
Author: Matthew Mosesohn <email address hidden>
Date: Tue Nov 18 17:13:31 2014 +0400

Create dedicated puppet volume for rsync container

    New host volume /etc/puppet maps to rsync container /puppet,
    freeing up /etc/puppet:/etc/puppet mapping, which may be enabled
    for some desired testing function.
    This also has a desired benefit to use bundled manifests when
    deploying rsync container, but still serve arbitrary host puppet
    contents via rsync.

Change-Id: I6e7f29679a7d475a6c49dd6e3d498c14c943408f
Closes-Bug: #1382531

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-21: Fix merged to fuel-library (master)

#12

Reviewed: https://review.openstack.org/135295
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bfe13c7aa589b57cb584e85c08ec39ad0b39ec7b
Submitter: Jenkins
Branch: master

commit bfe13c7aa589b57cb584e85c08ec39ad0b39ec7b
Author: Matthew Mosesohn <email address hidden>
Date: Tue Nov 18 17:13:31 2014 +0400

Create dedicated puppet volume for rsync container

Change-Id: I6e7f29679a7d475a6c49dd6e3d498c14c943408f
Closes-Bug: #1382531

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2014-11-23:

#13

This commit was reverted and now it is blocked on either of these two bugs for a solution:
https://bugs.launchpad.net/fuel/+bug/1395047
https://bugs.launchpad.net/fuel/+bug/1395008

1395047 will provide a complete solution, but 1395008 is a partial workaround only.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-24: Related fix proposed to fuel-main (master)

#14

Related fix proposed to branch: master
Review: https://review.openstack.org/136725

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-24: Related fix proposed to fuel-main (stable/5.1)

#15

Related fix proposed to branch: stable/5.1
Review: https://review.openstack.org/136731

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-11-24:

#16

related https://bugs.launchpad.net/fuel/+bug/1395658

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-25: Related fix merged to fuel-main (master)

#17

Reviewed: https://review.openstack.org/136725
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=e3c385db9f5ae3b73117dc47238659a4d3bfac0b
Submitter: Jenkins
Branch: master

commit e3c385db9f5ae3b73117dc47238659a4d3bfac0b
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 24 12:58:28 2014 +0400

Remove puppet run from static containers

    Additional puppet runs have no impact on deployment
    of rsync and rsyslog containers because they do not
    change their configuration based on how a user deploys
    their environment.

    Change-Id: Ic5aac99620a8b6e24c5c5c2be9d246186df08a6c
    Closes-Bug: #1395658
    Related-Bug: #1382531

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-25: Related fix merged to fuel-main (stable/5.1)

#18

Reviewed: https://review.openstack.org/136731
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=ace6f48b84a2bf60898de7188688e2ee0acf259e
Submitter: Jenkins
Branch: stable/5.1

commit ace6f48b84a2bf60898de7188688e2ee0acf259e
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 24 12:58:28 2014 +0400

Remove puppet run from static containers

    Additional puppet runs have no impact on deployment
    of rsync and rsyslog containers because they do not
    change their configuration based on how a user deploys
    their environment.

    Change-Id: Ic5aac99620a8b6e24c5c5c2be9d246186df08a6c
    Closes-Bug: #1395658
    Related-Bug: #1382531

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Related blueprints

Fuel Master CI Tests

Remote bug watches

Bug watches keep track of this bug in other bug trackers.