Fuel for OpenStack

fuel-agent installs Ubuntu package instead of MOS ones

Bug #1488844 reported by Anastasia Palkina
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vladimir Kozhukalov
Fuel for OpenStack 7.0
6.1.x
Invalid
Critical
Sergii Rizvan
Fuel for OpenStack 6.1-updates

Bug Description

1. Create new environment
2. Choose Neutron, VLAN
3. Choose Ceph for volumes
4. Choose Murano
5. Add 3 controller, 1 compute, 3 ceph
6. Start deployment
7. Stop deployment during provisioning
8. Wait until nodes become 'Pending addition'
9. Deploy the environment again. It was successful
10. But controller (node-3) has gone away and back online many times

The default set of APT repositories is used. So Ubuntu package installed instead special package for ohai.

root@node-3:~# apt-cache policy ohai
ohai:
  Installed: 6.14.0-2
  Candidate: 6.14.0-2~u14.04+mos1
  Version table:
 *** 6.14.0-2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status
     6.14.0-2~u14.04+mos1 0
       1050 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages
root@node-3:~#

"build_id": "2015-08-25_03-34-40", "build_number": "228", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-25_03-34-40", "build_number": "228", "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"

See original description
Anastasia Palkina (apalkina)
description: updated
Alexei Sheplyakov (asheplyakov)
Changed in fuel:
assignee: nobody → Alexei Sheplyakov (asheplyakov)
status: New → In Progress
Anastasia Palkina (apalkina)
description: updated
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

Apparently fuel-agent installs a wrong version of ohai into the IBP image:

[root@fuel ~]# zcat /var/www/nailgun/targetimages/env_1_ubuntu_1404_amd64.img.gz > /tmp/env1_root.ext4
[root@fuel ~]# mount -t ext4 -o loop,rw,data=ordered,errors=remount-ro /tmp/env1_root.ext4 /mnt
[root@fuel ~]# chroot /mnt apt-cache policy ohai
ohai:
  Installed: 6.14.0-2
  Candidate: 6.14.0-2
  Version table:
 *** 6.14.0-2 0
        100 /var/lib/dpkg/status

summary: - Ubuntu package installed instead special package for ohai
+ fuel-agent installs Ubuntu package instead of MOS ones
Changed in fuel:
assignee: Alexei Sheplyakov (asheplyakov) → Fuel Python Team (fuel-python)
status: In Progress → Confirmed
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

It's fuel-agent bug => reassigning to fuel-python team

Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fuel build team (fuel-build)
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

Apparently fuel-agent creates bogus pinning rules which does not match packages from the 'universe' section:

# cat /etc/apt/preferences.d/fuel-image-mos.pref
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0, c=main
Pin-Priority: 1050
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0, c=restricted
Pin-Priority: 1050

These rules does not match packages residing in the `universe' section (which
is where the `ohai' package shipped with Ubuntu belongs to). As a result the original
version of `ohai' package gets installed (which is wrong and re-introduces LP #1463835):

# apt-cache policy ohai
ohai:
  Installed: (none)
  Candidate: 6.14.0-2
  Version table:
     6.14.0-2 0
        500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64 Packages
     6.14.0-2~u14.04+mos1 0
        500 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages

Adjusting the pinning rules fixes the problem:

# cat > /etc/apt/preferences.d/fuel-image-mos.pref <<-EOF
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0
Pin-Priority: 1050
EOF

# apt-cache policy ohai
ohai:
  Installed: (none)
  Candidate: 6.14.0-2~u14.04+mos1
  Version table:
     6.14.0-2 0
        500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64 Packages
     6.14.0-2~u14.04+mos1 0
       1050 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/217108

Changed in fuel:
assignee: Fuel build team (fuel-build) → Alexei Sheplyakov (asheplyakov)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/217112

Changed in fuel:
assignee: Alexei Sheplyakov (asheplyakov) → Vladimir Kozhukalov (kozhukalov)
assignee: Vladimir Kozhukalov (kozhukalov) → Alexei Sheplyakov (asheplyakov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Alexei Sheplyakov (asheplyakov) → Vladimir Kozhukalov (kozhukalov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/217112
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3d18e9e2519c22a9a6404580a64c59c1d86faf21
Submitter: Jenkins
Branch: master

commit 3d18e9e2519c22a9a6404580a64c59c1d86faf21
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Aug 26 15:28:07 2015 +0300

    Fix pinning rules so a package from main can override the one from
    universe

    Current pinning rules don't match packages from the `universe' section.
    As a result a customized MOS package (which always resides in the `main'
    section) has lower priority than its Ubuntu counterpart:

     # apt-cache policy ohai
     ohai:
       Installed: (none)
       Candidate: 6.14.0-2
       Version table:
          6.14.0-2 0
             500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64
    Packages
          6.14.0-2~u14.04+mos1 0
             500 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/
    mos7.0/main amd64 Packages

    As a result the original version of package (shipped with Ubuntu) gets
    installed into the IBP image which might reintroduce bugs fixed by
    patched packages and/or break features which require customized
    packages.

    Change-Id: I9326849751a8616fcdd4e3a57297bbb0f49def50
    Co-Authored-By: Alexey Sheplyakov <email address hidden>
    Closes-Bug: #1488844

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/217108
Committed: https://git.openstack.org/cgit/stackforge/fuel-agent/commit/?id=bf324e8f3981b050e89f710ab64411ef0d60ff04
Submitter: Jenkins
Branch: master

commit bf324e8f3981b050e89f710ab64411ef0d60ff04
Author: Alexey Sheplyakov <email address hidden>
Date: Wed Aug 26 14:48:23 2015 +0300

    Fix pinning rules so a package from main can override the one from universe

    Current pinning rules don't match packages from the `universe' section.
    As a result a customized MOS package (which always resides in the `main'
    section) has lower priority than its Ubuntu counterpart:

     # apt-cache policy ohai
     ohai:
       Installed: (none)
       Candidate: 6.14.0-2
       Version table:
          6.14.0-2 0
             500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64 Packages
          6.14.0-2~u14.04+mos1 0
             500 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages

    As a result the original version of package (shipped with Ubuntu) gets
    installed into the IBP image which might reintroduce bugs fixed by
    patched packages and/or break features which require customized
    packages.

    Closes-Bug: #1488844
    Change-Id: I655b34bf0de80d26ba3f8c25fbb6b17bbaefc076

Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 7.0-246

{"build_id": "2015-08-27_18-45-15", "build_number": "246", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-27_18-45-15", "build_number": "246", "api": "1.0", "fuel-library_sha": "800b28137e131334d447c353db213b1d43f9bedb", "nailgun_sha": "cbe7dcab16fbdacaa8e80e36a925191e7986a068", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "8844dc5964fb197b64525f04b6ec15cbbcfe44f0", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "83048d68609854324ceeaf04242e68d658cfb55d", "release": "7.0", "fuelmain_sha": "8ead1228c637ba80e7dead37dfe09d85f00b7c29"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "800b28137e131334d447c353db213b1d43f9bedb", "nailgun_sha": "cbe7dcab16fbdacaa8e80e36a925191e7986a068", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "8844dc5964fb197b64525f04b6ec15cbbcfe44f0", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "83048d68609854324ceeaf04242e68d658cfb55d", "release": "7.0", "fuelmain_sha": "8ead1228c637ba80e7dead37dfe09d85f00b7c29"}

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

Hello MOS-Maintenance team, please take a look at this critical bug. It looks like 6.1 is affected too.

Sergii Golovatiuk (sgolovatiuk)
tags: added: customer-found
Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

One can workaround the issue following the steps:
1) Once a node is provisioned, ssh on a node
2) edit /etc/apt/preferences.d/* files so they do not contain section based prefereces
I.e. instead of having many section based stanzas
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0, c=main
Pin-Priority: 1050
it should contain single stanza w/o section
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0
Pin-Priority: 1050
3) run apt-get dist-upgrade

Revision history for this message
Polina Petriuk (ppetriuk) wrote :

Would the workaround for MOS6.1 look similar?

Polina Petriuk (ppetriuk)
tags: added: support
Sergey Shevorakov (sshevorakov)
tags: added: to-rca
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/242526

Vadim Rovachev (vrovachev)
tags: added: on-verification
Revision history for this message
Sergii Rizvan (srizvan) wrote :

Bug was not reproduced on MOS 6.1. Close as invalid.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/6.1)

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/242526
Reason: Bug was not reproduced on MOS 6.1.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.