Fuel for OpenStack

Endless restart of swift-object-reconstructor and swift-container-reconciler

Bug #1587047 reported by Ilya Chukhnakov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Oleksiy Molchanov
Fuel for OpenStack 10.0
Mitaka
Fix Released
High
Oleksiy Molchanov
Fuel for OpenStack 9.1

Bug Description

Detailed bug description:
The swift-object-reconstructor and swift-container-reconciler services fail to start and restart endlessly after rebooting the cluster.

Steps to reproduce:
1. deploy a new environment from the Fuel WebUI
2. after all nodes are up and services have settled, shut down the environment and the fuel master and then start it again (https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#howto-shut-down-the-whole-cluster)
3. check dmesg for the following repeating messages on the controller nodes:
   swift-object-reconstructor main process (5143) terminated with status 1
   swift-object-reconstructor main process ended, respawning
   swift-container-reconciler main process (8570) terminated with status 1
   swift-container-reconciler main process ended, respawning

Expected results:
container-reconciler and object-reconstructor either start successfully
or do not start at all.

Actual result:
The swift-object-reconstructor and swift-container-reconciler services fail to start and are being restarted indefinitely.

When started manually, produce the following errors:
swift@node-1$ /usr/bin/python /usr/bin/swift-container-reconciler /etc/swift/container-server.conf
Unable to find container-reconciler config section in /etc/swift/container-server.conf

swift@node-1$ /usr/bin/python /usr/bin/swift-object-reconstructor /etc/swift/object-server.conf
Unable to find object-reconstructor config section in /etc/swift/object-server.conf

Workaround:
echo '[container-reconciler]' >> /etc/swift/container-server.conf
echo '[object-reconstructor]' >> /etc/swift/object-server.conf

Impact:
1. container-reconciler and object-reconstructor are not working
2. increased CPU consumption due to OS trying to restart services

Description of the environment:
MOS 9.0 build 372
or
MOS 9.0-mos iso 475

OpenStack Release: Mitaka on Ubuntu 14.04
Compute: QEMU
Network: Neutron with tunneling segmentation
Storage Backends: Cinder LVM over iSCSI for volumes
No additional projects/plugins

Total Nodes: 5
Controller + Cinder: 3
Compute: 2

See original description
Revision history for this message
Ilya Chukhnakov (ichukhnakov) wrote :
Oleksiy Molchanov (omolchanov)
Changed in fuel:
assignee: nobody → Oleksiy Molchanov (omolchanov)
importance: Undecided → High
status: New → Confirmed
tags: added: area-library
Maksim Malchuk (mmalchuk)
no longer affects: fuel/newton
tags: added: team-bugfix
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

I was not able to reproduce your case. These services are enabled on boot, but actually are stopped after boot. Seems that the situation you described happens only if you run 'service $service_name start'

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Ilya Chukhnakov (ichukhnakov) wrote :

I've successfully reproduced this issue on a more recent fuel-9.0-mos-475-2016-06-12_06-18-00.iso with a simple 1 controller + 1 compute environment.

These services indeed do not start on a freshly deployed environment, but they do appear once you 'poweroff' the nodes and start them again.

Ilya Chukhnakov (ichukhnakov)
description: updated
Changed in fuel:
status: Invalid → Confirmed
Ilya Chukhnakov (ichukhnakov)
description: updated
Oleksiy Molchanov (omolchanov)
Changed in fuel:
status: Confirmed → In Progress
Oleksiy Molchanov (omolchanov)
Changed in puppet-swift:
assignee: nobody → Oleksiy Molchanov (omolchanov)
status: New → In Progress
no longer affects: puppet-swift
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/332152

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/332152
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=cd059f1cc27e18fb04058567f7499f130d84de14
Submitter: Jenkins
Branch: master

commit cd059f1cc27e18fb04058567f7499f130d84de14
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Jun 21 16:49:04 2016 +0300

    Disable unneeded swift services

    This patch totally disable swift-object-reconstructor and
    swift-container-reconciler as they are not used, but are
    enabled by default. After reboot they are trying to start
    in a loop, consuming resources, but fail due to missing
    config-files.

    Change-Id: I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3
    Closes-Bug: 1587047

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/340261

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/340261
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0fc960be9dd8595f72b99aaaa7693567386feffa
Submitter: Jenkins
Branch: stable/mitaka

commit 0fc960be9dd8595f72b99aaaa7693567386feffa
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Jun 21 16:49:04 2016 +0300

    Disable unneeded swift services

    This patch totally disable swift-object-reconstructor and
    swift-container-reconciler as they are not used, but are
    enabled by default. After reboot they are trying to start
    in a loop, consuming resources, but fail due to missing
    config-files.

    Change-Id: I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3
    Closes-Bug: 1587047
    (cherry picked from commit cd059f1cc27e18fb04058567f7499f130d84de14)

Tatyana Kuterina (tkuterina)
tags: added: on-verification
Revision history for this message
Tatyana Kuterina (tkuterina) wrote :

Verified on 9.1 snapshot #76

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-library9.0-9.0.0-1.mos8495.noarch
 rubygem-astute-9.0.0-1.mos753.noarch
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8748.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-openstack-metadata-9.0.0-1.mos8748.noarch
 python-packetary-9.0.0-1.mos142.noarch
 nailgun-mcagents-9.0.0-1.mos753.noarch
 fuel-utils-9.0.0-1.mos8495.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-misc-9.0.0-1.mos8495.noarch
 fuel-ostf-9.0.0-1.mos938.noarch
 fuel-notify-9.0.0-1.mos8495.noarch
 fuel-nailgun-9.0.0-1.mos8748.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-mirror-9.0.0-1.mos142.noarch
 fuel-migrate-9.0.0-1.mos8495.noarch

FUEL_QA_COMMIT=bfb750898b0f5ef196eb0c8a295cc29279487ade
UBUNTU_MIRROR_ID=ubuntu-2016-07-30-170657
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-07-29-200321
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-07-29-200321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/371945

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/371983

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/371983
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c126e6bbae508a1acf776afff80cf2de5297f442
Submitter: Jenkins
Branch: master

commit c126e6bbae508a1acf776afff80cf2de5297f442
Author: Maksim Malchuk <email address hidden>
Date: Sat Sep 17 18:09:57 2016 +0300

    Don't stop non-existent services for UCA in Trusty

    Workaround the fix added by LP#1587047 [1] because UCA version of the
    'swift-object' and 'swift-container' packages in Trusty doen't have
    the corresponding services.

    [1] I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3

    Change-Id: I37745c752b432d00d89c3068873dfe6513ddd2bc
    Closes-Bug: #1610947
    Related-Bug: #1587047
    Signed-off-by: Maksim Malchuk <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/371945
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=26e41ea93f880e4a3d565a3e9557299cc7daab70
Submitter: Jenkins
Branch: stable/mitaka

commit 26e41ea93f880e4a3d565a3e9557299cc7daab70
Author: Maksim Malchuk <email address hidden>
Date: Sat Sep 17 18:09:57 2016 +0300

    Don't stop non-existent services for UCA in Trusty

    Workaround the fix added by LP#1587047 [1] because UCA version of the
    'swift-object' and 'swift-container' packages in Trusty doen't have
    the corresponding services.

    [1] I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3

    Change-Id: I37745c752b432d00d89c3068873dfe6513ddd2bc
    Closes-Bug: #1610947
    Related-Bug: #1587047
    Signed-off-by: Maksim Malchuk <email address hidden>

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.