Fuel for OpenStack

[Ceph] Do not restart Ceph service on new Ceph OSD node deployment

Bug #1479880 reported by Miroslav Anashkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Bogdan Dobrelya
Fuel for OpenStack 7.0
5.1.x
Invalid
High
MOS Maintenance
Fuel for OpenStack 5.1.1-updates
6.0.x
Invalid
High
MOS Maintenance
Fuel for OpenStack 6.0-updates

Bug Description

The following improvement greatly increased the OSD deployment robustness
https://github.com/stackforge/fuel-library/commit/c20b2f8463e962734222382a77f26b91a53fd264

However, there is still one issue:
https://github.com/stackforge/fuel-library/blob/c20b2f8463e962734222382a77f26b91a53fd264/deployment/puppet/ceph/manifests/init.pp#L136

`Class['ceph::conf'] -> Class['ceph::osds'] ~> Service['ceph']`

`Class ceph::osds::osd` does both for each OSD , `ceph deploy osd prepare` and `ceph deploy osd activate`.

After that, Puppet code sends notification to Service['ceph']
As a result, Service['ceph'] is being restarted, followed by all deployed OSD restart on the entire node.
This step is not necessary, since all the related Ceph OSD services has been already started by `ceph deploy osd activate`.
It only creates huge load (especially in cases when OSD is added to long-existing cluster under the load) and may lead to unstable errors when particular OSD services fail to start.

In order to remove this service restart please change the problematic code line as following:
Class['ceph::conf'] -> Class['ceph::osds'] -> Service['ceph']

Miroslav Anashkin (manashkin)
Changed in fuel:
importance: Undecided → High
milestone: none → 7.0
assignee: nobody → Fuel Library Team (fuel-library)
Bogdan Dobrelya (bogdando)
Changed in fuel:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/207761

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
status: Triaged → In Progress
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

We don't support master node updates in 5.1/5.1.1 and 6.0 maintenance updates. Setting to Invalid for 5.1.1 and 6.0 updates.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/207761
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=141f6f7ed7039644b5e451cc521f2fc492ae6e81
Submitter: Jenkins
Branch: master

commit 141f6f7ed7039644b5e451cc521f2fc492ae6e81
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Jul 31 09:47:37 2015 +0200

    Restart ceph service only on conf change

    W/o this patch, puppet sends notification
    to Service['ceph'] unconditionally. As a
    result, Service['ceph'] is being restarted,
    followed by all deployed OSD restart on the
    entire node.

    This is a problem since all the related
    Ceph OSD services has been already started
    by `ceph deploy osd activate` and this notify
    with restart only creates huge load
    (especially in cases when OSD is added to
    long-existing cluster under the load) and may
    lead to unstable errors when particular OSD
    services fail to start.

    The solution is to trigger ceph service restart
    only on configuration updates.

    Closes-bug: #1479880

    Change-Id: Ib60436d50379041a6a1368b0fa170b9b58cc484c
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Oleksiy Molchanov (omolchanov)
tags: added: on-verification
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Verified

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "284"
  build_id: "284"
  nailgun_sha: "5c33995a2e6d9b1b8cdddfa2630689da5084506f"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "8283dc2932c24caab852ae9de15f94605cc350c6"
  fuel-library_sha: "f81fdabe6c05be7a3d11d88a7c3a8f3931921c73"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"

tags: removed: on-verification
Changed in fuel:
status: Fix Committed → Fix Released
Roman Rufanov (rrufanov)
tags: added: customer-found support
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.