Fuel for OpenStack

Pacemaker fails whole deploy because of insufficient disk space on mongo partition

Bug #1572957 reported by Vyacheslav Struk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Maksim Malchuk
Fuel for OpenStack 10.0
8.0.x
Fix Released
High
Maksim Malchuk
Fuel for OpenStack 8.0-updates
Mitaka
Fix Released
High
Maksim Malchuk
Fuel for OpenStack 9.0

Bug Description

Fuel 8.0 has no minimal disk space checks for mongo(at least) logical volume (mounted to /var/lib/mongo)

Deploy fails on pacemaker which dismissing the controller node during deploy because of low disk space.
Default node-health-strategy is 'migrate-on-red' so the process is triggering by SysInfo health check resource.

How to reproduce:

1) Make an environment with a single controller, add mongo role to the controller, set mongo partition to 11 Gb.
2) Start deploy and wail till it fails.
Check it on the controller node:

# df -h
  Filesystem Size Used Avail Use% Mounted on
  ...
  /dev/mapper/mongo-mongodb 11G 11G 34M 100% /var/lib/mongo

# cibadmin -Q | grep health_disk
          <nvpair id="status-1-#health_disk" name="#health_disk" value="red"/>

# crm resource list
   Clone Set: clone_p_vrouter [p_vrouter]
       Stopped: [ node-1.domain.tld ]
   vip__management (ocf::fuel:ns_IPaddr2): Stopped
   vip__baremetal (ocf::fuel:ns_IPaddr2): Stopped
   vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Stopped
   vip__vrouter (ocf::fuel:ns_IPaddr2): Stopped
   vip__public (ocf::fuel:ns_IPaddr2): Stopped
   Master/Slave Set: master_p_conntrackd [p_conntrackd]
       Stopped: [ node-1.domain.tld ]
   sysinfo_node-1.domain.tld (ocf::pacemaker:SysInfo): Stopped
   Clone Set: clone_p_haproxy [p_haproxy]
       Stopped: [ node-1.domain.tld ]

============================

# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

Tags: area-library
Vyacheslav Struk (vstruk)
summary: - Pacemaker fails to deploy because of insufficient disk space for mongo
+ Pacemaker fails whole deploy because of insufficient disk space on mongo
partition
Dmitry Klenov (dklenov)
no longer affects: fuel/future
Changed in fuel:
milestone: none → 10.0
tags: added: area-library
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

The minimum size is determined in nailgun logic. It is 10gb
https://github.com/openstack/fuel-web/blob/stable/8.0/nailgun/nailgun/extensions/volume_manager/manager.py#L830

Perhaps the minimum size should be 20gb. Moving to fuel-python

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Maksim Malchuk (mmalchuk)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/309351

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/310599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/309351
Reason: correct fix: https://review.openstack.org/310599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/310599
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1bfdc6d08377f471f4e975f250d77179f8997bcd
Submitter: Jenkins
Branch: master

commit 1bfdc6d08377f471f4e975f250d77179f8997bcd
Author: Maksim Malchuk <email address hidden>
Date: Thu Apr 28 00:53:29 2016 +0300

    Use default size for the MongoDB replication operation log

    We should use default size for the replication operation log if the
    value not defined in Hiera. On 64-bit systems, the 'oplog' is
    typically 5% of available disk space.

    Change-Id: Iafd5bc518f59e018b0e37410927e0e6f635ff0db
    Closes-Bug: #1572957
    Signed-off-by: Maksim Malchuk <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310709

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/310719

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/310709
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=51ddf5f9d924686cf807f81b0eaae9e8b3090549
Submitter: Jenkins
Branch: stable/mitaka

commit 51ddf5f9d924686cf807f81b0eaae9e8b3090549
Author: Maksim Malchuk <email address hidden>
Date: Thu Apr 28 00:53:29 2016 +0300

    Use default size for the MongoDB replication operation log

    We should use default size for the replication operation log if the
    value not defined in Hiera. On 64-bit systems, the 'oplog' is
    typically 5% of available disk space.

    Change-Id: Iafd5bc518f59e018b0e37410927e0e6f635ff0db
    Closes-Bug: #1572957
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 1bfdc6d08377f471f4e975f250d77179f8997bcd)

Dmitriy Kruglov (dkruglov)
tags: added: on-verification
Revision history for this message
Dmitriy Kruglov (dkruglov) wrote :

Verified on MOS 9.0, ISO build 418.
The issue is fixed.

ISO details:
root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 418
cat /etc/fuel_build_number:
 418
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8398.noarch
 rubygem-astute-9.0.0-1.mos746.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8709.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8709.noarch
 fuel-notify-9.0.0-1.mos8398.noarch
 nailgun-mcagents-9.0.0-1.mos746.noarch
 python-fuelclient-9.0.0-1.mos316.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8398.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8398.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2706.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 fuel-misc-9.0.0-1.mos8398.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8709.noarch

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/8.0)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/8.0
Review: https://review.opendev.org/310719
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.