Fuel for OpenStack

Generating a diagnostic snapshot triggers to an error "exit code:1 stderr:"

Bug #1543491 reported by Sergii Turivnyi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Georgy Kibardin
Fuel for OpenStack 10.0
Mitaka
Fix Released
High
Georgy Kibardin
Fuel for OpenStack 9.0

Bug Description

Steps to reproduce:
Deploy environment 3 controllers, 2 ironic+ceph
Active use it for 2 weeks
Generate Diagnostic snapshot

Expected result:
Snapshot was generated successfully

Actual result:
Generating a diagnostic snapshot triggers to an error "exit code:1 stderr:" (see attachment)

Logs:

[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 8,2G 859M 91% /var
/dev/mapper/os-varlog 101G 8,7G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 8,8G 223M 98% /var
/dev/mapper/os-varlog 101G 8,8G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 8,9G 86M 100% /var
/dev/mapper/os-varlog 101G 8,8G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9,5G 7,5G 1,6G 83% /
devtmpfs 7,8G 0 7,8G 0% /dev
tmpfs 7,8G 0 7,8G 0% /dev/shm
tmpfs 7,8G 202M 7,6G 3% /run
tmpfs 7,8G 0 7,8G 0% /sys/fs/cgroup
/dev/vda3 197M 108M 90M 55% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 9,5G 9,0G 37M 100% /var
/dev/mapper/os-varlog 101G 8,8G 87G 10% /var/log
tmpfs 1,6G 0 1,6G 0% /run/user/0
[root@fuel ~]#

Possible way to fix: save logs to /dev/mapper/os-varlog

See original description
Revision history for this message
Sergii Turivnyi (sturivnyi) wrote :
Sergii Turivnyi (sturivnyi)
summary: - Generating a diagnostic snapshot triggers to an error "exit cone:1
+ Generating a diagnostic snapshot triggers to an error "exit code:1
stderr:"
description: updated
Ilya Kutukov (ikutukov)
Changed in fuel:
assignee: nobody → Fuel Python Team (fuel-python)
milestone: none → 9.0
status: New → Confirmed
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Sergey, what Fuel version did you use?

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Sergii Turivnyi (sturivnyi) wrote :

{u'release': u'8.0', u'api': u'1', u'openstack_version': u'liberty-8.0', u'feature_groups': [u'mirantis'], u'auth_required': True}

Dmitry Pyzhov (dpyzhov)
tags: added: area-python
Changed in fuel:
importance: Critical → High
status: Incomplete → Confirmed
Dmitry Pyzhov (dpyzhov)
tags: added: module-shotgun
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Georgy Kibardin (gkibardin)
Fuel Devops McRobotson (fuel-devops-robot)
Changed in fuel:
milestone: 9.0 → 10.0
Georgy Kibardin (gkibardin)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Georgy Kibardin (gkibardin) wrote :

As usual there are to solutions: fast to implement and more optimal from storage space handling point of view.
The first, a fastest solution would be just using big enough partition for the whole /var (not /var/log - not sure what practical outcome this decision brings)
The second, more optimal solution, would be to compress snapshot data on nodes and then "tar" on the master node when necessary on the fly during downloading it via Fuel UI. This would free a lot of space required to store uncompressed data.

Revision history for this message
Georgy Kibardin (gkibardin) wrote :

The second part can be done in bounds of https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/310449

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/310452

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Georgy Kibardin (<email address hidden>) on branch: master
Review: https://review.openstack.org/310452

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/310449
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=410e0b9c23020aaf977e699d2d643036193092b7
Submitter: Jenkins
Branch: master

commit 410e0b9c23020aaf977e699d2d643036193092b7
Author: Georgy Kibardin <email address hidden>
Date: Wed Apr 27 15:37:01 2016 +0300

    Don't make a copy of master node logs

    Master node host creds are injected into shotgun config making it
    connect to localhost via ssh and make a copy of logs in /var/dump.
    Without creds, which aren't necessary anymore since we don't use
    containers, it just makes a symlink saving a lot of space.
    Also fixes incorrect excludes.

    Change-Id: I2e1a60bd71955b2c607f46d2cde60daa49811be7
    Partial-Bug: #1543491

Revision history for this message
Georgy Kibardin (gkibardin) wrote :

Since we are going to deprecate shotgun we've decided not to fix space issue. At least, because it is not shotgun issue.
The fix only improves diagnostic so that we know what the problem is.

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/322756

Sofiia Andriichenko (sandriichenko)
tags: added: on-verification
Revision history for this message
Sofiia Andriichenko (sandriichenko) wrote :

[root@nailgun ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 2.0G 7.1G 22% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 183M 1.7G 10% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vda3 197M 129M 68M 66% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 20G 3.8G 16G 20% /var
/dev/mapper/os-varlog 31G 1.9G 27G 7% /var/log
tmpfs 380M 0 380M 0% /run/user/0
tmpfs 380M 0 380M 0% /run/user/493
[root@nailgun ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.5G 2.0G 7.1G 22% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 183M 1.7G 10% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vda3 197M 129M 68M 66% /boot
/dev/vda2 200M 0 200M 0% /boot/efi
/dev/mapper/os-var 20G 5.9G 14G 31% /var
/dev/mapper/os-varlog 31G 1.9G 27G 7% /var/log
tmpfs 380M 0 380M 0% /run/user/0
tmpfs 380M 0 380M 0% /run/user/493

verified on:
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 465
cat /etc/fuel_build_number:
 465
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8454.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8454.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch
 fuel-notify-9.0.0-1.mos8454.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8454.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8454.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos935.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.