Fuel for OpenStack

Some OSTF failed if Fill root filesystem on primary controller

Bug #1489497 reported by Dmitry Tyzhnenko on 2015-08-27

This bug report is a duplicate of: Bug #1493520: OpenStack is down after filling the root filesystem. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Confirmed	High	Unassigned	Fuel for OpenStack 7.0

Bug Description

After corrupt root filesystem on primary controller OSTF test Launch instance with file injection failed

Scenario:
1. Create new environment
2. Choose Neutron, TUN
3. Choose Sahara
4. Add 3 controller
5. Add 1 compute
6. Add 1 cinder
7. Verify networks
8. Deploy the environment
9. Verify networks
10. Run OSTF tests
11. Corrupt root filesystem on primary controller. For example, remount as r/o
12. Wait for ~30 min
13. Verify networks
14. Run OSTF tests

Expected rusult:
All step pass

Actual result:
step 14 failed - Launch instance with file injection was failed with Execution command on Instance fails with unexpected result error

Fuel iso 7.0-229

{
    'build_id': '2015-08-25_09-19-20',
    'build_number': '229',
    'release_versions': {'2015.1.0-7.0': {'VERSION': {
        'build_id': '2015-08-25_09-19-20',
        'build_number': '229',
        'api': '1.0',
        'fuel-library_sha': 'f1443266fa7a32eaf8ccc131677389079339bb43',
        'nailgun_sha': '87a569c06ecd478cae523bd437d4e0af8014dd5b',
        'feature_groups': ['mirantis'],
        'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
        'openstack_version': '2015.1.0-7.0',
        'fuel-agent_sha': '4c2ab9d6c623d345086c6e2874d1df81fd96a942',
        'production': 'docker',
        'python-fuelclient_sha': 'fc7b63aa6900fe3b2c183108ba6a13e868bc0472',
        'astute_sha': '53c86cba593ddbac776ce5a3360240274c20738c',
        'fuel-ostf_sha': '3ad03d076c46347691cc3480dd19d34e37b73df4',
        'release': '7.0',
        'fuelmain_sha': '28d4bfcff7a0fb1b37504dbcac4998789df17935',
        }}},
    'auth_required': true,
    'api': '1.0',
    'fuel-library_sha': 'f1443266fa7a32eaf8ccc131677389079339bb43',
    'nailgun_sha': '87a569c06ecd478cae523bd437d4e0af8014dd5b',
    'feature_groups': ['mirantis'],
    'fuel-nailgun-agent_sha': 'e01693992d7a0304d926b922b43f3b747c35964c',
    'openstack_version': '2015.1.0-7.0',
    'fuel-agent_sha': '4c2ab9d6c623d345086c6e2874d1df81fd96a942',
    'production': 'docker',
    'python-fuelclient_sha': 'fc7b63aa6900fe3b2c183108ba6a13e868bc0472',
    'astute_sha': '53c86cba593ddbac776ce5a3360240274c20738c',
    'fuel-ostf_sha': '3ad03d076c46347691cc3480dd19d34e37b73df4',
    'release': '7.0',
    'fuelmain_sha': '28d4bfcff7a0fb1b37504dbcac4998789df17935',
}

logs snapshot - env T1722197 (id:2) - https://drive.google.com/a/mirantis.com/file/d/0B8U7EvTbuAOlWmVpVHNwMTk4M3c/view?usp=sharing

Tags:

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-08-28:

May be a dup of https://bugs.launchpad.net/fuel/+bug/1484121

tags:

added: ha

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-08-28:

In the snapshot attached the OSTF test was passed
2015-08-27 13:20:24 SUCCESS Launch instance with file injection (fuel_health.tests.smoke.test_nova_create_instance_with_connectivity.TestNovaNetwork.test_009_create_server_with_file)

Changed in fuel:
status:	New → Incomplete

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-08-28:

Sorry, wrong cluster ID :)

Changed in fuel:
status:	Incomplete → New

Revision history for this message

Dmitry Tyzhnenko (dtyzhnenko) wrote on 2015-08-28:

It seems this bug is not a duplicate of https://bugs.launchpad.net/fuel/+bug/1484121 because bugs have different scenarios. In 1484121 primary controller destroyed, but in this bug, primary controller alive with the filesystem which remounted to read-only state.

Nastya Urlapova (aurlapova) on 2015-08-28

Changed in fuel:
status:	New → Confirmed

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2015-08-28:

I'm not sure this bug is valid as we have added in errors=panic to the controllers to ensure that if the filesystem corrupts they will reboot themselves. The only way the root volume is going to go read-only is via manual intervention which we cannot control or do anything about. If you do this test but zero out the root file system is this still a problem?

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-31: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/218756

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Michael Polenchuk (mpolenchuk)
status:	Confirmed → In Progress

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-08-31: Re: OSTF test Launch instance with file injection was failed with Execution command on Instance fails with unexpected result error

I agree with @Alex, the test case seems had been obsoleted after the errors=panic change introduced for nodes

Changed in fuel:
status:	In Progress → Invalid
tags:	added: to-be-covered-by-tests

Michael Polenchuk (mpolenchuk) on 2015-08-31

Changed in fuel:
assignee:	Michael Polenchuk (mpolenchuk) → nobody

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-01: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/218756
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5cc9abf2502055a1cd2e70d04598803251813fd1
Submitter: Jenkins
Branch: master

commit 5cc9abf2502055a1cd2e70d04598803251813fd1
Author: Michael Polenchuk <email address hidden>
Date: Mon Aug 31 10:55:40 2015 +0300

Align nova kombu reconnect delay with neutron

Align nova(compute/ctrl) services kombu reconnect delay option with
neutron services in order to mitigate a reconnecting to amqp server.

    Change-Id: I8e3fd877a9db4a761fbe82def440c2da86800fe1
    Upstream-ChangeId: Iceb43c27f6476e023cb816ed1f2eda998f1165f3
    Partial-Bug: #1489497
    Partial-Bug: #1484121

Artem Hrechanychenko (agrechanichenko) on 2015-09-21

summary:

- OSTF test Launch instance with file injection was failed with Execution
- command on Instance fails with unexpected result error
+ Some OSTF failed if Fill root filesystem on primary controller

Revision history for this message

Artem Hrechanychenko (agrechanichenko) wrote on 2015-09-21:

Reproduced on baremetall lab with 298 ISO according to https://mirantis.testrail.com/index.php?/tests/view/1859021&group_by=cases:section_id&group_order=asc&group_id=6292

List of failed OSTF test:

OSTF tests - Request instance list
Instance list is unavailable. Please refer to OpenStack logs for more details.

Check that required services are running
Some nova services have not been started.. Please refer to OpenStack logs for more details.

Create volume and boot instance from it
Volume was not created Please refer to OpenStack logs for more details.

Create volume and attach it to instance
Volume was not created. Please refer to OpenStack logs for more details.

Check network connectivity from instance via floating IP
Failed to create rule in security group. Please refer to OpenStack logs for more details.

Create keypair
Keypair can not be created. Please refer to OpenStack logs for more details

Create security group
Failed to create rule in security group. Please refer to OpenStack logs for more details.

Launch instance
Failed to create rule in security group. Please refer to OpenStack logs for more details.

Launch instance with file injection
Failed to create rule in security group. Please refer to OpenStack logs for more details.

Launch instance, create snapshot, launch instance from snapshot
Image can not be booted. Please refer to OpenStack logs for more details.

RabbitMQ replication
Failed to check message replication! Please refer to OpenStack logs for more details.

Typical stack actions: create, delete, show details, etc. failed without reason of warning message

Check disk space outage on controller and compute nodes
Nearly disk outage state detected on host(s): [u'172.16.39.131']. Please refer to OpenStack logs for more details.

Steps to reproduce

1)Deploy any environment with 3 controllers and NeutronTUN or NeutronVLAN
2) dd if=/dev/zero of=/root/file bs=32KB
3)Wait for ~5 min
4) Verify networks
5) Run OSTF tests

Actual results - some of OSTF test are failed. List described bellow.

After delete big file in fill root partition - all OSTF test passed.