nova-dpm

Failure of tempest test-case

Bug #1721730 reported by sreeteja
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nova-dpm
New
High
sreeteja

Bug Description

As there is only one tempest testcase that is suiting our environment
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern

This is failing sometimes due to multiple issues
1.Failed to establish ssh connection
2.Failed to bring server to ACTIVE state
3.Failed to delete volume

I tried all these manually they are working fine

I tried to change the TIMEOUT_SCALING_FACTOR in the code as well as some config parameter values like ready_wait, ssh_timeout,build_timeout ..etc but able to see the same failure errors each time..

Here is the config file :

[network]
default_network = 192.168.234.0/24
public_router_id =
public_network_id = c952d72a-cf42-4d2c-9550-c9d084e42e45
project_networks_reachable = false
project_network_cidr = 192.168.234.0/24

[DEFAULT]
debug = True
log_file = tempest.log
use_stderr = False
use_syslog = False

[oslo_concurrency]
lock_path = /opt/stack/data/tempest

[compute]
max_microversion = latest
flavor_ref_alt = 12345
flavor_ref = 12345
image_ref_alt = fcef1cfa-4b0b-4586-9529-a4fc759f5655
image_ref = fcef1cfa-4b0b-4586-9529-a4fc759f5655
build_timeout = 190
fixed_network_name = provider
create_isolated_networks = False
ready_wait = 100

[volume]
max_microversion = latest
build_timeout = 300
volume_size = 2

[identity]
auth_version = v3
admin_domain_scope = True
user_unique_last_password_count = 2
user_lockout_duration = 5
user_lockout_failure_attempts = 2
uri_v3 = http://9.152.150.201/identity/v3
uri = http://9.152.150.201/identity/v2.0/

[auth]
use_dynamic_credentials = True
tempest_roles = Member
admin_domain_name = Default
admin_project_name = admin
admin_password = openstack
admin_username = admin
create_isolated_networks = False

[identity-feature-enabled]
forbid_global_implied_dsr = True
security_compliance = True
api_v2_admin = False
api_v2 = True

[image-feature-enabled]
api_v1 = False
deactivate_image = True

[validation]
network_for_ssh = provider
image_ssh_user = root
ssh_timeout = 196
ip_version_for_ssh = 4
run_validation = True
connect_method = floating

[compute-feature-enabled]
vnc_console = True
attach_encrypted_volume = True
live_migrate_back_and_forth = False
block_migration_for_live_migration = False
change_password = False
live_migration = False
resize = True
personality = False

[network-feature-enabled]
api_extensions = address-scope,agent,allowed-address-pairs,auto-allocated-topology,availability_zone,binding,default-subnetpools,dhcp_agent_scheduler,dvr,ext-gw-mode,external-net,extra_dhcp_opt,extraroute,flavors,l3-flavors,l3-ha,l3_agent_scheduler,multi-provider,net-mtu,network-ip-availability,network_availability_zone,pagination,port-security,project-id,provider,quotas,rbac-policies,router,router_availability_zone,security-group,service-type,sorting,standard-attr-description,standard-attr-revisions,standard-attr-timestamp,subnet-service-types,subnet_allocation,tag,tag-ext
port_security = True
ipv6_subnet_attributes = True
ipv6 = True

[scenario]
img_file =
img_dir = /opt/stack/devstack/files

[volume-feature-enabled]
api_extensions = all
backup = False
api_v1 = False
api_v3 = True
manage_volume = False
manage_snapshot = True

[service_available]
trove = False
ironic = False
sahara = False
horizon = True
ceilometer = False
heat = False
swift = False
cinder = True
neutron = True
nova = True
glance = True
key = True

[object-storage-feature-enabled]
discoverable_apis = all

sreeteja (sreeteja)
Changed in nova-dpm:
importance: Undecided → Critical
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

To dive into this, we should collect the logs of the various participating components:
* dpm-driver
* zhmcclient
* HMC
* SE

HINT: If all logs use UTC-0 in their timestamps, that would be helpful.

Revision history for this message
sreeteja (sreeteja) wrote :
Download full text (5.2 KiB)

-ebf95baf5ca3'}
        Body:
2017-10-09 12:54:32,945 12405 INFO [tempest.lib.common.ssh] Creating ssh connection to '172.24.4.12:22' as 'root' with public key authentication
2017-10-09 12:54:36,257 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 1. Retry after 2 seconds.
2017-10-09 12:54:41,757 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 2. Retry after 3 seconds.
2017-10-09 12:54:48,257 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 3. Retry after 4 seconds.
2017-10-09 12:54:55,757 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 4. Retry after 5 seconds.
2017-10-09 12:55:04,257 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 5. Retry after 6 seconds.
2017-10-09 12:55:13,757 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 6. Retry after 7 seconds.
2017-10-09 12:55:24,257 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 7. Retry after 8 seconds.
2017-10-09 12:55:35,757 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 8. Retry after 9 seconds.
2017-10-09 12:55:48,267 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 9. Retry after 10 seconds.
2017-10-09 12:56:01,787 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 10. Retry after 11 seconds.
2017-10-09 12:56:16,297 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 11. Retry after 12 seconds.
2017-10-09 12:56:31,807 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172.24.4.12). Number attempts: 12. Retry after 13 seconds.
2017-10-09 12:56:48,317 12405 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to root@172.24.4.12 ([Errno None] Unable to connect to port 22 on 172...

Read more...

Revision history for this message
sreeteja (sreeteja) wrote :

This is the tempest log in HMC I am able to see the partition launched in active state with login credentials displayed (Non-emergency shell mode)

Revision history for this message
sreeteja (sreeteja) wrote :

        Body: {"server": {"OS-EXT-STS:task_state": "deleting", "addresses": {}, "links": [{"href": "http://9.152.151.140/compute/v2.1/servers/b0e931ba-5953-47c5-8a6d-5137a0312738", "rel": "self"}, {"href": "http://9.152.151.140/compute/servers/b0e931ba-5953-47c5-8a6d-5137a0312738", "rel": "bookmark"}], "image": "", "OS-EXT-STS:vm_state": "building", "OS-SRV-USG:launched_at": null, "flavor": {"id": "12345", "links": [{"href": "http://9.152.151.140/compute/flavors/12345", "rel": "bookmark"}]}, "id": "b0e931ba-5953-47c5-8a6d-5137a0312738", "security_groups": [{"name": "tempest-TestVolumeBootPattern-1708102144"}], "user_id": "2312da2e111e4229ae1d8a90d3337619", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 0, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "BUILD", "updated": "2017-09-13T10:09:49Z", "hostId": "3bdbf6fdd7747b7dbcd1371dc7ce664516d9fd0516d709618856167f", "OS-SRV-USG:terminated_at": null, "key_name": "tempest-TestVolumeBootPattern-1647803906", "name": "tempest-TestVolumeBootPattern-server-1535705781", "created": "2017-09-13T10:06:39Z", "tenant_id": "eb2305f88725446bbeb5e3dd138a45f2", "os-extended-volumes:volumes_attached": [{"id": "7e20642c-4a8b-432f-83dc-262fb3c419f0"}], "metadata": {}}}
2017-09-13 12:12:59,379 34607 ERROR [tempest.common.compute] Server b0e931ba-5953-47c5-8a6d-5137a0312738 failed to delete in time
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute Traceback (most recent call last):
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute File "/opt/stack/tempest/tempest/common/compute.py", line 253, in create_test_server
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute clients.servers_client, server['id'])
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute File "/opt/stack/tempest/tempest/common/waiters.py", line 127, in wait_for_server_termination
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute raise lib_exc.TimeoutException
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute TimeoutException: Request timed out
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute Details: None
2017-09-13 12:12:59.379 34607 ERROR tempest.common.compute
2017-09-13 12:12:59,442 34607 INFO [tempest.lib.common.rest_client] Request (TestVolumeBootPattern:_run_cleanups): 400 DELETE http://9.152.151.140/volume/v3/eb2305f88725446bbeb5e3dd138a45f2/volumes/7e20642c-4a8b-432f-83dc-262fb3c419f0 0.061s
2017-09-13 12:12:59,442 34607 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None

Revision history for this message
sreeteja (sreeteja) wrote :

sometimes it fails due to this issue like unable to delete server within time ....For this I tried to
increase the 'ready_wait' parameter which means that it waits this long to start with the next process

Sreeram Vancheeswaran (sreeram-vancheeswaran)
Changed in nova-dpm:
assignee: nobody → sreeteja (sreeteja)
Revision history for this message
Sreeram Vancheeswaran (sreeram-vancheeswaran) wrote :

Sreeteja, as discussed reducing the prior to High, as the issue is sporadic

Changed in nova-dpm:
importance: Critical → High
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.