tacker doesn't cleanup the resources created during the functional tests run

Bug #1771494 reported by Dinesh Bhor
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tacker
In Progress
Undecided
Naoaki Horie

Bug Description

Tacker functional test cases runs on a real environment.
During the functional test cases run it creates many resources like VNFD's and VNF's.
If any test case fails it doesn't cleanup the created resources which makes developer to cleanup all the resources manually one by one which is a tedious process.

Steps to reproduce:
===================
1. Make some changes in the code which ensures that the test cases will fail.
2. Run "./tools/prepare_functional_test.sh " and "tox -e functional"

Result of "tox -e functional":
----------------------------------------------------------------------------------------
          ----------------------snip----------------------
----------------------------------------------------------------------------------------

tacker.tests.functional.vnfm.test_vnfm_param.VnfmTestParam.test_vnf_param_tosca_template
----------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tacker/tests/functional/vnfm/test_vnfm_param.py", line 116, in test_vnf_param_tosca_template
        values_dict)
      File "tacker/tests/functional/vnfm/test_vnfm_param.py", line 61, in _test_vnf_create
        vnf_instance = self.client.create_vnf(body=vnf_dict)
      File "/opt/stack/tacker/.tox/functional/local/lib/python2.7/site-packages/tackerclient/v1_0/client.py", line 100, in with_params
        ret = self.function(instance, *args, **kwargs)
      File "/opt/stack/tacker/.tox/functional/local/lib/python2.7/site-packages/tackerclient/v1_0/client.py", line 438, in create_vnf
        return self.post(self.vnfs_path, body=body)
      File "/opt/stack/tacker/.tox/functional/local/lib/python2.7/site-packages/tackerclient/v1_0/client.py", line 296, in post
        headers=headers, params=params)
      File "/opt/stack/tacker/.tox/functional/local/lib/python2.7/site-packages/tackerclient/v1_0/client.py", line 209, in do_request
        self._handle_fault_response(status_code, replybody)
      File "/opt/stack/tacker/.tox/functional/local/lib/python2.7/site-packages/tackerclient/v1_0/client.py", line 183, in _handle_fault_response
        exception_handler_v10(status_code, des_error_body)
      File "/opt/stack/tacker/.tox/functional/local/lib/python2.7/site-packages/tackerclient/v1_0/client.py", line 68, in exception_handler_v10
        status_code=status_code)
    tackerclient.common.exceptions.InternalServerError: ERROR: Internal Error

======
Totals
======
Ran: 30 tests in 3709.0000 sec.
 - Passed: 8
 - Skipped: 4
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 18
Sum of execute time for each test: 5095.5820 sec.

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:06:45.746295
 - Worker 1 (1 tests) => 0:17:08.702473
 - Worker 2 (28 tests) => 1:01:45.153858

Slowest Tests:
Test id Runtime (s)
-------------------------------------------------------------------------------------------------------------------------------- -----------
tacker.tests.functional.vnfm.test_tosca_vnf.VnfTestToscaCreate.test_create_delete_vnf_static_ip 1028.702
tacker.tests.functional.vnfm.test_tosca_vnf.VnfTestToscaCreateFlavorCreation.test_create_delete_vnf_tosca_no_monitoring 424.592
tacker.tests.functional.nfvo.test_nfvo.NsdTestCreate.test_create_delete_network_service 407.749
tacker.tests.functional.vnfm.test_tosca_vnf_block_storage.VnfBlockStorageTestToscaCreate.test_create_delete_vnf_tosca_from_vnfd 405.746
tacker.tests.functional.vnfm.test_tosca_vnf.VnfTestToscaCreate.test_re_create_delete_vnf 387.145
tacker.tests.functional.vnfm.test_tosca_vnf_scale.VnfTestToscaScale.test_vnf_tosca_scale 366.509
tacker.tests.functional.vnfm.test_tosca_vnf.VnfTestToscaCreate.test_create_delete_vnf_from_template 353.542
tacker.tests.functional.nfvo.test_vim.VimTestCreate.test_create_delete_local_vim 340.464
tacker.tests.functional.nfvo.test_nfvo.NsdTestCreate.test_create_delete_nsd 317.850
tacker.tests.functional.vnfm.test_tosca_vnf_multiple_vdu.VnfTestToscaMultipleVDU.test_create_delete_tosca_vnf_with_multiple_vdus 155.931
ERROR: InvocationError: '/opt/stack/tacker/tools/ostestr_compat_shim.sh'
_______________________________________________________________________________ summary ________________________________________________________________________________
ERROR: functional: commands failed

The resources created by tacker during the functions tests run are below:
=========================================================================

nfv@ubuntu:~$ tacker vnfd-list
Deprecated: tacker command line is deprecated, will be deleted after Rocky is released.
+--------------------------------------+------------------------------------------------+-----------------+------------------------------+
| id | name | template_source | description |
+--------------------------------------+------------------------------------------------+-----------------+------------------------------+
| 279958d9-5cd4-453d-a7cd-a3e9e59b87a3 | ping monitor vnf with tosca template | onboarded | VNFD With Ping Monitor |
| 39268af7-c7bb-451b-8748-c9ddc18820e2 | test_tosca_vnf_scale_all | onboarded | sample-tosca-vnfd-scaling |
| 47851a9d-3045-4ef7-96b7-6ec8c77ff589 | sample_cirros_vnf_no_monitoring_vim_id | onboarded | VNFD With no Monitor |
| 51b49142-65dc-466d-8546-95b448bb9b5d | test_tosca_vnf_with_cirros | onboarded | Demo example |
| 5d8dc2cb-dcad-44f5-8dc5-042a7c7dc372 | ping monitor multi vdu vnf with tosca template | onboarded | Multi VDU monitoring exam... |
| 72c22ca8-a96f-4617-9e8f-783e687626a5 | sample-tosca-vnfd-multi-vdu | onboarded | Demo example |
| 7b3ace9c-1b7b-4946-aa98-b96b9bf89cd0 | tosca_vnfd_with_auto_flavor | onboarded | Demo example |
| 808d1f7b-ec47-436f-91b2-9d15777ec536 | test-ns-vnfd1 | onboarded | Demo example |
| 97e73706-1d71-469d-8f11-442245b976ec | cirros_vnfd_tosca_param | onboarded | VNF TOSCA template with i... |
| bce87fd9-237d-4d57-a2ba-e43cd87a74d0 | test-ns-vnfd2 | onboarded | Demo example |
| d14e4792-6963-4d36-8b21-83b1f1c303bd | test_tosca_vnf_with_cirros_no_monitoring | onboarded | VNFD with predefined prop... |
| fafeee6a-e77b-4ea8-a1e7-408c12d97e97 | sample_cirros_vnf_no_monitoring_default_vim | onboarded | VNFD With no Monitor |
+--------------------------------------+------------------------------------------------+-----------------+------------------------------+
nfv@ubuntu:~$ tacker vnf-list
Deprecated: tacker command line is deprecated, will be deleted after Rocky is released.
+--------------------------------------+--------------------------------------------------------------------------------------+---------------------------------------+--------+--------------------------------------+--------------------------------------+
| id | name | mgmt_url | status | vim_id | vnfd_id |
+--------------------------------------+--------------------------------------------------------------------------------------+---------------------------------------+--------+--------------------------------------+--------------------------------------+
| b671da54-c92a-4bf3-a7f6-ef516704b24b | test_tosca_vnf_with_cirros_inline | {"VDU1": "192.168.120.10"} | ACTIVE | 1c451871-c979-404e-9425-1e6f95185376 | 3af6774c-4924-4f2e-b44d-586d765b018a |
| c796e267-2b2b-4127-b662-2477f5ac442b | create_vnf_bce87fd9-237d-4d57-a2ba-e43cd87a74d0_23b1ac99-108d-4b3a-9c57-f63e13561ba7 | {"VDU1": "192.168.120.4"} | ACTIVE | 1c451871-c979-404e-9425-1e6f95185376 | bce87fd9-237d-4d57-a2ba-e43cd87a74d0 |
| dd00ee58-f6bb-44f0-a1b4-9c65bbe3e733 | tosca_vnf_with_auto_flavor | {"VDU1_flavor_func": "192.168.120.3"} | ACTIVE | 1c451871-c979-404e-9425-1e6f95185376 | 7b3ace9c-1b7b-4946-aa98-b96b9bf89cd0 |
| e6529218-64d0-467a-a03e-9e879890dc37 | test_tosca_vnf_scale_all | | ERROR | 1c451871-c979-404e-9425-1e6f95185376 | 39268af7-c7bb-451b-8748-c9ddc18820e2 |
| f6ebc0cc-de23-4f8a-9a5a-3c176e710373 | test_vnf | {"VDU1": "192.168.120.5"} | ACTIVE | 1c451871-c979-404e-9425-1e6f95185376 | daf7d666-bd83-4ce5-ad59-aa72055d4521 |
| fdefa218-842d-487c-91b9-7a879a25287f | test_tosca_vnf_with_cirros | | ERROR | 1c451871-c979-404e-9425-1e6f95185376 | 51b49142-65dc-466d-8546-95b448bb9b5d |
+--------------------------------------+--------------------------------------------------------------------------------------+---------------------------------------+--------+--------------------------------------+--------------------------------------+

Please see the paste file for more descriptive output of "tox -e functional":
http://paste.openstack.org/show/721057/

Expected result:
================
Tacker should ensure that all the resources created while running functional test cases are cleaned-up after the test run.

Environment details:
====================

Tacker:
commit 0c063bc15fbbc46e74638041969391a06520b307
Author: dineshbhor <email address hidden>
Date: Tue May 8 01:11:48 2018 -0700

    WIP: Implement blueprint 'vdu-affinity-policy'

    Change-Id: I18aaa9d00c8539ccc60cb6e70570dee124df70a3

Devstack:
commit b89bfa21b0e144d8160478b54a45a1087ea3e1df
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue May 8 06:12:17 2018 +0000

    Updated from generate-devstack-plugins-list

    Change-Id: I202b685740fe2b4ea53d115524d3bad01b038af6

summary: - tacker doesn't cleanup the resources created during the functions tests
+ tacker doesn't cleanup the resources created during the functional tests
run
description: updated
Changed in tacker:
assignee: nobody → Dinesh Bhor (dinesh-bhor)
Changed in tacker:
status: New → Confirmed
Changed in tacker:
status: Confirmed → In Progress
status: In Progress → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tacker (master)

Fix proposed to branch: master
Review: https://review.opendev.org/714816

Changed in tacker:
assignee: Dinesh Bhor (dinesh-bhor) → Naoaki Horie (naoaki-horie)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tacker (master)

Reviewed: https://review.opendev.org/714816
Committed: https://git.openstack.org/cgit/openstack/tacker/commit/?id=45c5229459647c89a4c047f70abf0031dfea1b90
Submitter: Zuul
Branch: master

commit 45c5229459647c89a4c047f70abf0031dfea1b90
Author: horie naoaki <email address hidden>
Date: Mon Jan 27 05:55:40 2020 +0000

    Delete remaining VNFD in functinal test

    This patch resolves an issue where if a functional test fails
    in the middle of a run, VNFD remain and the next functional test fails.

    This problem is that if the function test fails in the middle,
    the VNFD created in the function test remains without being deleted,
    so if the same VNFD name is used in the next function test executed,
    the VNFD creation will fail.

    Therefore, the following changes have been made.

    * Changed the timing of calling addCleanup()
    * Made VNFD creation a common function

    This patch fixes the following bugs, but VNF does not.

    Change-Id: I8a932fbe3379604b0c3173d2c06e6ed7abb8beaf
    Partial-Bug: #1771494

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.