Fuel for OpenStack

Deployment of master node failed with 'storage volume exists already'

  1. Series mitaka
  2. Bug #1566966
Bug #1566966 reported by Alexandr Kostrikov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Dmitry Belyaninov
Fuel for OpenStack 10.0
Mitaka
Fix Committed
High
Dmitry Belyaninov
Fuel for OpenStack 9.2

Bug Description

Detailed information:
On swarm run [0] we have a failure in installation of master node at first step:
        Snapshot: empty
libvirt: Storage Driver error : cannot unlink file '(null)': Bad address
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
2016-04-05 21:03:21,154 - ERROR __init__.py:66 -- get_admin_node_ip raised: libvirtError("storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already",)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/models/environment.py", line 144, in get_admin_node_ip
    self.d_env.nodes(
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/models/environment.py", line 216, in d_env
    boot_from=settings.ADMIN_BOOT_DEVICE)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 244, in describe_environment
    environment = cls.create_environment(config)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 283, in create_environment
    environment.create_node(config_node)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 385, in create_node
    new_vol.define()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/volume.py", line 34, in define
    self.driver.volume_define(self)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 897, in volume_define
    self.xml_builder.build_volume_xml(volume), 0)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/libvirt.py", line 2846, in createXML
    if ret is None:raise libvirtError('virStorageVolCreateXML() failed', pool=self)
libvirtError: storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already

Revision history for this message
Alexandr Kostrikov (akostrikov-mirantis) wrote :

Possible duplicate of https://bugs.launchpad.net/fuel/+bug/1326457

Changed in fuel:
importance: Undecided → High
Oleksiy Molchanov (omolchanov)
Changed in fuel:
status: New → Confirmed
Fuel Devops McRobotson (fuel-devops-robot)
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Passed on ISO 9.0-mos-354: https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.thread_1/107/testReport/(root)/setup_master/

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Andrey Lavrentyev (alavrentyev) wrote :

Swarm failure with the same symptoms: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha_one_controller/39/testReport/%28root%29/setup_master/

It affects 9.1 version

9.1 snapshot #177

Setting status back to Confirmed

Changed in fuel:
status: Invalid → Confirmed
Revision history for this message
Alexander Gubanov (ogubanov) wrote :

I faced with the same issue on MOS QA CI http://cz7776.bud.mirantis.net:8080/jenkins/job/9.x_Ironic/83/consoleFull

9.1 snapshot #193

Timur Nurlygayanov (tnurlygayanov)
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Fuel Sustaining (fuel-sustaining-team)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The issue was fixed on MOS CI by updating the images, Alexander Gubanov knows more details.

Revision history for this message
Alexander Gubanov (ogubanov) wrote :

I removed files from /var/lib/libvirt/images/ after that deploy is passed.

Kyrylo Galanov (kgalanov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Fuel Infra Apps (fuel-infra-apps)
Revision history for this message
Alexander Charykov (acharykov) wrote :

Doesn't belongs to infra-apps-team.

Changed in fuel:
assignee: Fuel Infra Apps (fuel-infra-apps) → nobody
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The issue affects automated tests and doesn't affect the product.

tags: added: non-release
Revision history for this message
Alexandra (aallakhverdieva) wrote :

Occurs on 9.1 snapshot #347
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/84/testReport/(root)/setup_master/

Revision history for this message
Alexandra (aallakhverdieva) wrote :

New fail:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_3/85/testReport/(root)/setup_master/

Revision history for this message
Alexandra (aallakhverdieva) wrote :

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bonding_ha_one_controller/148/testReport/(root)/setup_master/

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: nobody → Fuel QA Team (fuel-qa)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/408649

Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :

Fix was merged to "stable/mitaka"
We need some time to check that the bug will not be reproduced somewhere.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/414450

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/426280

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/newton)

Reviewed: https://review.openstack.org/414450
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1a5c0468e2b3a53a074fcd335a4b615c440e7c26
Submitter: Jenkins
Branch: stable/newton

commit 1a5c0468e2b3a53a074fcd335a4b615c440e7c26
Author: Dmitry Belyaninov <email address hidden>
Date: Thu Dec 8 14:43:24 2016 +0000

    Environment creation procedure modification

    Sometimes installation of environment failed
    because storage volume(s) exists already.
    The issue is hard-to-reproduce. Some logging and
    setting internal attribute _virt_env were added.

    Change-Id: Ib49c904ca7c961617f0f328230c2658a13bfdecf
    Related-Bug: 1566966
    (cherry picked from commit 1b2d807336f41d0d319f273790c708ce2929d1c5)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/426280
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=f6dbf49b7b7821c07d33ba8504a1b189bb2686ae
Submitter: Jenkins
Branch: master

commit f6dbf49b7b7821c07d33ba8504a1b189bb2686ae
Author: Dmitry Belyaninov <email address hidden>
Date: Thu Dec 8 14:43:24 2016 +0000

    Environment creation procedure modification

    Sometimes installation of environment failed
    because storage volume(s) exists already.
    The issue is hard-to-reproduce. Some logging and
    setting internal attribute _virt_env were added.

    Change-Id: Ib49c904ca7c961617f0f328230c2658a13bfdecf
    Related-Bug: 1566966
    (cherry picked from commit 1b2d807336f41d0d319f273790c708ce2929d1c5)

Dmitry Belyaninov (dbelyaninov)
Changed in fuel:
status: Confirmed → Fix Committed
assignee: Fuel QA Team (fuel-qa) → Dmitry Belyaninov (dbelyaninov)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.