Live migration fails on deployment with ceph and enabled ephemeral option

Bug #1466075 reported by Tatyanka
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Stanislav Makar
6.1.x
Invalid
High
MOS Maintenance
7.0.x
Fix Released
High
Stanislav Makar

Bug Description

Live Migration fails with message hypervisor can not access to the shared storage

Steps:
1. Create cluster
2. Add 1 node with controller and ceph OSD roles
3. Add 2 node with compute and ceph OSD roles
Storage settings:
 'volumes_ceph': True,
 'images_ceph': True,
 'ephemeral_ceph': True,
 'volumes_lvm': False
4. Deploy the cluster
5. Check ceph status
6. Run ostf tests Live migration
7. ssh on controller - launch instance and run live-migration manually

Expected result:
Instance successfully migrate

Actual:
Migration has not been started with message

node-1.test.domain.local is not on shared storage: Live migration can not be used without shared storage.

ISO: RC2 (522)

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "522"
  build_id: "2015-06-16_13-53-26"
  nailgun_sha: "fa8dec50f3df2626c97f6c38a897cf4e0f80b39d"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "1ea8017fe8889413706d543a5b9f557f5414beae"
  fuel-library_sha: "3528dddbd0c961290909d5e3e256f55ff75cd2fc"
  fuel-ostf_sha: "8fefcf7c4649370f00847cc309c24f0b62de718d"
  fuelmain_sha: "42020c36d6dec9fedf61faa68aa3674156d41977"

env can be reverted:
http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.ceph_ha_one_controller/18/ and on Ubuntu too

I see that storage option was correctly pushed to hiera
[root@node-2 ~]# hiera storage
{"volumes_ceph"=>true,
 "objects_ceph"=>false,
 "ephemeral_ceph"=>true,
 "iser"=>false,
 "volumes_lvm"=>false,
 "images_vcenter"=>false,
 "pg_num"=>512,
 "osd_pool_size"=>"2",
 "metadata"=>{"label"=>"Storage", "weight"=>60},
 "images_ceph"=>true}
[root@node-2 ~]# exit
logout
Connection to node-2 closed.
 and astute post task was run on computes nodes(1, 2) bit at the same time some option in nova.conf was not enabled:
#
# Options defined in nova.virt.libvirt.imagebackend
#

# VM Images format. Acceptable values are: raw, qcow2, lvm,
# rbd, default. If default is specified, then use_cow_images
# flag is used instead of this one. (string value)
# Deprecated group;name - DEFAULT;libvirt_images_type
#images_type=default

# The RADOS pool in which rbd volumes are stored (string
# value)
# Deprecated group;name - DEFAULT;libvirt_images_rbd_pool
#images_rbd_pool=rbd

if we enable values like:
# Deprecated group;name - DEFAULT;libvirt_images_type
#images_type=default
images_type=rbd

and
# Deprecated group;name - DEFAULT;libvirt_images_rbd_pool
#images_rbd_pool=rbd
images_rbd_pool=compute

It starts to work, I can not find the reason why this values was not updated during post task puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/astute/ceph_compute.pp"

Tags: ceph puppet
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
status: New → Confirmed
summary: - Live migration fails on deployment with ceph and enabled ephimaral
+ Live migration fails on deployment with ceph and enabled ephemeral
option
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Oleksiy Molchanov (omolchanov)
Stanislav Makar (smakar)
Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Stanislav Makar (smakar)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/192687

Revision history for this message
Maksym Strukov (unbelll) wrote :

Repro on 6.1-522

{"build_id": "2015-06-16_13-53-26", "build_number": "522", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-16_13-53-26", "build_number": "522", "api": "1.0", "fuel-library_sha": "3528dddbd0c961290909d5e3e256f55ff75cd2fc", "nailgun_sha": "fa8dec50f3df2626c97f6c38a897cf4e0f80b39d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "42020c36d6dec9fedf61faa68aa3674156d41977"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "3528dddbd0c961290909d5e3e256f55ff75cd2fc", "nailgun_sha": "fa8dec50f3df2626c97f6c38a897cf4e0f80b39d", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "42020c36d6dec9fedf61faa68aa3674156d41977"}

Repro steps: https://mirantis.testrail.com/index.php?/cases/view/20878

Changed in fuel:
assignee: Stanislav Makar (smakar) → Vladimir Kuklin (vkuklin)
status: Confirmed → In Progress
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Stanislav Makar (smakar)
Changed in fuel:
assignee: Stanislav Makar (smakar) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/6.1)

Reviewed: https://review.openstack.org/192919
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2e7a08ad9792c700ebf08ce87f4867df36aa9fab
Submitter: Jenkins
Branch: stable/6.1

commit 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
Author: Ryan Moe <email address hidden>
Date: Wed Jun 17 15:29:41 2015 -0700

    Revert "Fix the problem with ceph deployment on scale lab"

    This reverts commit 43b25e4b200c5b994cde81439454d6e2e908a88f.

    Closes-bug: #1466075

    Change-Id: Ib44cf45fb90dca1c8512048d89ceba2b2d47c35b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Stanislav Makar (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/192687
Reason: The patch https://review.openstack.org/#/c/193076/ includes this patch and patch https://review.openstack.org/#/c/190953/ which has been reverted

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

verified on {

    "build_id": "2015-06-18_17-46-09",
    "build_number": "523",
    "release_versions":

{

    "2014.2.2-6.1":

{

    "VERSION":

{

    "build_id": "2015-06-18_17-46-09",
    "build_number": "523",
    "api": "1.0",
    "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab",
    "nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3",
    "feature_groups":

            [
                "mirantis"
            ],
            "openstack_version": "2014.2.2-6.1",
            "production": "docker",
            "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b",
            "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae",
            "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d",
            "release": "6.1",
            "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab",
"nailgun_sha": "126e50e624b1b97043ab87dbf1298312b9771df3",
"feature_groups":

    [
        "mirantis"
    ],
    "openstack_version": "2014.2.2-6.1",
    "production": "docker",
    "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b",
    "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae",
    "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d",
    "release": "6.1",
    "fuelmain_sha": "237c6ec1e9c022d69e5f8ab859887352fc5cb74b"

}

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Dmitry Borodaenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/193065
Reason: A duplicate revert commit has passed CI and was merged instead: https://review.openstack.org/#/c/195466/

Revision history for this message
Alexander Kurenyshev (akurenyshev) wrote :

Repro on

api: '1.0'
astute_sha: 776157f722b13aff5f59bc098cf948793e6498ef
auth_required: true
build_id: 2015-06-24_16-12-23
build_number: '15'
feature_groups:
- mirantis
fuel-library_sha: 7d19bc3783177aebf64fa4c2ae20d845cbd5348f
fuel-ostf_sha: 69e7fa120e8efa7ed74d98efc63079d2f5c69d7b
fuelmain_sha: 3b866d2ff3091a60362327028085fa62fd16c5a0
nailgun_sha: b74f847ec89c4bff1addb830704206dc503125f0
openstack_version: 2014.2.2-7.0
production: docker
python-fuelclient_sha: 1b8574a7c4ea884862763a15c636b066d51f49e7
release: '7.0'
release_versions:
  2014.2.2-7.0:
    VERSION:
      api: '1.0'
      astute_sha: 776157f722b13aff5f59bc098cf948793e6498ef
      build_id: 2015-06-24_16-12-23
      build_number: '15'
      feature_groups:
      - mirantis
      fuel-library_sha: 7d19bc3783177aebf64fa4c2ae20d845cbd5348f
      fuel-ostf_sha: 69e7fa120e8efa7ed74d98efc63079d2f5c69d7b
      fuelmain_sha: 3b866d2ff3091a60362327028085fa62fd16c5a0
      nailgun_sha: b74f847ec89c4bff1addb830704206dc503125f0
      openstack_version: 2014.2.2-7.0
      production: docker
      python-fuelclient_sha: 1b8574a7c4ea884862763a15c636b066d51f49e7
      release: '7.0'

Revision history for this message
Sergey Vasilenko (xenolog) wrote :

live migration, at all, works properly on 7.0-51 ISO

Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Closing as invalid then. Please re-open if reproduced.

Revision history for this message
Stanislav Makar (smakar) wrote :

Guys
for 6.1 this is not merged, patch is here https://review.openstack.org/#/c/193076/

For 7.0 the patch is merged https://review.openstack.org/#/c/195468/ that is why all is working on 7.0

Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

I am passing this to sustaining team then

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Actually for 7.0 issue is partly fixed, according to migration still not finished with success and failed with next messages http://paste.openstack.org/show/395155/ http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.upgrade_one_controller/7/

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Mike Scherbakov (mihgen) wrote :

Folks, please assign this bug to Nova or Ceph team if goes into weeds of how particular Nova & Ceph components are configured or work. Please support those teams though as needed.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/192703

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

this issue was reverted in 6.1 and fixed in 7.0

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

verified on 140 iso for 7.0

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.