[vCenter] Can't create snapshot (>2GB) with vSphere 6

Bug #1595206 reported by Ilya Bumarskov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Sergey Nikitin
Mitaka
Won't Fix
High
Sergey Nikitin
Newton
Fix Committed
High
Sergey Nikitin

Bug Description

Fuel 9.0 mos-iso#495

Detailed bug description:
 Creation of snapshot stack in 'Image Uploading' state.

Steps to reproduce:
   - Create new env with vCenter 6.0
   - Add following nodes:
         * Controller
   - Deploy cluster
   - Create instance with 3 GB disk.
   - Try to create snapshot from instance.

Expected results:
  Snapshot was created successfully.

Actual result:
  Snapshot creation stacks in 'Image Uploading' state
  and after some time snapshot Killed. On vCenter side we see 'Export OVF template' task failed by timeout.

First look at this related issues:
https://bugs.launchpad.net/nova/+bug/1546454
https://bugs.launchpad.net/nova/+bug/1278690
https://bugs.launchpad.net/nova/+bug/1495429

Fix on review for bugs 1546454, 1278690, 1495429:
https://review.openstack.org/#/c/281134

Related issues:
https://bugs.launchpad.net/fuel/+bug/1529056
https://bugs.launchpad.net/glance/+bug/1498645

Changed in fuel:
importance: Undecided → Critical
milestone: none → 9.0
assignee: nobody → Alexander Arzhanov (aarzhanov)
tags: added: area-pce-vcenter
description: updated
description: updated
Changed in fuel:
importance: Critical → High
Changed in fuel:
status: New → Triaged
Revision history for this message
Sheena Conant (sheena-conant) wrote :

Given that this particular bug requires that the disk be completely full to manifest, Nathan and I agree that this can be a High priority bug with a comment in release notes.

Revision history for this message
Gregory Elkinbard (gelkinbard) wrote :

This was found at Reliance. And really is an issue for any non-trivial size snapshots.
This does need to be addressed, but the only work around available to Mirantis is to adjust the timeouts.

Revision history for this message
Ilya Elterman (ielterman) wrote :

Greg, please have your team figure out the right timeout setting and then have it release note with the recommendation to change the timeout.

tags: added: release-notes
summary: - [vCenter] Can't create snapshot (>1GB) with vSphere 6
+ [vCenter] Can't create snapshot (>2GB) with vSphere 6
description: updated
description: updated
description: updated
Revision history for this message
Alexander Arzhanov (aarzhanov) wrote :

Update after investigate this issue:

1. Increasing the timeouts does not help fix the problem!

2. To reproduce this bug not needed generate a file to fill all disk space (steps to reproduce in bug description updated)

3. To resolve this problem need apply this patch https://review.openstack.org/#/c/281134 (Please note "First look at this related issues" in bug description). I tested the fix https://review.openstack.org/#/c/281134 on Fuel 9.0 mos-iso#495 and vCenter 6 and "Export OVF template task failed by timeout" not reproduced.

Timeouts, I've tried to change - https://kb.vmware.com/kb/2144686

Revision history for this message
Alexander Arzhanov (aarzhanov) wrote :

For release notes:

1. Not updating the NFC lease - this is addressed with https://review.openstack.org/#/c/281134
2. HTTP timeout when uploading the snapshotted image to Glance - this happens because the OVF exports is taking long time when the root disk is large and HTTP upload to the vSphere datastore times out. The workaround it to increase the HTTP read timeout in vCenter:

Modify the readTimeoutMs setting to 600000 in vpxd.cfg:

<config>
  <vmacore>
    <http>
      <readTimeoutMs>600000</readTimeoutMs>
    </http>
  </vmacore>
..
</config>
Stop and restart vCenter.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

https://review.openstack.org/#/c/281134 (complete refactoring of this part of vmware driver in Nova) is not going to be back ported to stable/mitaka, thus, we should restraint from cherry-picking it do 9.0/mitaka downstream branch, unless absolutely necessary.

IMO, the workaround described by Alexander in #6 and increasing of the NFC timeout value (https://kb.vmware.com/kb/2038497) should be sufficient for 9.x.

tags: added: area-nova
tags: added: 10.0-reviewed
Revision history for this message
Sergey Nikitin (snikitin) wrote :

Marked as "In progress" because the patch is on review https://review.openstack.org/#/c/281134

tags: added: release-notes-done
removed: release-notes
Revision history for this message
Sergey Nikitin (snikitin) wrote :

fix is merged in newton

tags: added: release-notes
Revision history for this message
Maria Zlatkova (mzlatkova) wrote :

A release note has been added to 9.2 known issues: https://review.fuel-infra.org/#/c/29729.

tags: removed: release-notes
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.