[VMware] Time to time creating instance has failed during performance testing

Bug #1494252 reported by Olesia Tsvigun
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel Plugins
Invalid
High
Igor Gajsin
vmware-dvs
Fix Released
Undecided
Andrzej Skupień
Tags: dvs
summary: - [VMware] Time to Time creating instance has failed
+ [VMware] Time to time creating instance has failed
Changed in fuel-plugins:
milestone: none → 6.1
importance: Undecided → High
description: updated
Changed in fuel-plugins:
assignee: nobody → Igor Gajsin (igajsin)
Revision history for this message
Andrzej Skupień (kendriu) wrote :

I have committed today some changes that should fix this problem.

I'm Closing this bug. Please reopen if it occur again.

Changed in vmware-dvs:
assignee: nobody → Andrzej Skupień (kendriu)
status: New → Fix Released
tags: added: dvs
description: updated
Igor Gajsin (igajsin)
summary: - [VMware] Time to time creating instance has failed
+ [VMware] Time to time creating instance has failed during performance
+ testing
Revision history for this message
Vyacheslav Tabolin (slavchick) wrote :

I suppose to find out the reason in vSphere - 'Cannot complete operation due to concurrent modification by another operation'.

Neutron server logs needs.

Changed in fuel-plugins:
status: New → Incomplete
Revision history for this message
Andrzej Skupień (kendriu) wrote :

'Cannot complete operation due to concurrent modification by another operation' is normal communicate, when race condition occurs. In this case operation is repeated (decorator @wrap_retry).

It's normal situation in vmware driver operation flow and it doesn't cause any errors.

Revision history for this message
Andrzej Skupień (kendriu) wrote :

If you change some properties of PortGroup for example, then you first check configVersion of it current config. Then you put this version in configVersion field of ReconfigureDVPortgroup_Task communicate. DVS first checks if configVersion in received ReconfigureDVPortgroup_Task is the same as configVersion of current PortGroup state. If they are the same then DVS applies changes, if not then DVS "throws" exception "Cannot complete operation due to concurrent modification...", because this means that other, parallel operation already changed state of PG. In this case driver have to verify current state of PG and send (or not) his changes again.

Revision history for this message
Vyacheslav Tabolin (slavchick) wrote :

But @wrap_retry operation repeater may cause timeout problems on other modules. I can't deploy more then 4 VM's at the same time on lab correctly.

Revision history for this message
Andrzej Skupień (kendriu) wrote :

that's true, but I didn't found other solution except increasing timeout in other modules, such as haproxy and nova-compute.

Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Can not reproduced anymore at fuel-plugin-vmware-dvs 7.0
+----------------------------------------------------------------------------------------------+
| Response Times (sec) |
+--------------------+--------+--------+--------+---------+---------+--------+---------+-------+
| action | min | median | 90%ile | 95%ile | max | avg | success | count |
+--------------------+--------+--------+--------+---------+---------+--------+---------+-------+
| nova.boot_server | 28.624 | 37.951 | 55.442 | 68.896 | 236.14 | 43.365 | 99.9% | 1311 |
| nova.delete_server | 13.128 | 19.987 | 34.674 | 43.093 | 277.183 | 24.527 | 99.9% | 1311 |
| total | 47.823 | 60.865 | 87.136 | 107.421 | 310.639 | 67.892 | 99.9% | 1311 |
+--------------------+--------+--------+--------+---------+---------+--------+---------+-------+
Load duration: 4508.07324696
Full duration: 4587.78851795

Changed in fuel-plugins:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.