OpenStack Compute (nova)

TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction

  1. Series pike
  2. Bug #1746509
Bug #1746509 reported by Saulo
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
melanie witt
Pike
Fix Committed
High
melanie witt
Queens
Fix Committed
Medium
melanie witt

Bug Description

Hi, I was running OPenstack Newton with no nova_cell0 database and placement-api setup . After migrate to Openstack Pike and correctly setup the nova_cell0 and placement-api everything is working fine except the openstack server list on tenant that already exist .
For example :

1. For a new tenant created after the migration at Pike.
nova --os-project-name="New Project" list
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| c41c7e8d-4bc0-4a0f-a9d3-dc719ae2aff0 | SAMPLE_VM | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.8 |
| 3d3d3e10-f326-4a92-9253-1511e738d1cc | SECOND_INSTANCE | ACTIVE | - | Running | SAMPLE-SUBNET=192.168.0.5 |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
2. For a old tenant created before the migration at Newton .

nova --os-project-name="InterCon" list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-0dd4ef4d-54c2-4cfd-b8d9-636c5736ef5f)

And here the log related to this error .

2018-01-31 07:45:35.832 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Searching by: {'deleted': False} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2311
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Skipping already-collected cell0 list _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2503
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Listing 1000 instances in cell ea0a32fb-c148-4d4f-a42d-4e46356ff926 _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2506
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.934 2340 DEBUG nova.objects.service [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Generated UUID f9ae1e59-21be-4fc3-9b02-ebb4c35783ec for service 19 _from_db_object /usr/lib/python2.7/site-packages/nova/objects/service.py:245
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Unexpected exception in API method: TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336, in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 168, in detail
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 311, in _get_servers
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2415, in get_all
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2512, in _get_instances_by_filters_all_cells
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2543, in _get_instances_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1245, in get_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1235, in _get_by_filters_impl
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1198, in _make_instance_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 448, in _from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions objects.Service, db_inst['services'])
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 1121, in obj_make_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **extra_args)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in _from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions service.save()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 363, in save
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions db_service = db.service_update(self._context, self.id, updates)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return IMPL.service_update(context, service_id, values)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self.force_reraise()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 249, in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions with ctxt_mgr.writer.using(context):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return self.gen.next()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1034, in _transaction_scope
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions context=context) as resource:
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 684, in _produce_block
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions self._writer()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 702, in _writer
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions "Can't upgrade a READER transaction "
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
2018-01-31 07:45:35.938 2340 INFO nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2018-01-31 07:45:35.938 2340 DEBUG nova.api.openstack.wsgi [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1029
2018-01-31 07:45:35.939 2340 INFO nova.osapi_compute.wsgi.server [req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4 f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2 8433db4810f947168950770f8c93a4f2] 192.168.0.180 "GET /v2.1/servers/detail?all_tenants=True HTTP/1.1" status: 500 len: 626 time: 0.2633660

but at same tenant I can show a instance information .

nova --os-project-name="Intercon" show 5249185f-0a8e-4b24-a777-214ed967695f
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| INTERNAL-NETWORK network | 192.168.120.14 |
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | XXXXXXXXXXX |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-10-19T15:30:57.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-10-19T15:30:47Z |
| description | zabbix-intercon |
| flavor:disk | 40 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.medium |
| flavor:ram | 4096 |
| flavor:swap | 0 |
| flavor:vcpus | 2 |
| hostId | 723f077d53d4182fcb76bb87c7fc16efd7ad5fea5d939a9d1a61cb07 |
| id | 5249185f-0a8e-4b24-a777-214ed967695f |
| image | Centos-7 (d3f18d34-bb00-44e1-85cd-71bfae7ca24a) |
| key_name | saulo-fcm360_id_rsa |
| locked | False |
| metadata | {} |
| name | zabbix-intercon |
| os-extended-volumes:volumes_attached | [{"id": "4a30e8a4-86c3-4cef-9f89-5bccefd9df22", "delete_on_termination": false}] |
| progress | 0 |
| security_groups | Zabbix Access, default |
| status | ACTIVE |
| tags | [] |
| tenant_id | e082484fc4004da78407a0c0857ae8a9 |
| updated | 2018-01-31T00:06:36Z |
| user_id | ff9a3af4473c480cacd0c4e18d68edb4 |
+--------------------------------------+----------------------------------------------------------------------------------+

Ant help ?

Saulo Augusto Silva
sauloaugustosilva at gmail.com

Revision history for this message
Matt Riedemann (mriedem) wrote :

This was fixed in queens and backported to stable/pike, but we'd need to also backport https://review.openstack.org/#/c/512456/ to stable/ocata (newton is end of life upstream).

Revision history for this message
Matt Riedemann (mriedem) wrote :

A backport gets tricky because we don't have https://github.com/openstack/nova/commit/a5f800b02477e767883ed661e4b0f827d6debe54 in ocata.

Revision history for this message
Saulo (sauloaugustosilva) wrote :

Hi Mike ,
I read the https://bugs.launchpad.net/bugs/1722404 and there is point that bug was fixed at nova 16.0.2 release , but I am running nova 16.0.3 and still have same problem . If you need more information let me know .

Revision history for this message
Andrea Franceschini (afranceschini) wrote :
Download full text (8.4 KiB)

Hello Saulo,

I run in the very same problem doing the very same update as you (Ocata --> Pike).

I'm not sure the reason is the same, but I can show you the way i fixed (workarounded?) it, maybe it could help you too.

The reason the error:"Can't upgrade a READER transaction to a WRITE.." comes up can be found in the following lines of the error log you provided:

... ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in _from_db_object
... ERROR nova.api.openstack.extensions service.save()

The _from_db_object functions (module service.py) iterates over the service object and verifies if it contains the uuid, in case the uuid is missing a new one get generated and saved.

246 if 'uuid' not in service:
247 service.uuid = uuidutils.generate_uuid()
248 LOG.debug('Generated UUID %(uuid)s for service %(id)i',
249 dict(uuid=service.uuid, id=service.id))
250 service.save()
251
252 return service

But doing the save (a WRITE) on a context open for READ generates the above error (Can't upgrade a READER transaction to a WRITE..).

To workaround this I patched this module this way:

29 from nova.context import RequestContext
...
239 # service._context = context
240 service._context = RequestContext.from_dict(context.to_dict())
241 service.obj_reset_changes()
242
243 # TODO(dpeschman): Drop this once all services have uuids in database
244 if 'uuid' not in service:
245 service.uuid = uuidutils.generate_uuid()
246 LOG.debug('Generated UUID %(uuid)s for service %(id)i',
247 dict(uuid=service.uuid, id=service.id))
248 service.save()

This way you should not actually solve the problem but It should help you expose the real one.

In may case it showed up the following:

2018-02-08 15:43:13.088 9878 ERROR nova.api.openstack.extensions ServiceNotFound: Service 7 could not be found.

Looking in the service table in the nova DB i found this:

MariaDB [nova]> select * from services;
+---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid |
 +---------------------+---------------------+---------------------+----+------------+--------------------+-------------+--------------+----------+---------+-----------------------------+---------------------+-------------+---------+--------------------------------------+
| 2017-03-21 17:02:32 | 2018-02-08 15:07:47 | NULL | 1 | controller | nova-consoleauth | consoleauth | 2646887 | 0 | 0 | NULL | 2018-02-08 15:07:47 | 0 | 22 | 1adc00a7-17...

Read more...

Revision history for this message
Saulo (sauloaugustosilva) wrote :

Hi Andrea Franceschini ,

You were extremely correct . The problem was with some old service with old versions .
So I deleted all services those were not at version 22 and it worked like a charm .

Thank you to share you experience and help me out .

And that proves that is not a duplicated issue .

Saulo Augusto Silva

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/555058

melanie witt (melwitt)
tags: added: cells
Revision history for this message
melanie witt (melwitt) wrote :

I've reproduced this behavior in a functional test: https://review.openstack.org/555058

The problem occurs during an instance list across cells (no scatter-gather) when joining with a nova-compute service record with no uuid. In the API, we use 'services' as a column to join in order to do the instance 'show':

https://github.com/openstack/nova/blob/9465d1c/nova/api/openstack/compute/servers.py#L298

and when we go to query for instances from cell databases, we hit this error.

Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
melanie witt (melwitt) wrote :

Note that this is only reproducible on stable/pike -- the same bug does not exist after that.

Revision history for this message
melanie witt (melwitt) wrote :

Going to add a detailed explanation of what's going on here, now that I've traced through all of it.

We get a request to list instances from the API and by default that will do a join with the 'services' table to return service-related information with instances:

https://github.com/openstack/nova/blob/9465d1c/nova/api/openstack/compute/servers.py#L298

Then, in the compute/api, we go to list the instances across cells:

https://github.com/openstack/nova/blob/9465d1c/nova/compute/api.py#L2463

Which will call _get_instances_by_filters per cell:

https://github.com/openstack/nova/blob/9465d1c/nova/compute/api.py#L2561

Which will call objects.InstanceList.get_by_filters to do the database query:

https://github.com/openstack/nova/blob/9465d1c/nova/compute/api.py#L2593

Then, in the object code, we call the _get_by_filters_impl method (notice it's decorated with '@db.select_db_reader_mode', which will start the database transaction context tracking):

https://github.com/openstack/nova/blob/9465d1c/nova/objects/instance.py#L1222

And nested inside of that database transaction context tracking is a call to _make_instance_list:

https://github.com/openstack/nova/blob/9465d1c/nova/objects/instance.py#L1235

Which calls _from_db_object for each returned instance (joined with 'services' in this case):

https://github.com/openstack/nova/blob/9465d1c/nova/objects/instance.py#L1197

Which will construct a ServiceList of Service objects:

https://github.com/openstack/nova/blob/9465d1c/nova/objects/instance.py#L446

Which will call Service._from_db_object as the objects are being created:

https://github.com/openstack/oslo.versionedobjects/blob/c86c6ed/oslo_versionedobjects/base.py#L1120

Which will generate a uuid if one isn't present and attempt to save it, which will fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" because we're still nested under the '@db.select_db_reader_mode' decorator from the InstanceList._get_by_filters_impl method:

https://github.com/openstack/nova/blob/9465d1c/nova/objects/service.py#L243

------------------------------

This problem doesn't exist past Pike because the implementation in compute/api was changed to call the instance_list.get_instance_objects_sorted API instead:

https://github.com/openstack/nova/blob/stable/queens/nova/compute/api.py#L2460

And that API has exited the previous database transaction context manager before it goes to call _make_instance_list:

https://github.com/openstack/nova/blob/stable/queens/nova/compute/instance_list.py#L105

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/555092

Changed in nova:
assignee: nobody → melanie witt (melwitt)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/555093

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/pike)

Change abandoned by melanie witt (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/555058
Reason: Able to reproduce the same bug on master with:

https://review.openstack.org/#/c/555092

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/555914

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/555915

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/555923

Revision history for this message
melanie witt (melwitt) wrote :

This should be High actually, for the Pike version of the bug, because anyone upgrading from Ocata -> Pike will hit this and be unable to list instances.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/555092
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=464985a4d58f45ee6a5e8f8705cdcdddcd365038
Submitter: Zuul
Branch: master

commit 464985a4d58f45ee6a5e8f8705cdcdddcd365038
Author: melanie witt <email address hidden>
Date: Wed Mar 21 21:12:15 2018 +0000

    Add functional regression test for bug 1746509

    Related-Bug: #1746509

    Change-Id: I6f8f88e448c2c5d4b1f09d68b03d1b8763cb8ae8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/555093
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4
Submitter: Zuul
Branch: master

commit b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4
Author: melanie witt <email address hidden>
Date: Wed Mar 21 22:57:50 2018 +0000

    Move _make_instance_list call outside of DB transaction context

    The _make_instance_list method is used to make an InstanceList object
    out of database dict-like instance objects. It's possible while making
    the list that the various _from_db_object methods that are called might
    do their own database writes.

    Currently, we're calling _make_instance_list nested inside of a 'reader'
    database transaction context and we hit the error:

      TypeError: Can't upgrade a READER transaction to a WRITER
      mid-transaction

    during the _make_instance_list call if anything tries to do a database
    write. The scenario encountered was after an upgrade to Pike, older
    service records without UUIDs were attempted to be updated with UUIDs
    upon access, and that access happened to be during an instance list,
    so it failed when trying to write the service UUID while nested inside
    the 'reader' database transaction context.

    This simply moves the _make_instance_list method call out from the
    @db.select_db_reader_mode decorated _get_by_filters_impl method to the
    get_by_filters method to remove the nesting.

    Closes-Bug: #1746509

    Change-Id: Ifadf408802cc15eb9769d2dc1fc920426bb7fc20

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/555914
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=07a1cbb5a113e9abe497445cb5960b8412823bdd
Submitter: Zuul
Branch: stable/queens

commit 07a1cbb5a113e9abe497445cb5960b8412823bdd
Author: melanie witt <email address hidden>
Date: Wed Mar 21 21:12:15 2018 +0000

    Add functional regression test for bug 1746509

    Related-Bug: #1746509

    Change-Id: I6f8f88e448c2c5d4b1f09d68b03d1b8763cb8ae8
    (cherry picked from commit 464985a4d58f45ee6a5e8f8705cdcdddcd365038)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.openstack.org/555915
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=22b2a8e4645e15990a0f130a8866746497c5b5ee
Submitter: Zuul
Branch: stable/queens

commit 22b2a8e4645e15990a0f130a8866746497c5b5ee
Author: melanie witt <email address hidden>
Date: Wed Mar 21 22:57:50 2018 +0000

    Move _make_instance_list call outside of DB transaction context

    The _make_instance_list method is used to make an InstanceList object
    out of database dict-like instance objects. It's possible while making
    the list that the various _from_db_object methods that are called might
    do their own database writes.

    Currently, we're calling _make_instance_list nested inside of a 'reader'
    database transaction context and we hit the error:

      TypeError: Can't upgrade a READER transaction to a WRITER
      mid-transaction

    during the _make_instance_list call if anything tries to do a database
    write. The scenario encountered was after an upgrade to Pike, older
    service records without UUIDs were attempted to be updated with UUIDs
    upon access, and that access happened to be during an instance list,
    so it failed when trying to write the service UUID while nested inside
    the 'reader' database transaction context.

    This simply moves the _make_instance_list method call out from the
    @db.select_db_reader_mode decorated _get_by_filters_impl method to the
    get_by_filters method to remove the nesting.

    Closes-Bug: #1746509

    Change-Id: Ifadf408802cc15eb9769d2dc1fc920426bb7fc20
    (cherry picked from commit b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/555058
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=26870a01ac2644e79233239b1a6d09050572420f
Submitter: Zuul
Branch: stable/pike

commit 26870a01ac2644e79233239b1a6d09050572420f
Author: melanie witt <email address hidden>
Date: Wed Mar 21 21:12:15 2018 +0000

    Add functional regression test for bug 1746509

    Related-Bug: #1746509

    Change-Id: I6f8f88e448c2c5d4b1f09d68b03d1b8763cb8ae8
    (cherry picked from commit 464985a4d58f45ee6a5e8f8705cdcdddcd365038)
    (cherry picked from commit 07a1cbb5a113e9abe497445cb5960b8412823bdd)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/555923
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9a6310ddc4ec2eeef39a51dd5a2fa86ae9c4b972
Submitter: Zuul
Branch: stable/pike

commit 9a6310ddc4ec2eeef39a51dd5a2fa86ae9c4b972
Author: melanie witt <email address hidden>
Date: Wed Mar 21 22:57:50 2018 +0000

    Move _make_instance_list call outside of DB transaction context

    The _make_instance_list method is used to make an InstanceList object
    out of database dict-like instance objects. It's possible while making
    the list that the various _from_db_object methods that are called might
    do their own database writes.

    Currently, we're calling _make_instance_list nested inside of a 'reader'
    database transaction context and we hit the error:

      TypeError: Can't upgrade a READER transaction to a WRITER
      mid-transaction

    during the _make_instance_list call if anything tries to do a database
    write. The scenario encountered was after an upgrade to Pike, older
    service records without UUIDs were attempted to be updated with UUIDs
    upon access, and that access happened to be during an instance list,
    so it failed when trying to write the service UUID while nested inside
    the 'reader' database transaction context.

    This simply moves the _make_instance_list method call out from the
    @db.select_db_reader_mode decorated _get_by_filters_impl method to the
    get_by_filters method to remove the nesting.

    Closes-Bug: #1746509

    Change-Id: Ifadf408802cc15eb9769d2dc1fc920426bb7fc20
    (cherry picked from commit b1ed92c7af01a9ac7e122a541ce1bdb9be0524c4)
    (cherry picked from commit 22b2a8e4645e15990a0f130a8866746497c5b5ee)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.2

This issue was fixed in the openstack/nova 17.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.1.1

This issue was fixed in the openstack/nova 16.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b1

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.