nova-manage db archive_deleted_rows crash on tags table

I cannot reproduce this as when instance got destoried the related tags are also deleted:
https://github.com/openstack/nova/blob/683fbe3c84ee70e817ffb8396fefb9ff6cd8830d/nova/db/sqlalchemy/api.py#L1824
and there are no deleted field in the tags table, so their should be nothing to archieve in the tags table when running this command.

I my environment 'tags' table is empty and during nova-manage db archive_deleted_rows I've got above exception.

In my opinion it's because of:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6115
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6118

Table 'tags' do not have column 'id' and 'deleted' which are required by code above.

As I understand:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6219
all tables are processed except:
https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6224

As I have written above issue occurs on Queens release. Probably it's fixed on master branch.
It would be nice to have this also fixed on Queens.

Hmm, we run archive_deleted_rows in our CI so if this was a 100% failure I'm not sure why we wouldn't see it there.

http://logs.openstack.org/36/582536/1/check/nova-next/0852de7/logs/devstack-gate-post_test_hook.txt.gz#_2018-07-16_14_26_31_580

Do you have custom changes to nova?

You should be hitting this short-circuit:

https://github.com/openstack/nova/blob/stable/queens/nova/db/sqlalchemy/api.py#L6106

Do you have a shadow_tags table?

shadow_tags table exists:

       Table: shadow_tags
Create Table: CREATE TABLE `shadow_tags` (
  `resource_id` varchar(36) NOT NULL,
  `tag` varchar(80) NOT NULL,
  PRIMARY KEY (`resource_id`,`tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

According to our changes history this table appeared after upgrade to Ocata release.

Hi, I did not saw the above mentioned code in Ocata, Pike and Queens code, where actually is that code?

I'm not able trace piece of code which was responsible for shadow_tags table creation but somehow it happened during upgrade. I will try to reproduce it during:
- upgrade from newton to ocata
- fresh ocata installation
- fresh pike installation
- fresh queens installation

I've performed fresh installation using packstack of:
- newton
- ocata
- pike
- queens
None of aboved have shadow_tags table in nova database.

I've also performed db upgrade from newton to ocata and shadow_tags table was not introduced.

I don't know how to reproduce it although shadow_tags table appeared in our two openstack clouds after upgrade from newton to ocata.

I solved issue by deleting shadow_tags table.

I ran into the same issue:

# nova-manage db archive_deleted_rows --verbose --until-complete
 Use option "connection_recycle_time" from group "api_database".
Archiving..An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 2300, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 542, in archive_deleted_rows
    run, deleted_instance_uuids = db.archive_deleted_rows(max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1802, in archive_deleted_rows
    return IMPL.archive_deleted_rows(max_rows=max_rows)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5581, in archive_deleted_rows
    tablename, max_rows=max_rows - total_rows_archived)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5467, in _archive_deleted_rows_for_table
    column = table.c.id
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
    raise AttributeError(key)
AttributeError: id

Dropping shadow_tags solved the problem. The database was upgraded in-place at least since Juno/Kilo.