oslo.cache

zuul job dsvm-grenade error

Bug #1857114 reported by zhangyuhe
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Incomplete
Undecided
Unassigned
networking-ovn
Incomplete
Undecided
Unassigned
oslo.cache
New
Undecided
Unassigned

Bug Description

Many tasks have the following errors

2019-12-19 05:32:35.240 | ERROR nova.cmd.manage [None req-12e00068-1f43-43ea-9608-98b5c83b16bc None None] Error attempting to run <function fill_virtual_interface_list at 0x7f67744092f0>: ModuleNotFoundError: No module named 'memcache'
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage Traceback (most recent call last):
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/cmd/manage.py", line 504, in _run_migration
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  found, done = migration_meth(ctxt, count)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1015, in wrapper
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  return fn(*args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/virtual_interface.py", line 279, in fill_virtual_interface_list
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _set_or_delete_marker_for_migrate_instances(cctxt, marker)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 223, in wrapped
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  return f(context, *args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/virtual_interface.py", line 305, in _set_or_delete_marker_for_migrate_instances
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  instance.create()
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  return fn(self, *args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/instance.py", line 629, in create
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  self._load_ec2_ids()
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/instance.py", line 949, in _load_ec2_ids
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  self.ec2_ids = objects.EC2Ids.get_by_instance(self._context, self)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  result = fn(cls, context, *args, **kwargs)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 231, in get_by_instance
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  ec2_ids = cls._get_ec2_ids(context, instance)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 217, in _get_ec2_ids
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  ec2_ids['instance_id'] = id_to_ec2_inst_id(context, instance.uuid)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 57, in id_to_ec2_inst_id
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  int_id = get_int_id_from_instance_uuid(context, instance_id)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/objects/ec2.py", line 36, in memoizer
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _CACHE = cache_utils.get_client(expiration_time=_CACHE_TIME)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/cache_utils.py", line 54, in get_client
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _get_default_cache_region(expiration_time=expiration_time))
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/opt/stack/new/nova/nova/cache_utils.py", line 65, in _get_default_cache_region
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  cache.configure_cache_region(CONF, region)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  '%s.' % conf.cache.config_prefix)
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 608, in configure_from_config
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  "%sreplace_existing_backend" % prefix, False
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 436, in configure
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  _config_argument_dict, _config_prefix
2019-12-19 05:32:35.240 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 95, in from_config_dict
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  for key in config_dict
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 207, in __init__
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  super(MemcacheArgs, self).__init__(arguments)
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 115, in __init__
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  self._imports()
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage  import memcache # noqa
2019-12-19 05:32:35.241 | ERROR nova.cmd.manage ModuleNotFoundError: No module named 'memcache'

full log:
https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/grenade.sh.txt.gz

Revision history for this message
Brian Haley (brian-haley) wrote :

This doesn't look like a bug in networking-ovn, but instead with nova, I added them as an affected project.

Changed in networking-ovn:
status: New → Incomplete
Revision history for this message
zhangyuhe (ikukaku) wrote :

hey haley
I think this is a problem in oslo.cache

Revision history for this message
Matt Riedemann (mriedem) wrote :

nova is configured to use memcache as the caching backend in this job:

https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/etc/nova/nova.conf.txt.gz

[cache]
memcache_servers = localhost:11211
backend = dogpile.cache.memcached
enabled = True

I see dogpile.cache==0.9.0 installed:

https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/pip3-freeze.txt.gz

And I see memcached is installed:

https://6081989ad82b8be7d120-53e9ef0ccb08d29c43f34421d87aa388.ssl.cf1.rackcdn.com/698632/3/check/networking-ovn-dsvm-grenade/72e94e8/logs/dpkg-l.txt.gz

ii memcached 1.5.6-0ubuntu1.2 amd64 high-performance memory object caching system

The errors drop off in recent days as well - was this fixed or worked around somehow?

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22&from=7d

Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

Its not occurring now.

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22ModuleNotFoundError:%20No%20module%20named%20'memcache'%5C%22&from=7d

I have not seen this recently and grenade job working fine. I do not know exact fix for that but marking incomplete for now and we can reopen this if show up again.

Changed in nova:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.