OpenStack Octavia Charm

octavia charm-upgrade failed on pip install wheelhouse setuptools-scm

Bug #1891639 reported by Marian Gasparovic on 2020-08-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Octavia Charm	Fix Released	High	David Ames	OpenStack Octavia Charm 20.10

Bug Description

stein-bionic cloud

Cloud was deployed on Aug 12th, all went ok. Charms are specified in the bundle with cs:charmname. Today (Aug 14th) around 8:20 UTC I removed trilio applications and used juju deploy with original bundle.yaml to redeploy it.

This re-deploy triggered charm-upgrade in octavia (I assume also others, saw that also in designate). But octavia failed the upgrade with

2020-08-14 08:26:46 DEBUG upgrade-charm 2020-08-14 08:26:46 DEBUG upgrade-charm 2020-08-14 08:26:46 DEBUG upgrade-charm 2020-08-14 08:26:46 DEBUG upgrade-charm 2020-08-14 08:26:46 DEBUG upgrade-charm
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2451, in resolve
return functools.reduce(getattr, self.attrs, module)
AttributeError: module 'setuptools_scm.hg' has no attribute 'FILES_COMMAND'
/> During handling of the above exception, another exception occurred:
/> Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-6viilk9s/setuptools-scm/setup.py", line 53, in <module>
setuptools.setup(**scm_config())
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 574, in add_defaults
rcfiles = list(walk_revctrl())
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/setuptools/command/sdist.py", line 20, in walk_revctrl
for item in ep.load()(dirname):
File "/tmp/pip-install-6viilk9s/setuptools-scm/src/setuptools_scm/integration.py", line 22, in find_files
command = ep.load()
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/tmp/pip-build-env-cfh7oyz_/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2453, in resolve
raise ImportError(str(exc))
ImportError: module 'setuptools_scm.hg' has no attribute 'FILES_COMMAND'
/> ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-6viilk9s/setuptools-scm/
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-octavia-0/charm/hooks/upgrade-charm", line 8, in <module>
basic.bootstrap_charm_deps()
File "lib/charms/layer/basic.py", line 197, in bootstrap_charm_deps
_update_if_newer(pip, pre_install_pkgs)
File "lib/charms/layer/basic.py", line 284, in _update_if_newer
pkg], env=_get_subprocess_env())
File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-octavia-0/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'setuptools-scm']' returned non-zero exit status 1.
/>uniter.operation runhook.go:132 hook "upgrade-charm" failed: exit status 1
machinelock.go:186 machine lock released for uniter (run upgrade-charm hook)

Tags:

Revision history for this message

Marian Gasparovic (marosg) wrote on 2020-08-14:

bundle.yaml Edit (27.5 KiB, text/plain)

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-08-14:

Good to mention: the OpenStack charms 20.08 got released on August 13th.

This bug is probably related to:
https://bugs.launchpad.net/charm-mysql-router/+bug/1884460
https://github.com/juju-solutions/layer-basic/issues/181
https://bugs.launchpad.net/charm-ceph-fs/+bug/1885935

Changed in charm-octavia:
status:	New → Triaged
importance:	Undecided → High
summary:	- octavia charm-upgrade failed + octavia charm-upgrade failed on pip install wheelhouse setuptools-scm

Revision history for this message

Marian Gasparovic (marosg) wrote on 2020-08-14:

juju-crashdump-openstack-2020-08-14tar.gz Edit (178.4 MiB, application/octet-stream)

Marian Gasparovic (marosg) on 2020-08-14

tags:

added: cdo-qa

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-08-17:

Can easily be reproduced with:
juju deploy cs:octavia-21 --config openstack-origin=cloud:bionic-stein
juju upgrade octavia

Revision history for this message

David Ames (thedac) wrote on 2020-08-17:

TRIAGE:

We may be running into a setup tools SCM bug [0]. The workaround is to remove the .venv directory and retry.

This may be a layer-basic bug to fix the problem in charms.

As Aurelien has pointed out the '-' vs '_' in setuptools-scm may be at play.

Another oddity which may be unrelated is we are getting setuptools_scm-4.1.2.tar.gz when the wheelouse.txt says:
setuptools-scm>=3.0<=3.4.1

-----------
Workaround:

rm -rf /var/lib/juju/agents/unit-octavia-<UNIT>/.venv
juju resolved octavia/<UNIT>

[0] https://github.com/fink/fink-distributions/issues/219

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-08-18:

We end up with setuptools 4 because of a comma missing in "setuptools-scm>=3.0<=3.4.1". It should be "setuptools-scm>=3.0,<=3.4.1". However this doesn't solve the original problem, which is most likely [0] as David mentioned. Maybe we can put our workaround in the upgrade hook somehow?

0: https://github.com/fink/fink-distributions/issues/219

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-18: Related fix proposed to charm-octavia (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/746740

Revision history for this message

David Ames (thedac) wrote on 2020-08-18:

Layer basic fix to remove the venv on charm upgrade: https://github.com/juju-solutions/layer-basic/pull/188

Tested with the following:
juju deploy cs:octavia-21 --config openstack-origin=cloud:bionic-stein --contstraints mem=3040
# Built charm with layer-basic change.
juju upgrade-charm octavia --switch .

The bug is resolved by the change.

Changed in charm-octavia:
assignee:	nobody → David Ames (thedac)

David Ames (thedac) on 2020-08-18

Changed in charm-octavia:
milestone:	none → 20.10
status:	Triaged → In Progress

Revision history for this message

Cory Johns (johnsca) wrote on 2020-08-18:

Layer basic PR merged

Revision history for this message

David Ames (thedac) wrote on 2020-08-18:

#10

With [0] landed, [1] will rebuild Octavia with the update. Once [1] lands, we can back-port to stable.

[0] https://github.com/juju-solutions/layer-basic/pull/188
[1] https://review.opendev.org/#/c/746740/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-19: Related fix merged to charm-octavia (master)

#11

Reviewed: https://review.opendev.org/746740
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=009dd33942938cc29acfbd1ad267352b9d0c7fba
Submitter: Zuul
Branch: master

commit 009dd33942938cc29acfbd1ad267352b9d0c7fba
Author: Aurelien Lourot <email address hidden>
Date: Tue Aug 18 16:39:19 2020 +0200

Fix setuptools-scm's version range

Because of this typo we ended up installing version 4 instead of 3

Change-Id: Ic2d884b8c589d00531b372ba4f742cbe651c7e48
Related-Bug: #1891639

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-19: Fix proposed to charm-octavia (stable/20.08)

#12

Fix proposed to branch: stable/20.08
Review: https://review.opendev.org/746908

Felipe Reyes (freyes) on 2020-08-20

tags:

added: sts

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-08-20: Fix merged to charm-octavia (stable/20.08)

#13

Reviewed: https://review.opendev.org/746908
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=4cb3409e8929b5a5d90d4f24e1e3a23194c7cad6
Submitter: Zuul
Branch: stable/20.08

commit 4cb3409e8929b5a5d90d4f24e1e3a23194c7cad6
Author: Aurelien Lourot <email address hidden>
Date: Tue Aug 18 16:39:19 2020 +0200

Fix setuptools-scm's version range

Because of this typo we ended up installing version 4 instead of 3

(cherry picked from commit 009dd33942938cc29acfbd1ad267352b9d0c7fba)

    This triggers a rebuild and picks
    https://github.com/juju-solutions/layer-basic/pull/188 which resolves
    the linked bug.

Change-Id: I86e94f5a9fb4823f612718d5ef16fa9d263f3749
Closes-Bug: #1891639

Revision history for this message

David Ames (thedac) wrote on 2020-08-20:

#14

The fix to layer basic [0] has now landed in master [1] and 20.08 stable [2].

[0] https://github.com/juju-solutions/layer-basic/pull/188
[1] https://review.opendev.org/#/c/746740/
[2] https://review.opendev.org/#/c/746908/

Changed in charm-octavia:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

auto-github-juju-solutions-layer-basic #181
[closed] Edit
auto-github-fink-fink-distributions #219
[closed] Edit

Bug watches keep track of this bug in other bug trackers.