all-hosts options is not supported when using pip

I see this too when building the charm on a Focal system.

However, I don't see it in the stable, next nor freshly built charm from tip when built in CI.

I'm guessing this has to do with how charm build puts together the wheelhouse from the perspective of the Python interpreter running the ``charm build`` command, which may result in wheels incompatible with the target system.

The allow-hosts issue was attempted solved in layer-basic a while back [0][1], and I think what is happening here is that the assumptions made about what the world looks like on the installed series break with the deps put together by a Focal build host.

We currently have to build reactive charms on a Python 3.5 system to ensure it will deploy across all supported series.

0: https://github.com/juju-solutions/layer-basic/pull/169
1: https://github.com/juju-solutions/layer-basic/pull/172

I'm not sure why the build environment would have any effect, but the logic in layer:basic to decide whether or not to include allow_hosts is using the OS series as a proxy check for whether setuptools is new enough to disallow that option. It should just check the version directly instead.

Update

The problem is not layer-basic the problem is dependencies.

Several dependencies try to install newer versions of setuptools and setuptools_scm.

By adding setuptools<42.0.0 to octavia's wheelhouse.txt we get past the `allow-hosts` error. However that is just the beginning of trobules. Several other dependecies including importlib-resources and cmd2 want setuptools_scm.

I have yet to get a succesfull deploy.
NOTE this include building on Xenial.

With a wheelhouse:
setuptools<42.0.0
Jinja2<=2.10.1
keystoneauth1
pbr
python-novaclient
python-neutronclient
# REMOVEME -- fix for https://github.com/pyparsing/pyparsing/issues/164
pyparsing==2.4.2
neutron_lib
pytest-runner
# python-zipp 2.x introduces a requirement for setuptools 42.x which is not available.
# resolve conflict:
# layer-basic requires setuptools<42, zipp>=2.0.0 requires setuptools>42
# LP: #1862186
zipp < 2.0.0

On Bionic because the allow-hosts block is working we run into:

unit-octavia-0: 22:54:31 DEBUG unit.octavia/0.install Collecting importlib-resources
unit-octavia-0: 22:54:31 DEBUG unit.octavia/0.install Installing build dependencies: started
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Installing build dependencies: finished with status 'error'
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Complete output from command /var/lib/juju/agents/unit-octavia-0/.venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-by707cp0 --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- setuptools>=30.3 wheel setuptools_scm[toml]>=3.4.1:
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Looking in links: wheelhouse
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Collecting setuptools>=30.3
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Collecting wheel
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Collecting setuptools_scm[toml]>=3.4.1
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Could not find a version that satisfies the requirement setuptools_scm[toml]>=3.4.1 (from versions: 1.17.0)
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install No matching distribution found for setuptools_scm[toml]>=3.4.1
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install ----------------------------------------
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Command "/var/lib/juju/agents/unit-octavia-0/.venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-by707cp0 --no-warn-script-location --no-binary
:none: --only-binary :none: --no-index --find-links wheelhouse -- setuptools>=30.3 wheel setuptools_scm[toml]>=3.4.1" failed with error code 1 in None
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install Traceback (most recent call last):
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install File "/var/lib/juju/agents/unit-octavia-0/charm/hooks/install", line 8, in <module>
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install basic.bootstrap_charm_deps()
unit-octavia-0: 22:54:32 DEBUG unit.octavia/0.install File "lib/charms/layer/basic.py", lin...

Fix proposed to branch: master
Review: https://review.opendev.org/737648

I've created a PR to charm-build to process wheelhouse.txt files more holistically so that version pins in base layers can be inherited and apply while still allowing for higher layers to override those pins.

https://github.com/juju/charm-tools/pull/569

Reviewed: https://review.opendev.org/737648
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=6b4235ba06d7021f048b6bd26160749c9b9f09c3
Submitter: Zuul
Branch: master

commit 6b4235ba06d7021f048b6bd26160749c9b9f09c3
Author: David Ames <email address hidden>
Date: Tue Jun 23 22:13:52 2020 +0000

    Pin setuptools and dependencies of dependencies

    Pin setuptools to keep allow-hosts setting for off-line deployments for
    < focal.

    Pin dependecies of dependencies of netaddr.

    Update bundles for GSSS topology change.

    Adjust target_deploy_status to match new behaviour for OVN charms.

    Double the number of compute nodes, I suspect many of our failures
    is due to the test running out of resources due to the
    unpredictable nature of the number of instance Octavia will start.

    Remove payload_instance setup jobs as that is now handled by the
    test itself.

    Change-Id: Ia6631a29a0bd1a4ce5b992fecf047884205cd1e1
    Closes-Bug: #1884164