include_system_packages is True and may obscure dependency isolation in venvs

Bug #1760956 reported by Ryan Beisner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi Charm
Fix Released
Medium
Unassigned
OpenStack AODH Charm
Fix Released
Medium
Unassigned
OpenStack Barbican Charm
Fix Released
Medium
Unassigned
OpenStack Barbican SoftHSM Charm
Fix Released
Medium
Unassigned
OpenStack Ceph-FS Charm
Fix Released
Medium
Unassigned
OpenStack Designate Charm
Invalid
Medium
Unassigned
OpenStack Designate-Bind Charm
Fix Released
Medium
Unassigned
OpenStack Keystone LDAP integration
Fix Released
Medium
Unassigned
OpenStack Manila Charm
Fix Released
Medium
Unassigned
OpenStack Manila Generic Backend Charm
Fix Released
Medium
Unassigned
OpenStack Neutron API Open Daylight Charm (Retired)
Won't Fix
Medium
Unassigned
OpenStack Tempest Charm (Deprecated)
Fix Released
Medium
Unassigned
OpenStack openvswitch-odl charm
Won't Fix
Medium
Unassigned
charms.openstack
Invalid
Medium
Unassigned

Bug Description

Use of include_system_packages blurs the lines between libraries which are declared and built into a python virtualenv, and libraries which may or may not exist on the underlying distribution.

There is risk in that, and we should aim for full dependency isolation by setting this to False.

This should be coupled with an explicit declaration of use_venv: True, as use_venv has differing defaults across time and space.

Some (but not all) examples:

https://github.com/openstack/charm-aodh/blob/master/src/layer.yaml

https://github.com/openstack/charm-gnocchi/blob/master/src/layer.yaml

https://github.com/openstack/charm-designate/blob/master/src/layer.yaml

Still need to do a full grep to assess current use_venv and include_system_packages settings currently in the full suite reactive OpenStack charms.

Tags: uosci
Revision history for this message
Ryan Beisner (1chb1n) wrote :

The following confirms and details the current state of the two options, which are essentially at odds with one-another in terms of declaring dependencies in a sane way.

ie. When using a venv, we should never include system packages.

#### All src (reactive) OpenStack charms:
ubuntu@beisner-bastion:~/git/release-tools⟫ find -type f -name layer.yaml | sort
./aodh/src/layer.yaml
./barbican-softhsm/src/layer.yaml
./barbican/src/layer.yaml
./ceph-fs/src/layer.yaml
./designate-bind/src/layer.yaml
./designate/src/layer.yaml
./gnocchi/src/layer.yaml
./keystone-ldap/src/layer.yaml
./manila-generic/src/layer.yaml
./manila/src/layer.yaml
./neutron-api-odl/src/layer.yaml
./openvswitch-odl/src/layer.yaml
./tempest/src/layer.yaml

#### Those which take a stance on use_venv and/or include_system_packages:
ubuntu@beisner-bastion:~/git/release-tools⟫ find -type f -name layer.yaml | sort | xargs egrep "system|venv"
./aodh/src/layer.yaml: use_venv: True
./aodh/src/layer.yaml: include_system_packages: True
./barbican-softhsm/src/layer.yaml: use_venv: true
./barbican-softhsm/src/layer.yaml: include_system_packages: true
./barbican/src/layer.yaml: use_venv: true
./barbican/src/layer.yaml: include_system_packages: true
./designate/src/layer.yaml: use_venv: True
./designate/src/layer.yaml: include_system_packages: True
./gnocchi/src/layer.yaml: use_venv: True
./gnocchi/src/layer.yaml: include_system_packages: True
./keystone-ldap/src/layer.yaml: use_venv: True
./keystone-ldap/src/layer.yaml: include_system_packages: True
./manila/src/layer.yaml: use_venv: true
./manila/src/layer.yaml: include_system_packages: true
./neutron-api-odl/src/layer.yaml: use_venv: True
./neutron-api-odl/src/layer.yaml: include_system_packages: True
./openvswitch-odl/src/layer.yaml: use_venv: True
./openvswitch-odl/src/layer.yaml: include_system_packages: True

#### Those which do not declare either, leaving it up to version chance:
ceph-fs
designate-bind
manila-generic
tempest

Revision history for this message
Ryan Beisner (1chb1n) wrote :
summary: - include_system_packages is True and may obscure dependency isolation
+ include_system_packages is True and may obscure dependency isolation in
+ venvs
Revision history for this message
Ryan Beisner (1chb1n) wrote :

In an interesting and relevant twist, coincidentally the same day we're dealing with pip >= 10.0 and python-apt on a separate bug which affected test execution environments (bug #1760720):

At least in the tempest charm, if I leave use_venv True and set use_system_packages False, I get apt import errors at charm run time. The charms are clearly leaning on the fact that some system-installed libraries are installed outside of the virtualenv (via debs in this case). Lines, blurred. Wires, crossed.

This is just one example, but it is a clear example of how we have missed fully declaring dependencies from a pure python standpoint.

Further ideas, debate, conversation are of course welcome.

Frode Nordahl (fnordahl)
Changed in charm-gnocchi:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-aodh:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-barbican:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-barbican-softhsm:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-ceph-fs:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-designate:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-designate-bind:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-keystone-ldap:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-manila:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-manila-generic:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-tempest:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-neutron-api-odl:
importance: Undecided → Medium
status: New → Triaged
Changed in charm-openvswitch-odl:
importance: Undecided → Medium
status: New → Triaged
Changed in charms.openstack:
importance: Undecided → Medium
status: New → Triaged
Liam Young (gnuoy)
Changed in charm-designate:
status: Triaged → Invalid
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The charm uses:

options:
  basic:
    use_venv: True
    include_system_packages: False

Thus, this is resolved.

Changed in charm-gnocchi:
status: Triaged → Fix Released
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The src/layer.yaml now contains:

options:
  basic:
    use_venv: True
    include_system_packages: False

Thus this is fixed released now.

Changed in charm-aodh:
status: Triaged → Fix Released
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The charm now includes in the layer.yaml:

options:
  basic:
    use_venv: True
    include_system_packages: False

Therefore this is fixed released.

Changed in charm-barbican:
status: Triaged → Fix Released
Changed in charm-barbican-softhsm:
status: Triaged → Fix Released
Changed in charm-ceph-fs:
status: Triaged → Fix Released
Changed in charm-designate-bind:
status: Triaged → Fix Released
Changed in charm-keystone-ldap:
status: Triaged → Fix Released
Changed in charm-manila:
status: Triaged → Fix Released
Changed in charm-manila-generic:
status: Triaged → Fix Released
Changed in charm-tempest:
status: Triaged → Fix Released
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The charm is no longer being maintained.

Changed in charm-neutron-api-odl:
status: Triaged → Won't Fix
Changed in charm-openvswitch-odl:
status: Triaged → Won't Fix
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Marking as invalid as the charms now use venvs and are not dependent on system packages. charms.openstack works in this environment; thus there is nothing to do.

Changed in charms.openstack:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.