An "empty map" (which parses as Null) in an overlay can accidentally delete an entire charm or all of it's options

Bug #2002371 reported by Trent Lloyd
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju

Bug Description

If you have an overlay where one of the keys which should be a map (such as the 'options' section of an application, or, the application itself under 'applications') is left empty, it has the effect of deleting that entire section from the bundle.

Technically, we can understand why this is the case. Becuase YAML parsing is fairly implicit, these values are parsed not as a map but instead as empty/none/null. When merging the bundles, instead of merging the two maps it instead replaces the map with the empty value from the overlay, in the same way you might replace a string with another string. I assume (though have not checked) that the YAML merging basically replaces values in most cases unless it's an map in which case it merges them - possibly with some other special logic.

However this is surprising behaviour and likely to catch users off-guard and deploy a substantially incorrect bundle. I hit this because I commented out the only option set in an overlay bundle for nova-compute, which resulted in it deploying the wrong version of the software because the openstack-origin value was reset to the default 'distro' (along with all the other options which were set, though I have removed from the example here)

This kind of implicit parsing confusion exists in other places of YAML and is a common criticism of YAML. Another famous example is entries like "country: AU" parse as the string "AU" but "country: NO" will parse as a boolean False.

While I don't suggest we should solve every case I think this behaviour is so surprising and confusing and could be solved with some special casing. Possible suggestions

(1) Detect when an overlay sets a value that is expected to be, or already is, a map and throw a warning or error.

(2) Assume empty values to be an empty list, such as suggested in this similar issue for the Salt project:

(3) Parse literal empty values as an empty map but not explicit ~/!!null entries.

It's notable that users may in some cases actaually want to delete a previously set map? And these options may prevent that. But I do not feel like this is a very common likely cases versus an accidentally empty section.

I think this is maybe discussed here as a difference between yaml.v2 and yaml.v3 and similar ins/outs to the above. I'm not sure what the right answer is but my main request is "find a way to make it less likely a user does the wrong thing by accident".

= Example =


    charm: ch:nova-compute
    channel: wallaby/edge
      openstack-origin: cloud:focal-wallaby



# Result: The application is deployed but all of the options are deleted, so openstack-origin is set to the default value



# Result: The application is not deployed, as the entire charm was deleted

Revision history for this message
Trent Lloyd (lathiat) wrote :

I originally hit this because I had commented out the only option in an overlay

ubuntu@lathiat-bastion:~/stsstack-bundles/openstack$ cat ../overlays/vault-openstack-secrets.yaml
# encrypt: True
      ephemeral-device: cinder,32G,1
  - ['nova-compute:secrets-storage', 'vault:secrets']

description: updated
summary: - An "empty list" (which parses as Null) in an overlay can accidentally
+ An "empty map" (which parses as Null) in an overlay can accidentally
delete an entire charm or all of it's options
Revision history for this message
Trent Lloyd (lathiat) wrote :

There is apparently at least one place the null technique is intentionally used by someone here to delete kubeapi-load-balancer:

They do explicitly write "null" though instead of just leaving it empty

Revision history for this message
Harry Pidcock (hpidcock) wrote :

I think the best option here is to make a breaking change in 3.1 or 3.2 to introduce safer default behaviour, where the defaulting of fields is more explicit.

Changed in juju:
importance: Undecided → High
milestone: none → 3.1-beta1
status: New → Triaged
Changed in juju:
milestone: 3.1-beta1 → 3.1-rc1
Revision history for this message
Vitaly Antonenko (anvial) wrote :

Not sure we are ready to make this changes for 3.1, I'll change the milestone to 3.2

Changed in juju:
milestone: 3.1-rc1 → 3.2-beta1
Changed in juju:
milestone: 3.2-beta1 → 3.2-rc1
Changed in juju:
milestone: 3.2-rc1 → 3.2.0
Changed in juju:
milestone: 3.2.0 → 3.2.1
Changed in juju:
milestone: 3.2.1 → 3.2.2
Changed in juju:
milestone: 3.2.2 → 3.2.3
Changed in juju:
milestone: 3.2.3 → 3.2.4
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.