Juju has no CI for distro-info-data updates

Are you actually looking into this Nicholas?

The best protection for this would be an autopkgtest that would block landing of distro-info-data. It's clear the existing tests didn't stop this, although as mentioned, there is are future-provider tests that play with this idea. Ensuring juju works and responds as expected would best be done in a unit test imho, which have been expanded in light of the changes made. I believe the only remaining question for me is ensuring juju will always bootstrap to something (by adding a hardcoded backup default), as well as what the default bootstrap option will be.

On Fri, Nov 24, 2017 at 03:26:40PM -0000, Nicholas Skaggs wrote:
> The best protection for this would be an autopkgtest that would block
> landing of distro-info-data. It's clear the existing tests didn't stop
> this, although as mentioned, there is are future-provider tests that
> play with this idea. Ensuring juju works and responds as expected would

I suggest that you extend the future-provider tests to also manipulate
the other areas that Juju pays attention to, such as the LTS label and
the --devel label. If you aren't already, you should adopt the my
future-provider type tests into your upstream CI.

Since Juju is no longer maintained as a distribution package, I don't
think you have the option of blocking anything landing in the
distribution itself - certainly not a correct update to
distro-info-data - using autopkgtests. You have (for good reason)
de-coupled the Juju distribution from the Ubuntu distribution by using
snaps. A consequence of that de-coupling is that as the distribution
moves you need to actively keep up and not break, rather than passively
relying on autopkgtests and fixing issues as you update packages in the
distribution.

You might be able to arrange an additional hook to britney to check
and report against Juju CI before proposed migration. But I don't think
the autopkgtest infrastructure hooks are currently suitable for that.

This doesn't apply to existing Juju packages in the distribution that
you're still maintaining, but I think we need a long term solution here
rather than a short term one.

I think you were lucky that on this occasion it happened to be possible
to work around your bug by having Ubuntu revert its update temporarily.
I don't think this is necessarily going to be the case for your next
bug. It doesn't sound to me that your CI has good (or any!) coverage for
this class of bug, or that Juju's assumptions about distribution
releases is now correct.

"Ensuring juju works and responds as expected would best be done in a unit test..." - this is not a concern of unit tests. Just because individual parts have been tested, this does not guarantee that Juju would work as a whole. To draw a parallel - all individual parts of a plane maybe tested, but would *you* fly it once they are put together? Especially, if the assembled plane itself wasn't tested?

The underlying assumptions around how juju should treat information found in distro-info-data definitely needs verified and proven as part of a unit test. As it stands, I think there is still an edge case or two that afaik haven't been asserted against or coded for inside of juju. Without unit tests, these edge cases aren't considered in the codebase. Functional testing can't be depended upon to completely provide coverage for edge cases; this bug is proof of that.

Considering this closed as of 2.3.6 and 2.4-beta1. Both contain logic that allows juju to explicitly choose which series to bootstrap by default. Once shipped, juju will not change. Juju should not be affected by distro-info-data updates.