OpenStack Nova Compute Charm

openstack-upgrade action of cinder "conflicts" with nova-compute if deployed to same metal.

Bug #1929097 reported by Paul Goins on 2021-05-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Cinder Charm	New	Undecided	Unassigned
	OpenStack Nova Compute Charm	New	Undecided	Unassigned

Bug Description

Hi,

As noted on a separate bug (https://bugs.launchpad.net/charm-deployment-guide/+bug/1929091), when I recently tried doing an OpenStack upgrade following the charm deployment guide, I hit issues due to interactions between cinder-volume (cinder deployed onto the metals with enabled-services=volume) and nova-compute.

If I set action-managed-upgrades=true and updated openstack-origin on cinder-volume, then executed the upgrade-openstack action, the "apt-get update" and "apt-get upgrade" performed during this process ended up also upgrading the nova-compute packages. This meant that, when I went around to performing the upgrade on nova-compute, the actions claimed that things were already upgraded.

Likewise, if I did the same but upgraded nova-compute first, cinder-volume would claim that it has already been upgraded.

While I did compare the end result on my recent upgrade (stein->train) and saw no real difference between the packages installed with either order, this gave me pause as I was not sure if all actions related to the upgrade process were being done appropriately for whichever charm was upgraded last.

My concerns:

1. In a case of a "conflict" like this, are we sure that we're running all needed upgrade-related actions? It seems to me like we might not; do_action_managed_upgrade() in charmhelpers returns False in the "no upgrade available" case, and would avoid even calling the upgrade callback, which may result in e.g. skipping cinder DB migrations if nova-compute is upgraded first (if relevant for such cinder units).

2. Are there any issues or workarounds which should be added to the charm deployment guide to help address this?

Best Regards,
Paul Goins

Paul Goins (vultaire) on 2021-05-20

summary:

- openstack-upgrade action "conflicts" with nova-compute if deployed to
- same metal.
+ openstack-upgrade action of cinder "conflicts" with nova-compute if
+ deployed to same metal.

Revision history for this message

Andrea Ieri (aieri) wrote on 2023-08-09 (last edit on 2023-08-09):

This is probably caused by the do_openstack_upgrade function running a full apt upgrade after switching the sources:

hooks/nova_compute_utils.py
857 │ def do_openstack_upgrade(configs=None):
858 │ """Perform an uprade of cinder. Takes care of upgrading
859 │ packages, rewriting configs + database migration and
860 │ potentially any other post-upgrade actions.
861 │
862 │ :param configs: The charms main OSConfigRenderer object.
863 │
864 │ """
865 │ new_src = config('openstack-origin')
866 │ new_os_rel = get_os_codename_install_source(new_src)
867 │
868 │ juju_log('Performing OpenStack upgrade to %s.' % (new_os_rel))
869 │
870 │ configure_installation_source(new_src)
871 │ dpkg_opts = [
872 │ '--option', 'Dpkg::Options::=--force-confnew',
873 │ '--option', 'Dpkg::Options::=--force-confdef',
874 │ ]
875 │ apt_update()
876 │ apt_upgrade(options=dpkg_opts, fatal=True, dist=True)

Compare with ceph-osd, which only uses apt_install on the specific ceph packages during the upgrade:

lib/charms_ceph/utils.py
2680 │ def upgrade_osd(new_version, kick_function=None):
2681 │ """Upgrades the current OSD
2682 │
2683 │ :param new_version: str. The new version to upgrade to
2684 │ """
2685 │ if kick_function is None:
2686 │ kick_function = noop
2687 │
2688 │ current_version = get_version()
2689 │ status_set("maintenance", "Upgrading OSD")
2690 │ log("Current Ceph version is {}".format(current_version))
2691 │ log("Upgrading to: {}".format(new_version))
2692 │
2693 │ try:
2694 │ add_source(config('source'), config('key'))
2695 │ apt_update(fatal=True)
2696 │ except subprocess.CalledProcessError as err:
2697 │ log("Adding the Ceph sources failed with message: {}".format(
2698 │ err))
2699 │ status_set("blocked", "Upgrade to {} failed".format(new_version))
2700 │ sys.exit(1)
2701 │
2702 │ kick_function()
2703 │
2704 │ try:
2705 │ # Upgrade the packages before restarting the daemons.
2706 │ status_set('maintenance', 'Upgrading packages to %s' % new_version)
2707 │ apt_install(packages=determine_packages(), fatal=True)

The nova compute charm has the same problem.

This is probably caused by the do_openstack_upgrade function running a full apt upgrade after switching the sources:

hooks/nova_compute_utils.py
 857   │ def do_openstack_upgrade(configs=None):
 858   │     """Perform an uprade of cinder. Takes care of upgrading
 859   │     packages, rewriting configs + database migration and
 860   │     potentially any other post-upgrade actions.
 861   │ 
 862   │     :param configs: The charms main OSConfigRenderer object.
 863   │ 
 864   │     """
 865   │     new_src = config('openstack-origin')
 866   │     new_os_rel = get_os_codename_install_source(new_src)
 867   │ 
 868   │     juju_log('Performing OpenStack upgrade to %s.' % (new_os_rel))
 869   │ 
 870   │     configure_installation_source(new_src)
 871   │     dpkg_opts = [
 872   │         '--option', 'Dpkg::Options::=--force-confnew',
 873   │         '--option', 'Dpkg::Options::=--force-confdef',
 874   │     ]
 875   │     apt_update()
 876   │     apt_upgrade(options=dpkg_opts, fatal=True, dist=True)

Compare with ceph-osd, which only uses apt_install on the specific ceph packages during the upgrade:

lib/charms_ceph/utils.py
2680   │ def upgrade_osd(new_version, kick_function=None):
2681   │     """Upgrades the current OSD
2682   │ 
2683   │     :param new_version: str. The new version to upgrade to
2684   │     """
2685   │     if kick_function is None:
2686   │         kick_function = noop
2687   │ 
2688   │     current_version = get_version()
2689   │     status_set("maintenance", "Upgrading OSD")
2690   │     log("Current Ceph version is {}".format(current_version))
2691   │     log("Upgrading to: {}".format(new_version))
2692   │ 
2693   │     try:
2694   │         add_source(config('source'), config('key'))
2695   │         apt_update(fatal=True)
2696   │     except subprocess.CalledProcessError as err:
2697   │         log("Adding the Ceph sources failed with message: {}".format(
2698   │             err))
2699   │         status_set("blocked", "Upgrade to {} failed".format(new_version))
2700   │         sys.exit(1)
2701   │ 
2702   │     kick_function()
2703   │ 
2704   │     try:
2705   │         # Upgrade the packages before restarting the daemons.
2706   │         status_set('maintenance', 'Upgrading packages to %s' % new_version)
2707   │         apt_install(packages=determine_packages(), fatal=True)

The nova compute charm has the same problem.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.