snap-octavia-diskimage-retrofit

image build failure - dpkg: error processing package grub-efi-amd64-signed

Bug #2037935 reported by Nobuto Murata
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
charm-octavia-diskimage-retrofit
Invalid
Undecided
Unassigned
snap-octavia-diskimage-retrofit
Fix Released
High
James Page

Bug Description

The symptom is pretty close to the previous bug:
https://bugs.launchpad.net/snap-octavia-diskimage-retrofit/+bug/1995130

With yoga deployment, the Octavia retrofitting process started failing this week. I'm not sure if it's the root cause, but I see one update was released for jammy on 2023-09-29 for grub-efi-amd64-signed.
https://launchpad.net/ubuntu/+source/grub2-signed

[/var/log/juju/unit-octavia-diskimage-retrofit-0.log]

[ 63.1] Running: pre-install.d/42-ubuntu-ppa
[ 63.2] Running: pre-install.d/99-apt-get-update
[ 65.2] Running: pre-install.d/99-package-uninstalls
[ 65.3] Installing: initramfs-tools
[ 66.0] Running: install.d/00-baseline-environment
[ 66.8] Running: install.d/00-up-to-date
virt-dib: error: libguestfs error: sh: Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  grub-efi-amd64-bin grub-efi-amd64-signed
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2980 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
grub-efi-amd64-signed amd64 1.187.4~22.04.1+2.06-2ubuntu14.2 [1389 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64
grub-efi-amd64-bin amd64 2.06-2ubuntu14.2 [1591 kB]
Fetched 2980 kB in 2s (1573 kB/s)
(Reading database ...

...

(Reading database
... 64309 files and directories currently installed.)
Preparing to unpack
.../grub-efi-amd64-signed_1.187.4~22.04.1+2.06-2ubuntu14.2_amd64.deb ...
Unpacking grub-efi-amd64-signed (1.187.4~22.04.1+2.06-2ubuntu14.2) over
(1.187.3~22.04.1+2.06-2ubuntu14.1) ...
Preparing to unpack .../grub-efi-amd64-bin_2.06-2ubuntu14.2_amd64.deb ...
Unpacking grub-efi-amd64-bin (2.06-2ubuntu14.2) over (2.06-2ubuntu14.1)
...
Setting up grub-efi-amd64-bin (2.06-2ubuntu14.2) ...
Setting up grub-efi-amd64-signed (1.187.4~22.04.1+2.06-2ubuntu14.2) ...
Trying to migrate /boot/efi into esp config
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script
subprocess returned error exit status 1
Errors were encountered while processing:
 grub-efi-amd64-signed
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

If reporting bugs, run virt-dib with debugging enabled and include the
complete output:

  virt-dib -v -x [...]
"
2023-10-02 04:05:06 ERROR unit.octavia-diskimage-retrofit/0.juju-log server.go:316 action "retrofit-image" failed: "Command '['octavia-diskimage-retrofit', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpb0m5_m00', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpzmylc49x']' returned non-zero exit status 1." "Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-diskimage-retrofit-0/charm/actions/retrofit-image", line 66, in main
    action(args)
  File "/var/lib/juju/agents/unit-octavia-diskimage-retrofit-0/charm/actions/retrofit-image", line 44, in retrofit_image
    instance.retrofit(
  File "/var/lib/juju/agents/unit-octavia-diskimage-retrofit-0/charm/lib/charm/openstack/octavia_diskimage_retrofit.py", line 274, in retrofit
    output = subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['octavia-diskimage-retrofit', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpb0m5_m00', '/var/snap/octavia-diskimage-retrofit/common/tmp/tmpzmylc49x']' returned non-zero exit status 1.

Revision history for this message
Nobuto Murata (nobuto) wrote :
Revision history for this message
Nobuto Murata (nobuto) wrote :

Attaching ~field-critical as a workaround is not known yet.

Revision history for this message
Nobuto Murata (nobuto) wrote :

with debug on and the following steps:

# cd /var/snap/octavia-diskimage-retrofit/common

# wget https://cloud-images.ubuntu.com/releases/jammy/release-20230927/ubuntu-22.04-server-cloudimg-amd64.img

# octavia-diskimage-retrofit -d \
    ubuntu-22.04-server-cloudimg-amd64.img \
    output.img |& tee octavia-diskimage-retrofit_debug.log

Revision history for this message
Nobuto Murata (nobuto) wrote (last edit ):

It was a simple thing. The fix from the previous bug is not included in the stable channel nor any snap channel.

[added]
https://bugs.launchpad.net/snap-octavia-diskimage-retrofit/+bug/1995130
https://github.com/openstack-charmers/octavia-diskimage-retrofit/pull/32/files

[removed]
https://github.com/openstack-charmers/octavia-diskimage-retrofit/commit/6d294e6325078e2175f4950d7ef83814192d2f7c

So a workaround is to use the daily stream of cloud images. But I will keep the field-critical as it's not considered as a safe workaround.

channels:
  latest/stable: –
  latest/candidate: –
  latest/beta: 0.9.12 2021-07-16 (236) 185MB devmode
  latest/edge: 1.0.0+git7.gf0dac14 2023-03-19 (352) 279MB classic
  1.0/stable: 1.0.1 2023-01-19 (344) 277MB classic
  1.0/candidate: ↑
  1.0/beta: ↑
  1.0/edge: 1.0.1+git2.g35bba4a 2023-05-27 (356) 279MB classic
installed: 1.0.1 (344) 277MB classic

$ diff -u /snap/octavia-diskimage-retrofit/{344,356}/usr/local/lib/elements/bug1895835/pre-install.d/42-drop-grub-efi-pkgs
-> no diff

# cat /snap/octavia-diskimage-retrofit/356/usr/local/lib/elements/bug1895835/pre-install.d/42-drop-grub-efi-pkgs
====
#!/bin/sh

set -e

if [ "$DIB_RELEASE"x != jammyx ]; then
    # Remove EFI packages to work around LP: #1895835
    echo "Removing GRUB EFI packages to work around LP: #1895835..."
    DEBIAN_FRONTEND=noninteractive apt-get remove \
        --allow-remove-essential \
        --ignore-missing \
        --purge \
        -y \
        grub-efi-amd64-bin grub-efi-amd64-signed
fi
====

-> no logic for jammy.

Revision history for this message
James Page (james-page) wrote :

revision 352 in latest/edge should have this fix in it.

We need to sort out pushing that to a stable branch and backporting to the 1.0 track

Revision history for this message
James Page (james-page) wrote :

Tested successfully with 1.0/edge - promoted to 1.0/candidate in preparation for release to stable.

Revision history for this message
James Page (james-page) wrote :

https://paste.ubuntu.com/p/44FfW3sZNy/

Revision history for this message
Nobuto Murata (nobuto) wrote :

I've tested the revision 362, and it works for me.

Also, I've confirmed that the generated image works for amphora instances to act as a load balancer for Kubernetes control plane.

octavia-diskimage-retrofit 1.0.1+git4.gb7bdb03 362 1.0/candidate canonical** classic

$ juju run-action --wait octavia-diskimage-retrofit/leader retrofit-image
unit-octavia-diskimage-retrofit-0:
  UnitId: octavia-diskimage-retrofit/0
  id: "52"
  results: {}
  status: completed
  timing:
    completed: 2023-10-04 02:13:43 +0000 UTC
    enqueued: 2023-10-04 02:06:58 +0000 UTC
    started: 2023-10-04 02:07:02 +0000 UTC

$ openstack image show amphora-haproxy-x86_64-ubuntu-22.04-20230927 --format yaml
checksum: 9489e807188adab30605134911dae6f2
container_format: bare
created_at: '2023-10-04T02:13:27Z'
disk_format: qcow2
file: /v2/images/2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc/file
id: 2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc
min_disk: 0
min_ram: 0
name: amphora-haproxy-x86_64-ubuntu-22.04-20230927
owner: ec98e711291f42048d8161a6b50f9491
properties:
  direct_url: rbd://5bf8c2a8-60cc-11ee-b944-7fe79167e4b1/glance/2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc/snap
  locations:
  - metadata:
      store: ceph
    url: rbd://5bf8c2a8-60cc-11ee-b944-7fe79167e4b1/glance/2ba11f7c-a1db-47d4-84a5-c8d2102bf8dc/snap
  os_hash_algo: sha512
  os_hash_value: a49e8c5e1d06dfe7820811f3c8b4df14c89c5821b31e4b9da1ef1e630e3cdf2ad40b49bf81c54ed966d7b3cd69da678cedda585b630fb203cb435d28bf06f563
  os_hidden: false
  source_product_name: com.ubuntu.cloud:server:22.04:amd64
  source_version_name: '20230927'
  stores: ceph
protected: false
schema: /v2/schemas/image
size: 920911872
status: active
tags:
- octavia-amphora
- octavia-diskimage-retrofit
updated_at: '2023-10-04T02:13:42Z'
virtual_size: 6656360448
visibility: shared

$ openstack loadbalancer list --format yaml
- id: a4724fe2-e6fa-47d9-8b4d-75dee5548046
  name: openstack-integrator-d4793dc51966-kubernetes-control-plane
  operating_status: ONLINE
  project_id: fa1080b882c44deeb58303548ccb56ce
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 10.5.5.14

Changed in charm-octavia-diskimage-retrofit:
status: New → Invalid
Revision history for this message
James Page (james-page) wrote :

Testing looks good - just doing a quick run through CI to confirm:

https://review.opendev.org/c/openstack/charm-octavia-diskimage-retrofit/+/897326

Changed in snap-octavia-diskimage-retrofit:
status: New → Fix Committed
importance: Undecided → High
assignee: nobody → James Page (james-page)
Revision history for this message
James Page (james-page) wrote :

Pushed to 1.0/stable channel - marking Fix Released.

Changed in snap-octavia-diskimage-retrofit:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.