Apt/Grub issue with 22.04 Upgrade (Missing libcrypto.so.1.1)

For clarity, when I talk about libssl being removed above, I am referring specifically to libssl1.1. Looking at the logs (and various posts online), it looks like there is a move from libssl1.1 to libssl3 as part of this upgrade which seems to be the source of the issue.

Upon further inspection of the log, I realised that the issue occurs when /etc/grub.d/bin/grubcfg_proxy is being run but this file is not part of grub itself. Looking through the package contents, it looks like grubcfg-cfg is included in grub-customizer which is installed on my system so, this is perhaps actually and issue with the grub-customizer package?

It looks like it is not the first time this package has this kind of compatibility issues: https://askubuntu.com/questions/1168054/how-to-fix-kernel-issue-after-upgrading-from-18-10-to-19-04

My first guess, just the askubuntu link, was that it was a leftover from an old version of the grub-customizer package that hasn't been removed as it is in /etc and subject to conffile handling, but I'm not so sure that's entirely right, as I'd gathered that the old version was from before Disco and this is a fresh Impish install.

I looked into the details, and my conclusions are that the grub-customizer package has a completely broken removal/upgrade story.

The package only installs some binaries, without any custom maintainer scripts or fancy things. Once you run the main program, it goes and modifies the /etc/grub.d scripts automatically for you to customize them as specified. That part already feels weird to me, but wouldn't necessarily cause problems.

However, there are some customizations that are apparently not doable by simply editing the existing configuration scripts. For those, the program moves all existing conf scripts out of the way, and instead installs "proxy scripts" written in a custom language that basically filter the output of the original scripts. The language interpreter is a standard ELF binary with standard dynamic linking, which seems to need the whole Qt stack for some reason. Said interpreter is copied to /etc/grub.d/bin/grubcfg_proxy, presumably so that the system keeps working even if the user uninstalls grub-customizer.

That logic is actually broken, since removing the grub-customizer package might lead to the removal of its dependencies, e.g. the Qt stack, leading to a broken binary.

My suggestion for the path forward would be to patch the application to have /etc/grub.d/bin/grubcfg_proxy be a shell script that invokes the dpkg-managed proxy if it's available, else prints out a warning and executes the original script. We couple that with a post-rm maintainer script that ensures that the /etc-installed script is the one from the current version of the package. We'd need to SRU this to Impish, Focal, and Jammy.

That wouldn't fix every issue with the package, notably leaving the purge situation still broken, but at least that'd fix the problem at hand.

Thoughts?

In the meantime, you can do this to fix your system:

sudo cp /usr/lib/grub-customizer/grubcfg-proxy /etc/grub.d/bin/grubcfg_proxy

EDIT: see https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1969353/comments/14 for a new workaround since the package has been completely removed from the archive

Bonjour Simon, I actually fixed my system at the time by downloading the libssl1.1 deb from Impish but your solution would have worked as well. It looks like this grubcfg_proxy file gets copied when running grub-customizer so, presumably, since I didn’t run the tool after the upgrade, it still had the old version of grubcfg_proxy which relied on libssl1.1.

It looks like if the _proxy files in the grub.d directory (10_linux_proxy,…) linked to /usr/lib/grub-customizer/grubcfg-proxy rather than a copy of the file (or perhaps simpler if grubcfg_proxy was a symlink back to /usr/lib/…), this wouldn’t have been an issue?

This said, I also agree that, more boradly, if the _proxy files are left after grub-customizer is deleted, this is potentially a recipe for disaster…

Status changed to 'Confirmed' because the bug affects multiple users.

I just realized that my workaround only worked as long as the grub-customizer was in the archive. Since we've removed it due to the above issues, the workaround would probably be something like this (completely untested):

orig_dir=/etc/grub.d/proxifiedScripts
main_dir=/etc/grub.d
backup_dir=/etc/grub.d/proxyscript_backup

mkdir -p $backup_dir
for script in $(ls $orig_dir); do
    # we're assuming there's only one proxy script for each proxied file
    proxy_script=$(ls ${main_dir}/*_${script}_proxy)
    # back up the "proxy script"
    cp $main_dir/$proxy_script $backup_dir
    # overwrite it with the original shell script
    cp $orig_dir/$script /main_dir
done

the final "/main_dir" looks like a typo. also, the first cp would fail as $proxy_script already includes the full path. One problem I see is that, for me at least, the files in the proxifiedScripts folders have been stripped of their numbers ("10_linux" is called "linux" for instance). so before copying the file back, you would need to take the script name and strip the _proxy part to find the original name:

#!/bin/bash
main_dir=/etc/grub.d
orig_dir=$main_dir/proxifiedScripts
backup_dir=$main_dir/proxyscript_backup

mkdir -p $backup_dir
for script in $(ls $orig_dir); do
    # we're assuming there's only one proxy script for each proxied file
    proxy_script=$(ls ${main_dir}/*_"${script}"_proxy)
    new_script=$(echo "$proxy_script" | sed 's/_proxy//g')
    # back up the "proxy script"
    mv "$proxy_script" $backup_dir
    # overwrite it with the original shell script if it doesn't already exist
    if [ ! -f "$new_script" ]; then
       cp $orig_dir/"$script" "$new_script"
    else
        echo "file already exists"
    fi
done

Also, if 10_linux has been copied during the upgrade, chances are that the files in the proxy folder is an older version so I wouldn't really want the new files to be overwritten with the old ones so, a simple sudo rm /etc/grub.d/*_proxy works just as well for me to prevent the grub failure which I experienced post upgrade. People can always fire up grub-customizer again and regenerate the files if needed.

Alternatively, could it be as simple as copying /etc/grub.d/bin/grubcfg_proxy as part of the package install/update?

All good points there, just one thing:

Quoting Batwam (2022-04-22 17:34:37)
> Alternatively, could it be as simple as copying
> /etc/grub.d/bin/grubcfg_proxy as part of the package install/update?

You still have a broken system on package removal.

yes, you are right, espectially considering the point you made about the underlying dependancies this script relies on (libssl1.1 being one) which is what caused my Impish->Jammy upgrade issue in the first place.

By the way, just to be clear, it’s not just the package removal which is a problem. Upgrade is also a problem as per the original but report as the file in the bin folder isn’t getting updated while libssl is changed from 1.1 to 3. I’m just raising this as more people are going to be upgrading their system in the coming weeks and might be faced with the same problem I had.

Hi!
I've got (I think) the same problem here:

$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be REMOVED:
  linux-image-5.13.0-39-generic
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 10,3 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 223072 files and directories currently installed.)
Removing linux-image-5.13.0-39-generic (5.13.0-39.44) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.13.0-39-generic
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.0-27-generic
Found initrd image: /boot/initrd.img-5.15.0-27-generic
Found linux image: /boot/vmlinuz-5.13.0-40-generic
Found initrd image: /boot/initrd.img-5.13.0-40-generic
/etc/grub.d/10_linux_proxy: 3: /etc/grub.d/bin/grubcfg_proxy: not found
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-5.13.0-39-generic (--remove):
 installed linux-image-5.13.0-39-generic package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 linux-image-5.13.0-39-generic
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)

Hi all,
found out, that grub customizer provides backup of all the files before its install in /etc/grub.d/backup. One workaround is simply just follow RESTORE_INSTRUCTIONS placed there. Obviously there is a problem, that all changes made to grub after customizer install will be lost, but after I realized, what a mess this package made, I refuse to use it again.
Not sure, if this is possible after running release upgrade. I landed here thanks to release notes, so I am doing this as a precaution.

EDIT: This is really NOT the way. I messed up my grub by overwriting /boot/grub and I am so stupid, that this is the only folder I forget to backup before these changes. I will have some fun today...

Hi Michael, my understanding is that proxified scripts are backed up in proxifiedScripts. If you look at the previous comments, you will find a short script I proposes which removes the proxy files and put the original ones back in (unless you saved new ones during upgrade). You might find the files you need there.

I saw your script but I wanted just to try it my way. When I found out that grub is broken and couldn't find some fast and straightforward solution, I just made a clean install. All custom config backed up and few apps I use was installed in no time. Anyway thank you for help.

No worries.

There is actually a fairly easy solution which doesn’t involve relying on the scripts above and that’s the one from the very first post which is in in brief:
- download and install libssl1.1 from the Impish archive (https://packages.ubuntu.com/impish/libssl1.1). It doesn’t look like there is anything preventing people from having both libssl1.1 and libssl3.
- install it (do ‘sudo dpkg -i package.deb’
- fix apt by running ‘ sudo apt --fix-broken install'.

Thanks for the solution Batwam, can confirm it worked with no issues

Newbie Ubuntu user here, didn´t have any clue of what to do.
Batwam solution worked like a charm and it was very easy. Thank you very much!

glad it helped you Davide.

Im really not sure why the Ubuntu team decided to make this a "Won't Fix" as the solution, it would have been great to get some sort of explaination as the package clearly isn't bahaving the way it should.

This bug should close now.
The new version grub-customizer had fixed this bug.

I also don't agree with the Won't fix resolution. I had to do a manual procedure to fix it rather than a smooth upgrade. These are the reasons people don't come to Linux.