Grub update breaks automated dist-upgrade scripts on AMI images

Chris uploaded a reversion to -proposed:

grub2 (1.99-21ubuntu3.2) precise-proposed; urgency=low

  * Revert previous SRU. This caused AMI cloud images to prompt about the
    changed configuration file, breaking automated upgrades. (LP: #1009294)
  * This reopens bug #978464, which will break Ubuntu 10.04->12.04 upgrades
    if user does not opt-in to reinstall grub-pc bootloader when prompted.

 -- Christopher James Halse Rogers <email address hidden> Wed, 06 Jun 2012 14:14:48 +1000

The analysis here is incorrect. The previous SRU affected the generation of /boot/grub/grub.cfg, but it did *not* touch /etc/default/grub, which is what the ucf prompt on upgrade is about. Reverting the SRU not only won't fix the issue, it most likely will cause the same issue *again* for affected users on upgrade.

I'm looking now to try to figure out why ucf is prompting here. It *shouldn't*, because there are no changes to the ucf template between the old and new versions of the package; so the fact that there is a prompt points to a bug in the ucf invocation, a bug in ucf, or a bug in the AMI having messed with ucf's internals inappropriately.

I have a modified /etc/default/grub here, and can't reproduce this issue by upgrading/downgrading between the precise and precise-proposed versions of grub2. I can only trigger a ucf prompt by calling 'dpkg-reconfigure grub-pc' and changing some values there.

I believe this can be reproduced by:

 - installing grub
 - changing one or more of the settings in /etc/default/grub that are managed by debconf (GRUB_CMDLINE_LINUX, GRUB_CMDLINE_LINUX_DEFAULT, GRUB_TIMEOUT, GRUB_HIDDEN_TIMEOUT), and *also* changing one or more line in /etc/default/grub that is *not* managed by debconf
 - upgrading grub

I haven't seen the /etc/default/grub from the cloud image yet. Can someone attach a copy of the file for comparison?

And as to upgrading without a prompt, the correct workaround would be to set DEBIAN_FRONTEND=noninteractive in the environment, to suppress all debconf prompting.

Indeed, here's the diff between the /etc/default/grub in the precise AMI, and the default one provided by grub. So this is a bug with how the AMI is built, not a bug in the grub package which has no way to automatically merge this using ucf.

The AMI image build probably needs to inject the grub settings via debconf preseeding, so that ucf as shipped in the image knows the intended answers to the debconf questions.

--- etc/default/grub 2012-06-01 22:37:36.000000000 -0700
+++ /usr/share/grub/default/grub 2012-05-17 00:07:17.000000000 -0700
@@ -4,11 +4,11 @@
 # info -f grub -n 'Simple configuration'

 GRUB_DEFAULT=0
-#GRUB_HIDDEN_TIMEOUT=0
-#GRUB_HIDDEN_TIMEOUT_QUIET=true
+GRUB_HIDDEN_TIMEOUT=0
+GRUB_HIDDEN_TIMEOUT_QUIET=true
 GRUB_TIMEOUT=10
 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
-GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0"
+GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
 GRUB_CMDLINE_LINUX=""

 # Uncomment to enable BadRAM filtering, modify to suit your needs
@@ -17,7 +17,7 @@
 #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

 # Uncomment to disable graphical terminal (grub-pc only)
-GRUB_TERMINAL=console
+#GRUB_TERMINAL=console

 # The resolution used on graphical terminal
 # note that you can use only modes which your graphic card supports via VBE

duplicate of bug #759545 ?!

Simple fix to avoid ucf prompts when grub *hasn't* changed its template is to run 'DEBIAN_FRONTEND=noninteractive dpkg-reconfigure grub-pc' in the image build scripts after /etc/default/grub has been populated. This will confirm the script populated answers to the debconf questions as the known-good values, and avoid the next grub upload showing up as having a changed template.

As outlined by Steve in comment #7, the fix for this is in the cloud image building tools to add:
  DEBIAN_FRONTEND=noninteractive dpkg-reconfigure grub-pc

A suitable validation of this is to run a instance pre-grub2 update, run this command, then upgrade. If it doesn't prompt, then we should have a suitable resolution.

However, this resolution will not resolve currently deployed AMI's before this is resolved, only latest builds. ~smoser is assigned to land this fix in the build tools for newer AMI's.

My testing implies that the fix I've committed to the build scripts [1] will resolve this issue.
Heres what I did to test:
 * ran 3 instances of ami-a29943cb (us-east-1 ebs/ubuntu-precise-12.04-amd64-server-20120424)
    a.) control instance [verified config-changed prompt]
          apt-get update && apt-get install grub-pc
    b.) no user-data instance
          DEBIAN_FRONTEND=noninteractive dpkg-reconfigure grub-pc
          apt-get update && apt-get install grub-pc
          # verified no grub prompt
     c.) user-data that disabled running of grub-dpkg module in cloud-init
          DEBIAN_FRONTEND=noninteractive dpkg-reconfigure grub-pc
          sudo cloud-init-cfg grub_dpkg
          apt-get update && apt-get install grub-pc
          # verified no grub prompt

'c' was necessary above, because cloud-init modifies debconf preseed values in grub on first boot. I was concerned that having run 'dpkg-reconfigure grub-pc' in the build process, then booting and having cloud-init modify some portion of grub-pc debconf, then upgrading could cause a similar issue. It did not.

After we get images built with this change in them, we can verify that the issue is fixed by:
 * launching an instance of quantal or precise with serial > 20120607
 * adding a repository with a grub update (possibly a ppa with just a grub rebuild)
 * apt-get update
 * apt-get install grub-pc

--
[1] http://bazaar.launchpad.net/~ubuntu-on-ec2/vmbuilder/automated-ec2-builds/revision/485

- create ppa with newer version of grub
  - built a version of grub > available in archive by simply changing the version in the changelog for quantal version '0.97-29ubuntu66' to '0.97-99ubuntu66'
  - upload that to a ppa (https://launchpad.net/~smoser/+archive/lp1009294-test)
  - populate both quantal and precise in that ppa. (copied quantal to precise)
- run instance of
  ami-808220e9 us-east-1 ebs/ubuntu-precise-daily-amd64-server-20120620
  ami-facb6993 us-east-1 ebs/ubuntu-quantal-daily-amd64-server-20120619
- sudo apt-add-repository ppa:smoser/lp1009294-test -y
- sudo apt-get update
- verify apt-cache policy has ppa version available
- sudo apt-get install grub-pc

There was no prompt in the 'apt-get install grub-pc'

So, based on that, I'm marking this 'Fix Released' in Ubuntu (quantal).
Once we have new AMIs released of 12.04 we can mark this fix-released for Precise.

I just verified this is fix-released in:
 us-east-1 ami-82fa58eb ebs/ubuntu-precise-12.04-amd64-server-20120616

So are we ok to put the fix for bug 978464 back into place?

If that breaks the currently deployed AMI images we can work around that in preinstall.

Do we have known hashes of the old-style deployed AMI images, but unmodified by the user?
Or a way to detect that we are running on AMI and preserve them.

Ignore me: the revert was in -proposed, but got deleted / pulled.

Hello Scott, or anyone else affected,

Accepted grub2 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/grub2/1.99-21ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Adam I don't understand the comments in the proposed SRU changelog, this bug is solved outside of grub and doesn't require a new reversion.

marking this verification-done. just because.
This but is not related to the upload.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug has reappeared in the most-recent EBS backed ubuntu-xenial AMI in AWS:

while creating images, I'm running:
`sudo apt-get -y upgrade`

to which I get prompted with:

```
1513162160,,ui,message, amazon-ebs: A new version of /boot/grub/menu.lst is available%!(PACKER_COMMA) but the version installed
1513162160,,ui,message, amazon-ebs: currently has been locally modified.
1513162160,,ui,message, amazon-ebs:
1513162160,,ui,message, amazon-ebs: 1. install the package maintainer's version
1513162160,,ui,message, amazon-ebs: 2. keep the local version currently installed
1513162160,,ui,message, amazon-ebs: 3. show the differences between the versions
1513162160,,ui,message, amazon-ebs: 4. show a side-by-side difference between the versions
1513162160,,ui,message, amazon-ebs: 5. show a 3-way difference between available versions
1513162160,,ui,message, amazon-ebs: 6. do a 3-way merge between available versions (experimental)
1513162160,,ui,message, amazon-ebs: 7. start a new shell to examine the situation
1513162160,,ui,message, amazon-ebs:
```

The solution is:
`sudo DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confold\" upgrade"`

Which was the initial fix proposed for this bug at:
https://askubuntu.com/questions/146921/how-do-i-apt-get-y-dist-upgrade-without-a-grub-config-prompt/147079#147079
back in 2012

Interesting, how this reappeared in 2017.

OK, I think this is actually bug 1485685 (i.e. the problem is grub-legacy-ec2 rather than grub itself), so I'm removing cloud-images from this bug and we'll track it over there.