root partition may not be grown

This attachment does the following to a given block device input
  * create a partition table for the block device with first partition using roughly half disk space
  * mkfs first partition
  * mount partition
  * run growpart --dry-run
  * umount partition
  * run growpart

If you run it in a loop, it will most likely fail over 50 runs:
  for((i=0;i<50;i++)); do sudo ./go-sfdisk-crazy /dev/vdb; done

I've tested/developed this on openstack instance pointing it at ephemeral disk (/dev/vdb).

note, it does seem that maybe this should be calling partprobe.

This snippet seems to reproduce the issue for me sporadically, and much simpler than the attached script. It assumes there is a filesystem already on /dev/vdb1 and /dev/vdb is partitioned already.

sudo sh -c 'd=$1; p=${d}1; mp=/mnt; cleanup() { umount $mp >/dev/null 2>&1; } ; trap cleanup EXIT; umount $p >/dev/null 2>&1; while : ; do mount $p $mp && umount $mp && sfdisk --re-read $d|| exit 1; echo -n .; done' -- /dev/vdb

Experimenting a bit with the snipped from comment #5, I can see the same happening for a xen based guest. One interesting fact here is that this is temporary only. So adding a sleep 1 between the umount and the sfdisk command will suppress that failure.
So it feels like umount returns while the kernel internally still cleans up. And trying to revalidate the partition table while this is not finished fails.

Status changed to 'Confirmed' because the bug affects multiple users.

Adding a "sync" call between umount and sfdisk seems to avoid the problem. At least the loop ran until I stopped it instead of failing very early (between one and three iterations).

The snippet from comment #5 actually has a major flaw. Since there is no waiting in the loop after doing the sfdisk --re-read and so we do actually run into races with the udev triggered partition checks (blkid/cdrom_id). Adding a udevadm settle after the sfdisk call does prevent all failures I saw before.

Similarly the go-fsdisk-crazy script. Not sure why I had no issues before but this morning I saw it fail. Looking closer the failures started before the grow part, though. So it usually had issues on the partition setup and then all the rest went wrong too. And that went away after adding a settle after the resize.

There also was something very wrong with the CHS output which seemed to change (even later on the successful runs). Not yet sure I understand what is happening there. But overall I think we may be hunting in the wrong direction. Looking at the initial failing output it seems like there is just the message about sfdisk failing. Then (and I think that is in the recovery) re-read of the partition table fails. Would it be possible that sfdisk reported failure but triggered a partition table update anyway, then in the recovery the call fails because we still race against udev commands triggered by the previous call?

Sidenote: it seems the CHS output for a partially used drive will not necessarily match the drive. Even when fdisk did show the drive values when creating the partial partition. Adding a second partition that fills the complete disk will fix that... And sfdisk seems never happy with the values when doing a sfdisk -l...

regarding CHS, i'm fairly sure that its not important. the only reason I actually use it is so that I can specify (and see) values in sectors rather than at a larger unit.

Sorry to have brought us down the wrong road with some of my snippets above.

I just uploaded a new cloud-utils (growpart) which will show the original output of the failed sfdisk command.
In the original summary of the bug, all that is present is the output of the *restore* command after the sfdisk to change it failed.

Next builds will have new growpart and we can upload and try to catch failure again.

I've now uploaded some new images, with the improved growroot, and the output of a failed resize looks like this:

------
Begin: Running /scripts/local-bottom ... [ 0.892501] vda: vda1
GROWROOT: WARNING: resize failed: attempt to resize /dev/vda failed. sfdisk output below:
| Checking that no-one is using this disk right now ...
| OK
|
| Disk /dev/vda: 1305 cylinders, 255 heads, 63 sectors/track
| Old situation:
| Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
|
| Device Boot Start End #cyls #blocks Id System
| /dev/vda1 * 1 260 260 2088450 83 Linux
| /dev/vda2 0 - 0 0 0 Empty
| /dev/vda3 0 - 0 0 0 Empty
| /dev/vda4 0 - 0 0 0 Empty
| New situation:
| Units = sectors of 512 bytes, counting from 0
|
| Device Boot Start End #sectors Id System
| /dev/vda1 * 16065 20964824 20948760 83 Linux
| /dev/vda2 0 - 0 0 Empty
| /dev/vda3 0 - 0 0 Empty
| /dev/vda4 0 - 0 0 Empty
| Successfully wrote the new partition table
|
| Re-reading the partition table ...
| BLKRRPART: Device or resource busy
| The command to re-read the partition table failed.
| Run partprobe(8), kpartx(8) or reboot your system now,
| before using mkfs
| If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
| to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
| (See fdisk(8).)
FAILED: failed to resize
***** WARNING: Resize failed, attempting to revert ******
Re-reading the partition table ...
BLKRRPART: Device or resource busy
The command to re-read the partition table failed.
Run partprobe(8), kpartx(8) or reboot your system now,
before using mkfs
***** Appears to have gone OK ****
[ 1.006837] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
-------

So, growroot tried to resize, the resize failed (with BLKRRPART busy). Then it tried to re-set it back to the original, and that BLKRRPART failed also.

So something other than the mount has the block device open.

I've just uploaded what we believe/hope will be a fix to this issue.
After much discussion, we decided to add a 'udevadm settle' prior to invoking 'growpart'.

The suspicion is that in the initial flurry of device activation, one of the udev spawned events still had an open filehandle on the block device (/dev/vda in this case) when sfdisk made the BLKRRPART call.

Unfortunately we did not definitively prove this, or even catch a 'ps' output where there was a failure. The one thing I fear here is that udevadm settle is quite likely to add enough delay to hide the issue if it is not the real solution.

This bug was fixed in the package cloud-initramfs-tools - 0.4ubuntu1

---------------
cloud-initramfs-tools (0.4ubuntu1) precise; urgency=low

  * growroot: run 'udevadm settle' before attempting growpart to
    allow initial events to finish (LP: #937352)
 -- Scott Moser <email address hidden> Thu, 23 Feb 2012 22:19:20 -0500

Fix committed did not seem to fix this.

There is a lot of confusing information in the above comments.
To sum up the solution that I'll be uploading shortly, for precise we did:
 * add to cloud-initramfs-growroot a 'udevadm settle' before invoking growpart to apply the change. This ensures that any disk events for the root block device are finished before sfdisk is run.
 * in cloud-utils's growpart, use '--no-reread' argument to sfdisk when making changes to the disk.

This bug was fixed in the package cloud-utils - 0.25-0ubuntu4

---------------
cloud-utils (0.25-0ubuntu4) precise; urgency=low

  * growpart: invoke sfdisk with '--no-reread' to avoid udev race
    conditions (LP: #937352)
 -- Scott Moser <email address hidden> Tue, 28 Feb 2012 14:40:51 -0500