grub will only boot old kernels, new kernels lead to grub errors

Debugging this we have found that grub is getting the block numbers wrong.

vmlinuz-2.6.28-8-generic is the older good kerne, grub block dumpl:

    (hd0,0)402243640+96,402243744+6661

vmlinux-2.6.28-11-generic is the new broken kernel, grub block dump:

    (hd0,0)1546320+8,32+8,4096+8,402255872+8,4096+8,2792+8,402280448+4

Further looking at the filesystem with debugfs we find that the two inodes are in different formats:

-8-generic:
=======
Inode: 12566582 Type: regular Mode: 0644 Flags: 0x0
Generation: 3327387640 Version: 0x00000000
User: 0 Group: 0 Size: 3459328
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 6768
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x49a7d678 -- Fri Feb 27 12:03:04 2009
atime: 0x49a7d673 -- Fri Feb 27 12:02:59 2009
mtime: 0x49a4ddf1 -- Wed Feb 25 05:58:09 2009
Size of extra inode fields: 4
BLOCKS:
(0-11):50280455-50280466, (IND):50280467, (12-844):50280468-50281300
TOTAL: 846

-11-generic
=======
Inode: 12566556 Type: regular Mode: 0644 Flags: 0x80000
Generation: 2561410428 Version: 0x00000000:00000001
User: 0 Group: 0 Size: 3524096
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 6888
Fragment: Address: 0 Number: 0 Size: 0
 ctime: 0x49c20af8:66d68468 -- Thu Mar 19 09:06:00 2009
 atime: 0x49c223ad:599b6548 -- Thu Mar 19 10:51:25 2009
 mtime: 0x49c1831b:00000000 -- Wed Mar 18 23:26:19 2009
crtime: 0x49c20af8:244ae248 -- Thu Mar 19 09:06:00 2009
Size of extra inode fields: 28
BLOCKS:
(0-511):50281984-50282495, (512-860):50285056-50285404
TOTAL: 861

debugfs ls /boot gives me:

 12566529 (12) . 2 (12) ..
 12566559 (44) System.map-2.6.28-11-generic
 12566582 (52) vmlinuz-2.6.28-8-generic 12566562 (24) memtest86+.bin
 12566583 (32) config-2.6.28-8-generic
 12566584 (36) abi-2.6.28-8-generic 12566537 (12) grub
 12566585 (36) System.map-2.6.28-8-generic
 12566535 (28) abi-2.6.28-6-generic
 12566556 (36) vmlinuz-2.6.28-11-generic
 12566557 (32) config-2.6.28-11-generic
 12566558 (32) abi-2.6.28-11-generic
 12566563 (44) vmcoreinfo-2.6.28-11-generic
 12566586 (36) vmcoreinfo-2.6.28-8-generic
 12566565 (40) vmlinuz-2.6.28-11a-generic
 12566564 (44) initrd.img-2.6.28-11-generic
 12566530 (32) vmlinuz-2.6.28-6-generic
 12566534 (32) config-2.6.28-6-generic
 12566536 (36) System.map-2.6.28-6-generic
 12566561 (36) vmcoreinfo-2.6.28-6-generic
 12566533 (36) initrd.img-2.6.28-6-generic
 12566587 (3372) initrd.img-2.6.28-8-generic

This is from a mkfs.ext4 on a 4MB logical partition and then copying one file to it:

Inode: 12 Type: regular Mode: 0644 Flags: 0x80000
Generation: 740596109 Version: 0x00000001
User: 0 Group: 0 Size: 1871304
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 3658
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x49c27909 -- Thu Mar 19 17:55:37 2009
atime: 0x49c27909 -- Thu Mar 19 17:55:37 2009
mtime: 0x49c27909 -- Thu Mar 19 17:55:37 2009
BLOCKS:
(IND):1427, (0-14):1537-1551, (15-255):1186-1426, (256-511):3585-3840, (512-1023):3073-3584, (1024-1827):2049-2852
TOTAL: 1829

@Andy can you attached the output of dumpe2fs on this partition for me. Thanks!

# dumpe2fs -h /dev/sda1
dumpe2fs 1.41.4 (27-Jan-2009)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 5fb55e5f-9697-4ad3-ba1f-ad86a6d2c0ef
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file uninit_bg
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 19177472
Block count: 76690285
Reserved block count: 3834514
Free blocks: 71340966
Free inodes: 18732338
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1005
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Filesystem created: Thu Jan 29 12:53:03 2009
Last mount time: Thu Mar 19 16:17:09 2009
Last write time: Thu Mar 19 16:17:09 2009
Mount count: 4
Maximum mount count: 22
Last checked: Thu Mar 19 11:52:40 2009
Check interval: 15552000 (6 months)
Next check after: Tue Sep 15 12:52:40 2009
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 1f9263e0-52c8-4b69-95e6-3cfe2b83376e
Journal backup: inode blocks
Journal size: 128M

The bug occurs because ext4 can have inodes > 128 bytes and grub is assuming that inodes are 128 bytes in size.

Attached is the patch to fix this

Colin built me a stage2 with this fix, which I installed (remember to grub-install) and tested. Works well here.

This bug was fixed in the package grub - 0.97-29ubuntu51

---------------
grub (0.97-29ubuntu51) jaunty; urgency=low

  [ Colin Watson ]
  * more_scsi_disks.diff: Add support for up to 256 SCSI disk devices on
    Linux (LP: #335174). Due to BIOS disk numbering only at most 128 can be
    bootable, so ensure we don't crash if there are more.

  [ Colin King ]
  * ext4_fix_variable_sized_inodes.diff: Add support for ext4 variable
    sized inodes (LP: #345488). This is backwardly compatible with ext2
    ext3 fixed sized inodes.

 -- Colin Watson <email address hidden> Fri, 20 Mar 2009 16:12:22 +0000