Kernel panic: LVM not working in kernel since 2.6.20-1.3069 -- Repost

[Andy Baumhauer]

(Repost to fit into 64KB limit)

Kernel-2.6.20-1.3069 on x86_64 is the last kernel that has booted on my 
machine.

I have two SATA drives with /dev/sda1 as /boot and /dev/sda2 using LVM
with a non-default setup. /dev/sdb contains an FC6 install.

/dev/mapper/vg_host-lv_root labeled as /
/dev/mapper/vg_host-lv_var is a LUKS partition which decrypted maps to
/dev/mapper/var
/dev/mapper/vg_host-lv_home is a LUKS partion which decrypted maps to
/dev/mapper/home

swap is a dm-crypt partition with /dev/urandom as the key, relabeled at
boot to be "swap"

Keys are on a LUKS USB SD card, which I access at boot and decrypt (made
some changes to /etc/rc.sysinit before init_crypto() is called).

Don't be turned off by the encrypted disks. I'm 99% sure this isn't the
source of the panic, since the boot process isn't even making it to the
point of mounting the root filesystem (which isn't encrypted).

In kernel 2.6.20-1.3069, everything works fine. The system boots and
discovers the drives and the LVM partitions, then calls /etc/rc.sysinit
and boot continues after I decrypt the volumes.

In all kernels since then, the system panics before starting
/etc/rc.sysinit. It appears from watching the device discovery that LVM
isn't being initialized, so the later stages of init aren't seeing the
drives. I'm quite sure it isn't my two encrypted volumes, but I'm not
sure it isn't the non-standard naming of my volume groups (but I've
always renamed volume groups from VolumeGroup00 to something else).

Anyone have an idea? I need to get either a copy of the kernel-headers
for 3069, or get a new working kernel. I'm stuck trying to compile some
code that needs include/linux/limits.h and I don't have the headers, and
I can't get any new kernel to boot.

Here are the last few lines that I copied down before the panic (using
3084 kernel).

Loading mbcache.ko module
Loading jbd.ko module
Loading ext3.ko module
Loading dm-mod.ko module
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialized:
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Trying to resume device (LABEL=swap)
Creating root device
Mounting root filesystem
mount: could not find filesystem '/dev/root'
Setting up other filesystems
Setting up new root fs
setuproot: moving /dev/ failed: no such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: no such file or directory
setuproot: error mounting /sys no such file or directory
Switching to new root and running init
Unmounting old /dev
Unmounting old /proc
Unmounting old /sys
switchroot: mount failed: no such file or directory
Booting has failed;
Kernel panic - not syncing: Attempted to kill init!


Here are my configs:

/boot/grub/menu.lst:
title Fedora (2.6.20-1.3084.fc7)
root (hd0,0)
kernel /vmlinuz-2.6.20-1.3084.fc7 ro root=LABEL=/ vga=792 rhgb quiet
initrd /initrd-2.6.20-1.3084.fc7.img

/etc/crypttab:
swap /dev/mapper/vg_host-lv_swap /dev/urandom
swap,cipher=twofish-cbc-essiv:sha256
var /dev/mapper/vg_host-lv_var /keyring/var.key
home /dev/mapper/vg_host-lv_home /keyring/home.key

/etc/fstab:
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
# Encrypted filesystems
LABEL=/home /home ext3 defaults 1 2
LABEL=/var /var ext3 defaults 1 2
LABEL=swap swap swap defaults 0 0

/dev/mapper: (on kernel 3069)
total 0
drwxr-xr-x 2 root root 280 2007-04-18 16:02 .
drwxr-xr-x 16 root root 5160 2007-04-19 12:19 ..
crw------- 1 root root 10, 63 2007-04-18 12:01 control
brw-rw---- 1 root disk 253, 6 2007-04-18 16:02 home
brw-rw---- 1 root disk 253, 11 2007-04-18 16:02 swap
brw-rw---- 1 root disk 253, 5 2007-04-18 16:02 var
brw-rw---- 1 root disk 253, 1 2007-04-18 12:01 vg_host-lv_home
brw-rw---- 1 root disk 253, 0 2007-04-18 16:02 vg_host-lv_root
brw-rw---- 1 root disk 253, 3 2007-04-18 12:01 vg_host-lv_swap
brw-rw---- 1 root disk 253, 2 2007-04-18 12:01 vg_host-lv_var