A friend of mine recently built a fileserver from some spare stuff he had around along with a spare RHEL3 CD from his work. He has no RHEL entitlements, so no binary updates. (I would have encouraged him to at least use an upgradeable system, but that's neither here nor there, now.) As for the system, it's very basic with a system hard drive, a DVD drive, PCI IDE card, and five drives dedicated to a RAID5 volume. He followed the software RAID and LVM how-tos, eventually building a 930+GB RAID5 volume md0. He then used md0 as the sole PV to build a "raid_discs" VG. This VG was split into three LVs: home, music, and pics. Each LV was formatted EXT3 and set to mount in /etc/fstab: /dev/raid_discs/music /raid/music ext3 defaults 0 0 /dev/raid_discs/pictures /raid/pictures ext3 defaults 0 0 /dev/raid_discs/home /raid/home ext3 defaults 0 0 This done, he proceeded to copy 260G of media to the server. Thinking everything we living safely on a RAID5 device now, he proceeded to delete the sources. *sigh* Everything worked fine until he rebooted the server (cleanly, I'm told). After the machine came up, it wouldn't mount his LVs. The RAID device is there: [root@Server lvmconf]# cat /proc/mdstat Personalities : [raid5] read_ahead 1024 sectors Event: 1 md0 : active raid5 hdh[4] hdg[3] hdf[2] hde[1] hdc[0] 976793600 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU] unused devices: <none> From what I can tell, though, it doesn't look like LVM is seeing the RAID device as a PV anymore: [root@Server lvmconf]# pvdisplay -vv /dev/md0 pvdisplay -- checking physical volume name "/dev/md0" pvdisplay -- reading data of physical volume "/dev/md0" from disk pvdisplay -- no physical volume identifier on "/dev/md0" Am I reading this correctly? I tried to do a vgcfgrestore and got this: [root@Server lvmconf]# vgcfgrestore -n raid_discs -vv /dev/md0 vgcfgrestore -- INFO: using backup file "/etc/lvmconf/raid_discs.conf" vgcfgrestore -- locking logical volume manager vgcfgrestore -- restoring volume group "raid_discs" from "/etc/lvmconf/raid_discs.conf" vgcfgrestore -- checking existence of "/etc/lvmconf/raid_discs.conf" vgcfgrestore -- reading volume group data for "raid_discs" from "/etc/lvmconf/raid_discs.conf" vgcfgrestore -- reading physical volume data for "raid_discs" from "/etc/lvmconf/raid_discs.conf" vgcfgrestore -- reading logical volume data for "raid_discs" from "/etc/lvmconf/raid_discs.conf" vgcfgrestore -- checking volume group consistency of "raid_discs" vgcfgrestore -- checking volume group consistency of "raid_discs" vgcfgrestore -- reading physical volume "/dev/md0" vgcfgrestore -- ERROR "pv_read(): PV identifier invalid" reading physical volume "/dev/md0" Is there anything that we can do to save his data? I know that with partition tables, if you know them in advance, you can re-write them and, if you're exact, you won't lose any data. Is it possible to do the same with pvcreate? Could I re-init the PV, then have the VG suddenly be happy? Suggestions? Would it help matters to boot off a live CD like Helix and try recovery that way? I'm a bit hesitant to do that as I'd rather try to get things working with the tools that created the VG in the first place and I'm not sure how nicely LVM2 would play with LVM. OTOH, since we're dealing with base RHEL3, we could also be hitting a bug that was later patched. (After recovery, I'm getting him to change the OS ASAP.) Thank you so much. I really hope he hasn't borked his pics of his kids. pete