[linux-lvm] missing physical volumes after upgrade to rhel 5.4
Julie Ashworth
julie.ashworth at berkeley.edu
Sat Sep 26 01:45:46 UTC 2009
hi Mark,
Thank you so much for the thorough test and
response.
I created a test volume with a gpt disk label to
stay consistent with how the failed volumes were
labelled.
The only difference between our experiences, that
I can tell, is that the pvcreate command failed
with an error similar to 'disk doesn't exist, or
is being filtered'. I believe its because of the
gpt label. I zeroed out the first 512 bytes of
the disk label, and continued with the commands
(similar to the commands you used):
dd if=/dev/zero of=/dev/sdk bs=512 count=1
pvcreate -u 5wJCEA-IDC1-5GhI-jnEs-EpYF-8Uf3-sqPL4O /dev/sdk
vgcfgrestore -f /etc/lvm/backup/jetstor642 jetstor642
vgchange -ay
I had no problems, and the volumes were
recovered. <much celebration>
Its possible that my setup is so non-standard,
that most people won't be affected (presumably
by the upgrade) as I was. Reasons my setup is
non-standard:
1) I used ext3 to format >12TB volumes (ext3
has a 8TB limit).
2) I used parted and gpt disk labels
3) I created PV on a whole disk.
Luckily, my request for a test environment was
approved (after this experience) so I can
attempt to replicate the problem and identify
the cause of the disk label corruption.
Unfortunately, the environment will certainly
arrive too late (I do work at a uni ;P) for me
to make a timely contribution.
Please let me know if I can provide any more
information.
And thanks again.
Best,
Julie
On 25-09-2009 10.52 +0100, Mark Round wrote:
> I just tried this myself....
>
> 1. First create a new PV on a whole disk, VG and LV
> # pvcreate /dev/sdc
> # vgcreate test /dev/sdc
> # lvcreate -L2G -n testlv test
>
> 2. Format the LV, mount it and copy some data to it (just a random
> tarball)
> # mke2fs -j /dev/test/testlv
> # mount /dev/test/testlv /mnt
> # tar -C /mnt -xvzf ~/iscsitarget-0.4.17.tar.gz
> # umount /mnt
> # e2fsck /dev/test/testlv
> e2fsck 1.39 (29-May-2006)
> /dev/test/testlv: clean, 87/262144 files, 25554/524288 blocks
>
> 3. So the LV is OK. Now I'll make sure there's a config backup, then
> wipe the PV label...
> # vgcfgbackup test
> Volume group "test" successfully backed up.
> # vgchange -an test
> 0 logical volume(s) in volume group "test" now active
> [root at europa ~]# pvremove -ff /dev/sdc
> Really WIPE LABELS from physical volume "/dev/sdc" of volume group
> "test" [y/n]? y
> WARNING: Wiping physical volume label from /dev/sdc of volume group
> "test"
> Labels on physical volume "/dev/sdc" successfully wiped
>
> 4. Now, I'll try to recreate the PV using the backup data, and see if
> the contents are intact.
> # pvcreate --uuid="A0LDgs-KMlm-QEBR-sGNW-7Rlf-j3aU-x2JUKY"
> --restorefile=/etc/lvm/backup/test /dev/sdc
> Couldn't find device with uuid
> 'A0LDgs-KMlm-QEBR-sGNW-7Rlf-j3aU-x2JUKY'.
> Physical volume "/dev/sdc" successfully created
> # vgcfgrestore -f /etc/lvm/backup/test test
> Restored volume group test
>
> 5. Check to see if we can see the previously created LV, and mount it
> # lvs
> LV VG Attr LSize Origin Snap% Move Log Copy% Convert
> testlv test -wi--- 2.00G
> # vgchange -ay test
> 1 logical volume(s) in volume group "test" now active
> # e2fsck /dev/test/testlv
> e2fsck 1.39 (29-May-2006)
> /dev/test/testlv: clean, 87/262144 files, 25554/524288 blocks
> # mount /dev/test/testlv /mnt
> # ls /mnt
> iscsitarget-0.4.17 lost+found
>
> So, YMMV but it appears from my experiments that this operation should
> be safe, and should recover your volumes. I am concerned though about
> the news that the RHEL 5.3->5.4 upgrade may have caused this, as we're
> looking at making the same upgrade before not too long. Do you have any
> suspicion as to why this may have happened ? Have you filed a bug with
> Red Hat ?
>
> Regards,
>
> -Mark
>
> -----Original Message-----
> From: linux-lvm-bounces at redhat.com [mailto:linux-lvm-bounces at redhat.com]
> On Behalf Of Julie Ashworth
> Sent: 24 September 2009 11:06
> To: linux-lvm at redhat.com
> Subject: [linux-lvm] missing physical volumes after upgrade to rhel 5.4
>
> I apologize for the cross-posting (to rhelv5-list).
> The lvm list is a more relevant list for my problem,
> and I'm sorry I didn't realize this sooner.
>
> After an upgrade from rhel5.3 -> rhel5.4 (and reboot)
> I can no longer see PVs for 3 fibre-channel storage
> devices.
>
> The operating system still see the disk:
> ----------------------
> # multipath -l
> mpath2 (2001b4d28000064db) dm-1 JetStor,Volume Set # 00
> [size=12T][features=0][hwhandler=0][rw]
> \_ round-robin 0 [prio=0][active]
> \_ 11:0:1:0 sdj 8:144 [active][undef]
> mpath16 (1ACNCorp_FF01000113200019) dm-2 ACNCorp,R_LogVol-despo
> [size=15T][features=0][hwhandler=0][rw]
> \_ round-robin 0 [prio=0][active]
> \_ 11:0:2:0 sdk 8:160 [active][undef]
> mpath7 (32800001b4d00cf5b) dm-0 JetStor,Volume Set 416F
> [size=12T][features=0][hwhandler=0][rw]
> \_ round-robin 0 [prio=0][active]
> \_ 11:0:0:0 sdi 8:128 [active][undef]
> ----------------------
>
>
> There are files in /etc/lvm/backup/ that contain the
> original volume information, e.g.
> ----------------------
> jetstor642 {
> id = "0e53Q3-evHX-I5f9-CWqf-NPcw-IqmC-0fVcTO"
> seqno = 2
> status = ["RESIZEABLE", "READ", "WRITE"]
> flags = []
> extent_size = 8192 # 4 Megabytes
> max_lv = 0
> max_pv = 0
>
> physical_volumes {
>
> pv0 {
> id = "5wJCEA-IDC1-5GhI-jnEs-EpYF-8Uf3-sqPL4O"
> device = "/dev/dm-7" # Hint only
>
> status = ["ALLOCATABLE"]
> flags = []
> dev_size = 31214845952 # 14.5355 Terabytes
> pe_start = 384
> pe_count = 3810405 # 14.5355 Terabytes
> }
> }
> ----------------------
>
> The devices were formatted using parted on the entire disk,
> i.e. I didn't create a partition.
> The partition table is "gpt" (possible label types are
> "bsd", "dvh", "gpt", "loop", "mac", "msdos", "pc98"
> or "sun".)
>
>
> partition table information for one of the devices is below:
> --------------------------
> # parted /dev/sdi
> GNU Parted 1.8.1
> Using /dev/sdi
> Welcome to GNU Parted! Type 'help' to view a list of commands.
> (parted) print
>
> Model: JetStor Volume Set 416F (scsi)
> Disk /dev/sdi: 13.0TB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
>
> Number Start End Size File system Name Flags
> --------------------------
>
> output of some commands:
>
> $ pvdisplay
> returns nothing (no error)
>
> $ lvs -a -o +devices
> returns nothing (no error)
>
> $ pvck -vvvvv /dev/sdb
> #lvmcmdline.c:915 Processing: pvck -vvvvv /dev/sdb
> #lvmcmdline.c:918 O_DIRECT will be used
> #config/config.c:950 Setting global/locking_type to 3
> #locking/locking.c:245 Cluster locking selected.
> #locking/cluster_locking.c:83 connect() failed on local socket:
> Connection
> +refused
> #config/config.c:955 locking/fallback_to_local_locking not found in
> +config: defaulting to 1
> WARNING: Falling back to local file-based locking.
> Volume Groups with the clustered attribute will be inaccessible.
> #config/config.c:927 Setting global/locking_dir to /var/lock/lvm
> #pvck.c:32 Scanning /dev/sdb
> #device/dev-cache.c:260 /dev/sdb: Added to device cache
> #device/dev-io.c:439 Opened /dev/sdb RO
> #device/dev-io.c:260 /dev/sdb: size is 25395814912 sectors
> #device/dev-io.c:134 /dev/sdb: block size is 4096 bytes
> #filters/filter.c:124 /dev/sdb: Skipping: Partition table
> signature
> +found
> #device/dev-io.c:485 Closed /dev/sdb
> #metadata/metadata.c:2337 Device /dev/sdb not found (or ignored by
> filtering).
> -------------------------
>
> from doing google searches, I found this gem to restore a
> PV:
> pvcreate --uuid "cqH4SD-VrCw-jMsN-GcwH-omCq-ThpE-dO9KmJ"
> --restorefile /etc/lvm/backup/vg_04 /dev/sdd1
>
>
> however, the man page says to 'use with care'. I don't want
> to lose data. Can anybody comment on how safe it would be to
> run this?
>
> Thanks in advance,
> Julie Ashworth
>
>
> --
> Julie Ashworth <julie.ashworth at berkeley.edu>
> Computational Infrastructure for Research Labs, UC Berkeley
> http://cirl.berkeley.edu/
> PGP Key ID: 0x17F013D2
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
---end quoted text---
--
Julie Ashworth <julie.ashworth at berkeley.edu>
Computational Infrastructure for Research Labs, UC Berkeley
http://cirl.berkeley.edu/
PGP Key ID: 0x17F013D2
More information about the linux-lvm
mailing list