[linux-lvm] LVM - Hitachi USP tier 2 to tier 1 migration then "couldn't find device with uuid"

[Zach Buckholz]

I am looking for feedback regarding an issue with LVM, Hitachi USP storage
virtualization and SAN migration from tier 2 to tier 1.

We had a customer complaining about disk IO latency so the SAN team used
some Hitachi USP magic to migrate the servers existing SAN disks from tier 2
to tier 1. Three nights later I rebooted the server during a maintenance
window and the server hung with a console message
“Couldn’t find all physical volumes for volume group vg2
Volume group “vg2” not found”

I commented out the necessary lines in fstab so I could get the server back
on the network and ssh into it.

Once in it appeared as if 4 disks were missing.

Metadata does not appear to be corrupt. It just isn’t there.

Fdisk –l shows 4 disks as not having any partition information.
“ doesn't contain a valid partition table “

Did something happen during the tier 2 to tier 1 migration several nights
earlier, and I caused it to surface with the reboot? Or do you think
something else caused this.

Has anyone experienced storage migration using an Hitachi USP with LVM? Are
there any known gotcha’s?

Thanks
Zach


[root at server001 ~]# pvscan
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  Couldn't find device with uuid 'tRZN01-N7bG-SKAw-D9Fd-RxRF-lS4D-U5QfPt'.
  Couldn't find device with uuid 'jRwTAu-N0Zs-KBrC-7d6q-uUOA-EHpx-QJeydD'.
  Couldn't find device with uuid 'iu1aeD-XVe3-qSHr-cea3-ATGP-ML76-2isZwS'.
  Couldn't find device with uuid 'CgJvgU-cND9-Wtfe-cx3Z-g6sH-0tan-Qa21YC'.
  PV unknown device      VG vg2   lvm2 [58.25 GB / 0    free]
  PV unknown device      VG vg2   lvm2 [58.25 GB / 0    free]
  PV unknown device      VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-30          VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-37          VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-33          VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-35          VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-40          VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-38          VG vg2   lvm2 [58.25 GB / 0    free]
  PV unknown device      VG vg2   lvm2 [58.25 GB / 0    free]
  PV /dev/dm-41          VG vg3   lvm2 [992.00 MB / 0    free]
  PV /dev/dm-29          VG vg3   lvm2 [992.00 MB / 0    free]
  PV /dev/dm-23          VG vg3   lvm2 [992.00 MB / 0    free]
  PV /dev/dm-22          VG vg3   lvm2 [992.00 MB / 0    free]
  PV /dev/dm-26          VG vg3   lvm2 [992.00 MB / 0    free]
  PV /dev/dm-21          VG vg3   lvm2 [992.00 MB / 0    free]
  PV /dev/cciss/c0d0p2   VG vg1   lvm2 [68.22 GB / 35.31 GB free]
  Total: 17 [656.53 GB] / in use: 17 [656.53 GB] / in no VG: 0 [0   ]
 
-- 
Zachary Buckholz, Senior Unix Systems Administrator
University of Phoenix  |  IT Operations
4025 S. Riverpoint Parkway │  Phoenix, Arizona 85040
phone: 602.557.6809  |  cell: 602.377.8023  | email:
zach.buckholz at apollogrp.edu

P Please consider the environment before printing this email.
 




This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-lvm/attachments/20090311/586cf818/attachment.htm>