<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"> <meta name=Generator content="Microsoft Word 12 (filtered medium)"> <style>  </style>  </head> <body lang=DE link=blue vlink=purple> <div class=Section1> Hello,<o:p></o:p> This is our setup: We have 3 Linux servers (2.6.18 Centos 5), clustered, with a clvmd running one “big” volume group (15 SCSI disks a 69,9 GB).<o:p></o:p> After we got an hardware I/O error on one disk out gfs filesystem began to loop. <o:p></o:p> So we stopped all services and we determined the corrupted disk (/dev/sdh) and my intention was to do the following:<o:p></o:p> <![if !supportLists]>- <![endif]>pvmove /dev/sdh<o:p></o:p> <![if !supportLists]>- <![endif]>vgreduce my_volumegroup /dev/sdh<o:p></o:p> <![if !supportLists]>- <![endif]>do an intensive hardware check on the volume<o:p></o:p> <o:p> </o:p> <o:p> </o:p> But: that’s what happened during pvmove –v /dev/sdh:<o:p></o:p> …….<o:p></o:p> /dev/sdh: Moved: 78,6%<o:p></o:p> /dev/sdh: Moved: 79,1%<o:p></o:p> /dev/sdh: Moved: 79,7%<o:p></o:p> /dev/sdh: Moved: 80,0%<o:p></o:p> Updating volume group metadata<o:p></o:p> Creating volume group backup "/etc/lvm/backup/myvol_vg" (seqno 46).<o:p></o:p> Error locking on node server1: device-mapper: reload ioctl failed: Das Argument ist ungültig<o:p></o:p> Unable to reactivate logical volume "pvmove0"<o:p></o:p> ABORTING: Segment progression failed.<o:p></o:p> Removing temporary pvmove LV<o:p></o:p> Writing out final volume group after pvmove<o:p></o:p> Creating volume group backup "/etc/lvm/backup/myvol_vg" (seqno 48).<o:p></o:p> [root@hpserver1 ~]# pvscan<o:p></o:p> PV /dev/cciss/c0d0p2 VG VolGroup00 lvm2 [33,81 GB / 0 free]<o:p></o:p> PV /dev/sda VG fit_vg lvm2 [68,36 GB / 0 free]<o:p></o:p> PV /dev/sdb VG fit_vg lvm2 [68,36 GB / 0 free]<o:p></o:p> PV /dev/sdc VG fit_vg lvm2 [68,36 GB / 0 free]<o:p></o:p> PV /dev/sdd VG fit_vg lvm2 [68,36 GB / 0 free]<o:p></o:p> PV /dev/sde VG fit_vg lvm2 [66,75 GB / 46,75 GB free]<o:p></o:p> PV /dev/sdf VG fit_vg lvm2 [68,36 GB / 0 free]<o:p></o:p> PV /dev/sdg VG fit_vg lvm2 [68,36 GB / 0 free]<o:p></o:p> PV /dev/sdh VG fit_vg lvm2 [68,36 GB / 58,36 GB free]<o:p></o:p> PV /dev/sdj VG fit_vg lvm2 [68,36 GB / 54,99 GB free]<o:p></o:p> PV /dev/sdi VG fit_vg lvm2 [68,36 GB / 15,09 GB free]<o:p></o:p> PV /dev/sdk1 VG fit_vg lvm2 [68,36 GB / 55,09 GB free]<o:p></o:p> Total: 12 [784,20 GB] / in use: 12 [784,20 GB] / in no VG: 0 [0 ]<o:p></o:p> <o:p> </o:p> That sounded bad, and I didn’t have any idea what to do, but read, that pvmove can start at the point it was, so I started pvmove againg and now pvmove could move all data.<o:p></o:p> pvscan and vgscan -vvv showed me, that all data were moved from the /dev/sdh volume to the other volumes.<o:p></o:p> <o:p> </o:p> To be sure I restarted my cluster nodes, but they encountered problems mounting the gfs filesystems. <o:p></o:p> I got this error:<o:p></o:p> <o:p> </o:p> [root@server1 ~]# /etc/init.d/clvmd stop<o:p></o:p> Deactivating VG myvol_vg: Volume group "myvol_vg" inconsistent<o:p></o:p> WARNING: Inconsistent metadata found for VG myvol_vg - updating to use version 148<o:p></o:p> 0 logical volume(s) in volume group "myvol_vg" now active<o:p></o:p> [ OK ]<o:p></o:p> Stopping clvm: [ OK ]<o:p></o:p> [root@server1 ~]# /etc/init.d/clvmd start<o:p></o:p> Starting clvmd: [ OK ]<o:p></o:p> Activating VGs: 2 logical volume(s) in volume group "VolGroup00" now active<o:p></o:p> Volume group "myvol_vg" inconsistent<o:p></o:p> WARNING: Inconsistent metadata found for VG myvol_vg - updating to use version 151<o:p></o:p> Error locking on node server1: Volume group for uuid not found: tGRfaK5aW00pFRXcLtrdHAw5a4GNDVBtuFZZe8QKoX8sVA0XRTNoDQVWVftk8cSa<o:p></o:p> Error locking on node server1: Volume group for uuid not found: tGRfaK5aW00pFRXcLtrdHAw5a4GNDVBtqDfFtrJTFTGuju8nNjwtCdPGnzP3hh8k<o:p></o:p> Error locking on node server1: Volume group for uuid not found: tGRfaK5aW00pFRXcLtrdHAw5a4GNDVBtc22hBY40phdVvVdFBFX28PvfF7JrlIYz<o:p></o:p> Error locking on node server1: Volume group for uuid not found: tGRfaK5aW00pFRXcLtrdHAw5a4GNDVBtWfJ1EqXJ309gO3Gx0ZvpNekrmHFo9u2V<o:p></o:p> Error locking on node server1: Volume group for uuid not found: tGRfaK5aW00pFRXcLtrdHAw5a4GNDVBtCP6czghnQFEjNdv9DF6bsUmnK3eJ5vKp<o:p></o:p> Error locking on node server1: Volume group for uuid not found: tGRfaK5aW00pFRXcLtrdHAw5a4GNDVBt0KNlnblpwOfcnqIjk4GJ662dxOsL70GF<o:p></o:p> 0 logical volume(s) in volume group "myvol_vg" now active<o:p></o:p> [ OK ]<o:p></o:p> <o:p> </o:p> As I take a look at it, these 6 volumes are exactly the LVs which should be found and where all datas are stored.<o:p></o:p> <o:p> </o:p> The next step was in the beginning step by step and in the end stupid try and error. <o:p></o:p> This was one of the first actions:<o:p></o:p> <o:p> </o:p> [root@hpserver1 ~]# vgreduce --removemissing myvol_vg<o:p></o:p> Logging initialised at Tue Jul 1 10:00:52 2008<o:p></o:p> Set umask to 0077<o:p></o:p> Finding volume group "myvol_vg"<o:p></o:p> Wiping cache of LVM-capable devices<o:p></o:p> WARNING: Inconsistent metadata found for VG myvol_vg - updating to use version 229<o:p></o:p> Volume group "myvol_vg" is already consistent<o:p></o:p> <o:p> </o:p> We tried to deactivate the volume via vgchange –n y myvol_vg, we tried to “removemissing” and sadly after a few concurrent tries (dmsetup info –c, dmsetup mknodes and vgchange –n y myvol_vg) we can access our LVs, but we still get this message and we don’t know why:<o:p></o:p> <o:p> </o:p> Volume group "myvol_vg" inconsistent<o:p></o:p> WARNING: Inconsistent metadata found for VG myvol_vg - updating to use version 228<o:p></o:p> <o:p> </o:p> I’m a little bit worried about our data,<o:p></o:p> <o:p> </o:p> Regards<o:p></o:p> Andreas<o:p></o:p> <o:p> </o:p> </div> </body> </html>