[linux-lvm] Recovering from a failed pvmove

Tue Jun 28 18:12:41 UTC 2022

For a case like this vgcfgrestore is probably the best option.  man
vgcfgrestore.

You need to see if you have archived vg copies that you can revert to
before the "add" of the pv that went bad.

The archives are typically  in /etc/lvm/archiive/<vgname>* on RedHat
derivative OSes, not sure if they are different(and/or configured to exist)
on other distributions.

grep -i before /etc/lvm/archive/<vgname>* and see which archive was made
before the initial pv addition.  vgcfgrestore -f <goodconfig> should work
but I usually have to adjust command line options to get it work when I
have used it to revert configs.  I think in that case it will find the vg
and pvid correctly.  No cleanup should need to be done so long as the other
device is completely gone.

And you will probably need to answer some prompts and warnings, and then
reboot the machine, and/or do this all under a livecd rescue boot.

What kind of cheap ssd were you using?  I have had really bad luck with
ones without RAM.  I RMA'ed one that failed in under a week and the new one
also failed in a very similar way in under a week.

On Tue, Jun 28, 2022 at 1:38 AM Roger James <roger at beardandsandals.co.uk>
wrote:

> Hi,
>
> I am struggling to recover from a failed pvmove. Unfortunately I only have
> a limited knowledge of lvm. I setup my lvm configuration many years ago.
>
> I was trying to move a lv to a SSD using pvmove. Unfortunately my brand
> new SSD choose that moment to fail (never buy cheap SSDs, lesson learnt!").
>
> This is the current status.
>
> roger at dragon:~$ sudo pvs
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   PV VG Fmt Attr PSize PFree
>   /dev/sda1 wd lvm2 a-- <465.76g 0
>   /dev/sdb1 wd lvm2 a-- <465.76g <80.45g
>   /dev/sdc2 wd lvm2 a-- 778.74g 278.74g
>   /dev/sdd1 wd lvm2 a-- <465.76g 0
>   [unknown] wd lvm2 a-m <784.49g 685.66g
> roger at dragon:~$ sudo lvs
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
>   home wd -wi------- 1.46t
>   root wd -wI-----p- <108.83g
>
>   swap wd -wi------- 8.00g
>   work wd -wi------- 200.00g
> roger at dragon:~$ sudo vgs
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   VG #PV #LV #SN Attr VSize VFree
>   wd 5 4 0 wz-pn- 2.89t 1.02t
>
> This is a recap of what I have tried so far.
>
> roger at dragon:~$ sudo pvmove --abort
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   LVM command executed by lvmpolld failed.
>   For more information see lvmpolld messages in syslog or lvmpolld log
> file.
> roger at dragon:~$ sudo vgreduce --removemissing wd
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: Partial LV root needs to be repaired or removed.
>   WARNING: Partial LV pvmove0 needs to be repaired or removed.
>   There are still partial LVs in VG wd.
>   To remove them unconditionally use: vgreduce --removemissing --force.
>   To remove them unconditionally from mirror LVs use: vgreduce
> --removemissing --mirrorsonly --force.
>   WARNING: Proceeding to remove empty missing PVs.
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> roger at dragon:~$ sudo lvchange -an wd/root
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
> roger at dragon:~$ sudo vgreduce --removemissing wd
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: Partial LV root needs to be repaired or removed.
>   WARNING: Partial LV pvmove0 needs to be repaired or removed.
>   There are still partial LVs in VG wd.
>   To remove them unconditionally use: vgreduce --removemissing --force.
>   To remove them unconditionally from mirror LVs use: vgreduce
> --removemissing --mirrorsonly --force.
>   WARNING: Proceeding to remove empty missing PVs.
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
> roger at dragon:~$ sudo lvremove wd/pvmove0
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   WARNING: VG wd is missing PV uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ
> (last written to [unknown]).
>   WARNING: Couldn't find device with uuid
> uMtjop-PmMT-603f-GWWQ-fR4f-s4Sw-XSKNXZ.
>   Can't remove locked logical volume wd/pvmove0.
>
> I am quite happy to loose the root lv, I just need the home and work lvs.
> What am I missing?
>
> Help!
>
> Roger
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm at redhat.com
> https://listman.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-lvm/attachments/20220628/72baaa1c/attachment.htm>