<div dir="ltr"><div>Hello,</div><div><br></div><div>I've provisioned an LVM RAID 6 across 4 physical disks.
I'm trying to understand the RAID behavior after injecting the failure - removing physical disk /dev/sdc.
</div><div><br></div><div>pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf<br>vgcreate pool_vg /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg<br>lvcreate -l +100%FREE -n pool_lv --type raid6 pool_vg<br>mkfs.xfs /dev/pool_vg/pool_lv<br>echo "/dev/mapper/pool_vg-pool_lv /mnt xfs defaults,x-systemd.mount-timeout=30 0 0" >> /etc/fstab</div><div><br></div><div>Everything appears to be working fine:</div><div># pvs --segments -o pv_name,pv_size,seg_size,vg_name,lv_name,lv_attr,lv_size,seg_pe_ranges<br> PV PSize SSize VG LV Attr LSize PE Ranges<br> /dev/sda3 <49.00g <24.50g ubuntu-vg ubuntu-lv -wi-ao---- <24.50g /dev/sda3:0-6270<br> /dev/sda3 <49.00g 24.50g ubuntu-vg 0<br> /dev/sdc <100.00g 4.00m pool_vg [pool_lv_rmeta_0] ewi-aor--- 4.00m /dev/sdc:0-0<br> /dev/sdc <100.00g 99.99g pool_vg [pool_lv_rimage_0] iwi-aor--- 99.99g /dev/sdc:1-25598<br> /dev/sdd <100.00g 4.00m pool_vg [pool_lv_rmeta_1] ewi-aor--- 4.00m /dev/sdd:0-0<br> /dev/sdd <100.00g 99.99g pool_vg [pool_lv_rimage_1] iwi-aor--- 99.99g /dev/sdd:1-25598<br> /dev/sde <100.00g 4.00m pool_vg [pool_lv_rmeta_2] ewi-aor--- 4.00m /dev/sde:0-0<br> /dev/sde <100.00g 99.99g pool_vg [pool_lv_rimage_2] iwi-aor--- 99.99g /dev/sde:1-25598<br> /dev/sdf <100.00g 4.00m pool_vg [pool_lv_rmeta_3] ewi-aor--- 4.00m /dev/sdf:0-0<br> /dev/sdf <100.00g 99.99g pool_vg [pool_lv_rimage_3] iwi-aor--- 99.99g /dev/sdf:1-25598<br> /dev/sdg <100.00g 4.00m pool_vg [pool_lv_rmeta_4] ewi-aor--- 4.00m /dev/sdg:0-0<br> /dev/sdg <100.00g 99.99g pool_vg [pool_lv_rimage_4] iwi-aor--- 99.99g /dev/sdg:1-25598<br></div><div># lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg<br> LV Attr Cpy%Sync Health Devices<br> pool_lv rwi-aor--- 100.00 pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)<br> [pool_lv_rimage_0] iwi-aor--- /dev/sdc(1)<br> [pool_lv_rimage_1] iwi-aor--- /dev/sdd(1)<br> [pool_lv_rimage_2] iwi-aor--- /dev/sde(1)<br> [pool_lv_rimage_3] iwi-aor--- /dev/sdf(1)<br> [pool_lv_rimage_4] iwi-aor--- /dev/sdg(1)<br> [pool_lv_rmeta_0] ewi-aor--- /dev/sdc(0)<br> [pool_lv_rmeta_1] ewi-aor--- /dev/sdd(0)<br> [pool_lv_rmeta_2] ewi-aor--- /dev/sde(0)<br> [pool_lv_rmeta_3] ewi-aor--- /dev/sdf(0)<br> [pool_lv_rmeta_4] ewi-aor--- /dev/sdg(0)<br></div><div><br></div><div>After the /dev/sdc is removed and the system is rebooted, the RAID goes into "partial" health state and is no longer accessible.<br></div><div><br></div><div># lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg<br> WARNING: Couldn't find device with uuid 03KtEG-cJ5S-cMAD-RlL8-yBXM-jCav-EyD9I3.<br> WARNING: VG pool_vg is missing PV 03KtEG-cJ5S-cMAD-RlL8-yBXM-jCav-EyD9I3 (last written to /dev/sdc).<br> LV Attr Cpy%Sync Health Devices<br> pool_lv rwi---r-p- partial pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)<br> [pool_lv_rimage_0] Iwi---r-p- partial [unknown](1)<br> [pool_lv_rimage_1] Iwi---r--- /dev/sdd(1)<br> [pool_lv_rimage_2] Iwi---r--- /dev/sde(1)<br> [pool_lv_rimage_3] Iwi---r--- /dev/sdf(1)<br> [pool_lv_rimage_4] Iwi---r--- /dev/sdg(1)<br> [pool_lv_rmeta_0] ewi---r-p- partial [unknown](0)<br> [pool_lv_rmeta_1] ewi---r--- /dev/sdd(0)<br> [pool_lv_rmeta_2] ewi---r--- /dev/sde(0)<br> [pool_lv_rmeta_3] ewi---r--- /dev/sdf(0)<br> [pool_lv_rmeta_4] ewi---r--- /dev/sdg(0)</div><div><br></div><div><br></div><div>From what I understand, the RAID should be able to continue with a physical disk loss and be in a "degraded" state, not "partial", because the data is fully present on the surviving disks.</div><div><br></div><div></div><div>From /etc/lvm/lvm.conf:</div><div> # degraded<br> # Like complete, but additionally RAID LVs of segment type raid1,<br> # raid4, raid5, radid6 and raid10 will be activated if there is no<br> # data loss, i.e. they have sufficient redundancy to present the<br> # entire addressable range of the Logical Volume.<br> # partial<br> # Allows the activation of any LV even if a missing or failed PV<br> # could cause data loss with a portion of the LV inaccessible.<br> # This setting should not normally be used, but may sometimes<br> # assist with data recovery.</div><div><br></div><div>From <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_logical_volumes/assembly_configure-mange-raid-configuring-and-managing-logical-volumes">https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_logical_volumes/assembly_configure-mange-raid-configuring-and-managing-logical-volumes</a>:</div><div><br></div><div>"RAID is not like traditional LVM mirroring. LVM mirroring required
failed devices to be removed or the mirrored logical volume would hang.
RAID arrays can keep on running with failed devices. In fact, for RAID
types other than RAID1, removing a device would mean converting to a
lower level RAID (for example, from RAID6 to RAID5, or from RAID4 or
RAID5 to RAID0).</div><div><br></div><div>However, in my case, the RAID is not converted, it's simply not available.</div><div><br></div><div>This is running in a virtual machine on VMware ESXi 7, LVM version: 2.03.07(2) (2019-11-30).</div><div><br></div><div>Am I missing something obvious? Appreciate any insights.</div><div><br></div><div>Thanks,</div><div><br></div><div>Andrei<br>
</div></div>