<div dir="ltr">Hi!<br><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 5, 2016 at 3:53 PM Giuliano Procida <<a href="mailto:giuliano.procida@gmail.com">giuliano.procida@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 4 October 2016 at 23:14, Slava Prisivko <<a href="mailto:vprisivko@gmail.com" class="gmail_msg" target="_blank">vprisivko@gmail.com</a>> wrote:<br class="gmail_msg">
>> vgextend --restoremissing<br class="gmail_msg">
><br class="gmail_msg">
> I didn't have to, because all the PVs are present:<br class="gmail_msg">
><br class="gmail_msg">
> # pvs<br class="gmail_msg">
> PV VG Fmt Attr PSize PFree<br class="gmail_msg">
> /dev/sda2 vg lvm2 a-- 1.82t 1.10t<br class="gmail_msg">
> /dev/sdb2 vg lvm2 a-- 3.64t 1.42t<br class="gmail_msg">
> /dev/sdc2 vg lvm2 a-- 931.51g 195.18g<br class="gmail_msg">
<br class="gmail_msg">
Double-check in the metadata for MISSING. This is what I was hoping<br class="gmail_msg">
might be in your /etc/lvm/backup file.<br class="gmail_msg">
<br class="gmail_msg">
>> Actually, always run LVM commands with -v -t before really running them.<br class="gmail_msg">
><br class="gmail_msg">
> Thanks! I had backed up the rmeta* and rimage*, so I didn't feel the need<br class="gmail_msg">
> for using -t. Am I wrong?<br class="gmail_msg">
<br class="gmail_msg">
Well, some nasty surprises may be avoidable (particularly if also using -f).<br class="gmail_msg">
<br class="gmail_msg">
> Yes, I've noticed it. The problem was a faulty SATA cable (as I learned<br class="gmail_msg">
> later), so when I switched the computer on for the first time, /dev/sda was<br class="gmail_msg">
> missing (in the current device allocation). I switched off the computer,<br class="gmail_msg">
> swapped the /dev/sda and /dev/sdb SATA cable (without thinking about the<br class="gmail_msg">
> consequences) and switched it on. This time the /dev/sdb was missing. I<br class="gmail_msg">
> replaced the faulty cable with a new one and switched the machine back on.<br class="gmail_msg">
> This time sda, sdb and sdc were all present, but the RAID went out-of-sync.<br class="gmail_msg">
<br class="gmail_msg">
In swapping the cables, you may have changed the sd{a,b,c} enumeration<br class="gmail_msg">
but this will have no impact on the UUIDs that LVM uses to identify<br class="gmail_msg">
the PVs.<br class="gmail_msg"></blockquote><div>That's right, but the images went out-of-sync because during the first boot only sdb and sdc were present (so the content of sda should have been implied), during the second boot only sda and sdc were present (so the content of sdb should have been implied), but when I replaced the cable there was a conflict between these three.<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
> I'm pretty sure there were very few (if any) writing operations during the<br class="gmail_msg">
> degraded operating mode, so the I could recover by rebuilding the old mirror<br class="gmail_msg">
> (sda) using the more recent ones (sdb and sdc).<br class="gmail_msg">
<br class="gmail_msg">
Agreed, based on your check below.<br class="gmail_msg">
<br class="gmail_msg">
> Thanks, I used your raid5_parity_check.cc utility with the default stripe<br class="gmail_msg">
> size (64 * 1024), but it actually doesn't matter since you're just<br class="gmail_msg">
> calculating the total xor and the stripe size acts as a buffer size for<br class="gmail_msg">
> that.<br class="gmail_msg">
<br class="gmail_msg">
[I was little surprised to discover that RAID 6 works as a byte erasure code.]<br class="gmail_msg">
<br class="gmail_msg">
The stripe size and layout matters once if you want to adapt the code<br class="gmail_msg">
to extract or repair the data.<br class="gmail_msg">
<br class="gmail_msg">
> I get three unsynced stripes out of 512 (32 mib / 64 kib), but I would like<br class="gmail_msg">
> to try to reconstruct the test_rimage_1 using h the other two. Just in case,<br class="gmail_msg">
> here are the bad stripe numbers: 16, 48, 49.<br class="gmail_msg">
<br class="gmail_msg">
I've updated the utility (this is for raid5 = raid5_ls). Warning: not<br class="gmail_msg">
tested on out-of-sync data.<br class="gmail_msg">
<br class="gmail_msg">
<a href="https://drive.google.com/open?id=0B8dHrWSoVcaDYXlUWXEtZEMwX0E" rel="noreferrer" class="gmail_msg" target="_blank">https://drive.google.com/open?id=0B8dHrWSoVcaDYXlUWXEtZEMwX0E</a></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
<br class="gmail_msg">
# Assume the first sub LV has the out-of-date data and dump the<br class="gmail_msg">
correct(ed) LV content.<br class="gmail_msg">
./foo stripe $((64*1024)) repair 0 /dev/${lv}_rimage_* | cmp - /dev/${lv}<br class="gmail_msg"></blockquote><div>Thanks!</div><div><br></div><div>I tried to reassemble the array using 3 different pairs of correct LV images, but it doesn't work (I am sure because I cannot luksOpen a LUKS image which is in the LV, which is almost surely uncorrectable).<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
>> > The output of various commands is provided below.<br class="gmail_msg">
>> ><br class="gmail_msg">
>> > # lvs -a -o +devices<br class="gmail_msg">
>> ><br class="gmail_msg">
>> > test vg rwi---r--- 64.00m<br class="gmail_msg">
>> > test_rimage_0(0),test_rimage_1(0),test_rimage_2(0)<br class="gmail_msg">
>> > [test_rimage_0] vg Iwi-a-r-r- 32.00m /dev/sdc2(1)<br class="gmail_msg">
>> > [test_rimage_1] vg Iwi-a-r-r- 32.00m<br class="gmail_msg">
>> > /dev/sda2(238244)<br class="gmail_msg">
>> > [test_rimage_2] vg Iwi-a-r-r- 32.00m<br class="gmail_msg">
>> > /dev/sdb2(148612)<br class="gmail_msg">
>> > [test_rmeta_0] vg ewi-a-r-r- 4.00m /dev/sdc2(0)<br class="gmail_msg">
>> > [test_rmeta_1] vg ewi-a-r-r- 4.00m<br class="gmail_msg">
>> > /dev/sda2(238243)<br class="gmail_msg">
>> > [test_rmeta_2] vg ewi-a-r-r- 4.00m<br class="gmail_msg">
>> > /dev/sdb2(148611)<br class="gmail_msg">
<br class="gmail_msg">
The extra r(efresh) attributes suggest trying a resync operation which<br class="gmail_msg">
may not be possible on inactive LV.<br class="gmail_msg">
I missed that the RAID device is actually in the list.<br class="gmail_msg">
<br class="gmail_msg">
> After cleaning the dmsetup table of test_* and trying to lvchange -ay I get<br class="gmail_msg">
> practically the same:<br class="gmail_msg">
> # lvchange -ay vg/test -v<br class="gmail_msg">
[snip]<br class="gmail_msg">
> device-mapper: reload ioctl on (253:87) failed: Invalid argument<br class="gmail_msg">
> Removing vg-test (253:87)<br class="gmail_msg">
><br class="gmail_msg">
> device-mapper: table: 253:87: raid: Cannot change device positions in RAID<br class="gmail_msg">
> array<br class="gmail_msg">
> device-mapper: ioctl: error adding target to table<br class="gmail_msg">
<br class="gmail_msg">
This error occurs when the sub LV metadata says "I am device X in this<br class="gmail_msg">
array" but dmsetup is being asked to put the sub LV at different<br class="gmail_msg">
position Y (alas, neither are logged). With lots of -v and -d flags<br class="gmail_msg">
you can get lvchange to include the dm table entries in the<br class="gmail_msg">
diagnostics.<br class="gmail_msg"></blockquote><div>This is as useful as it gets (-vvvv -dddd):</div><div><div> Loading vg-test_rmeta_0 table (253:35)</div><div> Adding target to (253:35): 0 8192 linear 8:34 2048</div><div> dm table (253:35) [ opencount flush ] [16384] (*1)</div><div> Suppressed vg-test_rmeta_0 (253:35) identical table reload.</div><div> Loading vg-test_rimage_0 table (253:36)</div><div> Adding target to (253:36): 0 65536 linear 8:34 10240</div><div> dm table (253:36) [ opencount flush ] [16384] (*1)</div><div> Suppressed vg-test_rimage_0 (253:36) identical table reload.</div><div> Loading vg-test_rmeta_1 table (253:37)</div><div> Adding target to (253:37): 0 8192 linear 8:2 1951688704</div><div> dm table (253:37) [ opencount flush ] [16384] (*1)</div><div> Suppressed vg-test_rmeta_1 (253:37) identical table reload.</div><div> Loading vg-test_rimage_1 table (253:38)</div><div> Adding target to (253:38): 0 65536 linear 8:2 1951696896</div><div> dm table (253:38) [ opencount flush ] [16384] (*1)</div><div> Suppressed vg-test_rimage_1 (253:38) identical table reload.</div><div> Loading vg-test_rmeta_2 table (253:39)</div><div> Adding target to (253:39): 0 8192 linear 8:18 1217423360</div><div> dm table (253:39) [ opencount flush ] [16384] (*1)</div><div> Suppressed vg-test_rmeta_2 (253:39) identical table reload.</div><div> Loading vg-test_rimage_2 table (253:40)</div><div> Adding target to (253:40): 0 65536 linear 8:18 1217431552</div><div> dm table (253:40) [ opencount flush ] [16384] (*1)</div><div> Suppressed vg-test_rimage_2 (253:40) identical table reload.</div><div> Creating vg-test</div><div> dm create vg-test LVM-Pgjp5f2PRJipxvoNdsYmq0olg9iWwY5pJjiPmiesfxvdeF5zMvTsJC6vFfqNgNnZ [ noopencount flush ] [16384] (*1)</div><div> Loading vg-test table (253:84)</div><div> Adding target to (253:84): 0 131072 raid raid5_ls 3 128 region_size 1024 3 253:35 253:36 253:37 253:38 253:39 253:40</div><div> dm table (253:84) [ opencount flush ] [16384] (*1)</div><div> dm reload (253:84) [ noopencount flush ] [16384] (*1)</div><div> device-mapper: reload ioctl on (253:84) failed: Invalid argument</div></div><div><br></div><div>I don't see any problems here.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
You can check the rmeta superblocks with<br class="gmail_msg">
<a href="https://drive.google.com/open?id=0B8dHrWSoVcaDUk0wbHQzSEY3LTg" rel="noreferrer" class="gmail_msg" target="_blank">https://drive.google.com/open?id=0B8dHrWSoVcaDUk0wbHQzSEY3LTg</a></blockquote><div>Thanks, it's very useful!</div><div><br></div><div>/dev/mapper/vg-test_rmeta_0</div><div>found RAID superblock at offset 0</div><div> magic=1683123524</div><div> features=0</div><div> num_devices=3</div><div> array_position=0</div><div> events=56</div><div> failed_devices=0</div><div> disk_recovery_offset=18446744073709551615</div><div> array_resync_offset=18446744073709551615</div><div> level=5</div><div> layout=2</div><div> stripe_sectors=128</div><div>found bitmap file superblock at offset 4096:</div><div> magic: 6d746962</div><div> version: 4</div><div> uuid: 00000000.00000000.00000000.00000000</div><div> events: 56</div><div>events cleared: 33</div><div> state: 00000000</div><div> chunksize: 524288 B</div><div> daemon sleep: 5s</div><div> sync size: 32768 KB</div><div>max write behind: 0</div><div><br></div><div>/dev/mapper/vg-test_rmeta_1</div><div>found RAID superblock at offset 0</div><div> magic=1683123524</div><div> features=0</div><div> num_devices=3</div><div> array_position=4294967295</div><div> events=62</div><div> failed_devices=1</div><div> disk_recovery_offset=0</div><div> array_resync_offset=18446744073709551615</div><div> level=5</div><div> layout=2</div><div> stripe_sectors=128</div><div>found bitmap file superblock at offset 4096:</div><div> magic: 6d746962</div><div> version: 4</div><div> uuid: 00000000.00000000.00000000.00000000</div><div> events: 60</div><div>events cleared: 33</div><div> state: 00000000</div><div> chunksize: 524288 B</div><div> daemon sleep: 5s</div><div> sync size: 32768 KB</div><div>max write behind: 0</div><div><br></div><div>/dev/mapper/vg-test_rmeta_2</div><div>found RAID superblock at offset 0</div><div> magic=1683123524</div><div> features=0</div><div> num_devices=3</div><div> array_position=2</div><div> events=62</div><div> failed_devices=1</div><div> disk_recovery_offset=18446744073709551615</div><div> array_resync_offset=18446744073709551615</div><div> level=5</div><div> layout=2</div><div> stripe_sectors=128</div><div>found bitmap file superblock at offset 4096:</div><div> magic: 6d746962</div><div> version: 4</div><div> uuid: 00000000.00000000.00000000.00000000</div><div> events: 62</div><div>events cleared: 33</div><div> state: 00000000</div><div> chunksize: 524288 B</div><div> daemon sleep: 5s</div><div> sync size: 32768 KB</div><div>max write behind: 0 </div><div><br></div><div>The problem I see here is that events count is different for the three rmetas.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
<br class="gmail_msg">
> Here is the relevant /etc/lvm/archive (archive is more recent that backup)<br class="gmail_msg">
<br class="gmail_msg">
That looks sane, but you omitted the physical volumes section so there<br class="gmail_msg">
is no way to cross-check UUIDs and devices or see if there are MISSING<br class="gmail_msg">
flags.<br class="gmail_msg"></blockquote><div>The ids are the same and there are no MISSING flags. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
If you use<br class="gmail_msg">
<a href="https://drive.google.com/open?id=0B8dHrWSoVcaDQkU5NG1sLWc5cjg" rel="noreferrer" class="gmail_msg" target="_blank">https://drive.google.com/open?id=0B8dHrWSoVcaDQkU5NG1sLWc5cjg</a><br class="gmail_msg">
directly, you can get metadata that LVM is reading off the PVs and<br class="gmail_msg">
double-check for discrepancies. </blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
linux-lvm mailing list<br class="gmail_msg">
<a href="mailto:linux-lvm@redhat.com" class="gmail_msg" target="_blank">linux-lvm@redhat.com</a><br class="gmail_msg">
<a href="https://www.redhat.com/mailman/listinfo/linux-lvm" rel="noreferrer" class="gmail_msg" target="_blank">https://www.redhat.com/mailman/listinfo/linux-lvm</a><br class="gmail_msg">
read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/" rel="noreferrer" class="gmail_msg" target="_blank">http://tldp.org/HOWTO/LVM-HOWTO/</a><br class="gmail_msg">
</blockquote></div></div>