<div dir="ltr"><div dir="ltr" class="gmail_msg">Thanks!<br><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_quote gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Tue, Oct 4, 2016 at 12:49 PM Giuliano Procida <<a href="mailto:giuliano.procida@gmail.com" class="gmail_msg" target="_blank">giuliano.procida@gmail.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Before anything else, I would have suggested backing up the image and<br class="gmail_msg">
meta sub LVs, but it looks like you are just testing.<br class="gmail_msg"></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Already did. I'm not testing, I just renamed the LVs to "test_*" because the previous name doesn't matter. </div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">There is nothing particularly important there, but I would like to understand whether I would be able to recover should something alike happen in the future.</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
Clear down any odd state with dmsetup remove /dev/vg/... and then run:<br class="gmail_msg">
<br class="gmail_msg">
vgextend --restoremissing<br class="gmail_msg"></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">I didn't have to, because all the PVs are present:</div><div class="gmail_msg"><pre class="gmail_msg"># pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg lvm2 a-- 1.82t 1.10t
/dev/sdb2 vg lvm2 a-- 3.64t 1.42t
/dev/sdc2 vg lvm2 a-- 931.51g 195.18g</pre></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
Actually, always run LVM commands with -v -t before really running them.<br class="gmail_msg"></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Thanks! I had backed up the rmeta* and rimage*, so I didn't feel the need for using -t. Am I wrong?</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
On 4 October 2016 at 00:49, Slava Prisivko <<a href="mailto:vprisivko@gmail.com" class="gmail_msg" target="_blank">vprisivko@gmail.com</a>> wrote:<br class="gmail_msg">
> In order to mitigate cross-posting, here's the original question on<br class="gmail_msg">
> Serverfault.SE: LVM RAID5 out-of-sync recovery, but feel free to answer<br class="gmail_msg">
> wherever you deem appropriate.<br class="gmail_msg">
><br class="gmail_msg">
> How can one recover from an LVM RAID5 out-of-sync?<br class="gmail_msg">
<br class="gmail_msg">
I suppose it's supposed to recover mostly automatically.<br class="gmail_msg">
*If* your array is assembled (or whatever the LVM-equivalent<br class="gmail_msg">
termiology is) then you can force a given subset of PVs to be<br class="gmail_msg">
resynced.<br class="gmail_msg">
<a href="http://man7.org/linux/man-pages/man8/lvchange.8.html" rel="noreferrer" class="gmail_msg" target="_blank">http://man7.org/linux/man-pages/man8/lvchange.8.html</a> - look for rebuild<br class="gmail_msg">
However, this does not seem to be your problem.<br class="gmail_msg"></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Yeah, I tried, but in vain:</div><div class="gmail_msg"><div class="gmail_msg"># lvchange --rebuild /dev/sda2 vg/test -v</div><div class="gmail_msg"> Archiving volume group "vg" metadata (seqno 518).</div><div class="gmail_msg">Do you really want to rebuild 1 PVs of logical volume vg/test [y/n]: y</div><div class="gmail_msg"> Accepted input: [y]</div></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_msg"> vg/test must be active to perform this operation.</div></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> I have an LVM RAID5 configuration (RAID5 using the LVM tools).<br class="gmail_msg">
><br class="gmail_msg">
> However, because of a technical problem mirrors went out of sync. You can<br class="gmail_msg">
> reproduce this as explained in this Unix & Linux question:<br class="gmail_msg">
><br class="gmail_msg">
>> Playing with my Jessie VM, I disconnected (virtually) one disk. That<br class="gmail_msg">
>> worked, the machine stayed running. lvs, though, gave no indication the<br class="gmail_msg">
>> arrays were degraded.<br class="gmail_msg">
<br class="gmail_msg">
You should have noticed something in the kernel logs. Also, lvs should<br class="gmail_msg">
have reported that the array was now (p)artial.<br class="gmail_msg"></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"></blockquote><div class="gmail_msg">Yes, I've noticed it. The problem was a faulty SATA cable (as I learned later), so when I switched the computer on for the first time, /dev/sda was missing (in the current device allocation). I switched off the computer, swapped the /dev/sda and /dev/sdb SATA cable (without thinking about the consequences) and switched it on. This time the /dev/sdb was missing. I replaced the faulty cable with a new one and switched the machine back on. This time sda, sdb and sdc were all present, but the RAID went out-of-sync.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I'm pretty sure there were very few (if any) writing operations during the degraded operating mode, so the I could recover by rebuilding the old mirror (sda) using the more recent ones (sdb and sdc).</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
>> I re-attached the disk, and removed a second. Stayed<br class="gmail_msg">
>> running (this is raid6). Re-attached, still no indication from lvs. I ran<br class="gmail_msg">
>> lvconvert --repair on the volume, it told me it was OK. Then I pulled a<br class="gmail_msg">
>> third disk... and the machine died. Re-inserted it, rebooted, and am now<br class="gmail_msg">
>> unsure how to fix.<br class="gmail_msg">
<br class="gmail_msg">
So this is RAID6 rather than RAID5?<br class="gmail_msg">
And you killed 3 disks in a RAID 6 array?<br class="gmail_msg"></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Although I have RAID5, not the RAID6, but the principle is the same (as I explained in the previous paragraph). <br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
> If I had been using mdadm, I could have probably recovered the data using<br class="gmail_msg">
> `mdadm --force --assemble`, but I was not able to achieve the same using the<br class="gmail_msg">
> LVM tools.<br class="gmail_msg">
<br class="gmail_msg">
LVM is very different. :-(<br class="gmail_msg">
<br class="gmail_msg">
> I have tried to concatenate rmeta and rimage for each mirror and put them on<br class="gmail_msg">
> three linear devices in order to feed them to the mdadm (because LVM<br class="gmail_msg">
> leverages MD), but without success (`mdadm --examine` does not recognize the<br class="gmail_msg">
> superblock), because it appears that the mdadm superblock format differs<br class="gmail_msg">
> from the dm_raid superblock format (search for the "dm_raid_superblock").<br class="gmail_msg">
<br class="gmail_msg">
Not only that, but (as far as I can tell), LVM RAID 6 parity (well,<br class="gmail_msg">
syndrome) is calculated in a different manner to the older mdadm RAID;<br class="gmail_msg">
it uses an industry-standard layout instead of the (more obvious?) md<br class="gmail_msg">
layout.<br class="gmail_msg">
I wrote a utility to parity-check the default LVM RAID6 layout with<br class="gmail_msg">
the usual stripe size (easily adjusted) here:<br class="gmail_msg">
<a href="https://drive.google.com/open?id=0B8dHrWSoVcaDbkY3WmkxSmpfSVE" rel="noreferrer" class="gmail_msg" target="_blank">https://drive.google.com/open?id=0B8dHrWSoVcaDbkY3WmkxSmpfSVE</a><br class="gmail_msg">
<br class="gmail_msg">
You can use this to see to what degree the data in the image LVs are<br class="gmail_msg">
in fact in/out of sync. I've not attempted to add sync functionality<br class="gmail_msg">
to this.<br class="gmail_msg"></blockquote><div>Thanks, I used your raid5_parity_check.cc utility with the default stripe size (64 * 1024), but it actually doesn't matter since you're just calculating the total xor and the stripe size acts as a buffer size for that.</div><div><br></div><div>I get three unsynced stripes out of 512 (32 mib / 64 kib), but I would like to try to reconstruct the test_rimage_1 using h the other two. Just in case, here are the bad stripe numbers: 16, 48, 49.</div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
> I tried to understand how device-mapper RAID leverages MD, but was unable to<br class="gmail_msg">
> find any documentation while the kernel code is quite complicated.<br class="gmail_msg">
><br class="gmail_msg">
> I also tried to rebuild the mirror directly by using `dmsetup`, but it can't<br class="gmail_msg">
> rebuild if metadata is out of sync.<br class="gmail_msg">
><br class="gmail_msg">
> Overall, almost the only useful information I could find is RAIDing with LVM<br class="gmail_msg">
> vs MDRAID - pros and cons? question on Unix & Linux SE.<br class="gmail_msg">
<br class="gmail_msg">
Well, I would read through this as well (versions 6 and 7 also available):<br class="gmail_msg">
<a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/Logical_Volume_Manager_Administration/index.html" rel="noreferrer" class="gmail_msg" target="_blank">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/Logical_Volume_Manager_Administration/index.html</a><br class="gmail_msg">
<br class="gmail_msg"></blockquote><div> Thanks, but nothing particular relevant for my case there.</div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> The output of various commands is provided below.<br class="gmail_msg">
><br class="gmail_msg">
> # lvs -a -o +devices<br class="gmail_msg">
><br class="gmail_msg">
> test vg rwi---r--- 64.00m<br class="gmail_msg">
> test_rimage_0(0),test_rimage_1(0),test_rimage_2(0)<br class="gmail_msg">
> [test_rimage_0] vg Iwi-a-r-r- 32.00m /dev/sdc2(1)<br class="gmail_msg">
> [test_rimage_1] vg Iwi-a-r-r- 32.00m /dev/sda2(238244)<br class="gmail_msg">
> [test_rimage_2] vg Iwi-a-r-r- 32.00m /dev/sdb2(148612)<br class="gmail_msg">
> [test_rmeta_0] vg ewi-a-r-r- 4.00m /dev/sdc2(0)<br class="gmail_msg">
> [test_rmeta_1] vg ewi-a-r-r- 4.00m /dev/sda2(238243)<br class="gmail_msg">
> [test_rmeta_2] vg ewi-a-r-r- 4.00m /dev/sdb2(148611)<br class="gmail_msg">
><br class="gmail_msg">
> I cannot activate the LV:<br class="gmail_msg">
><br class="gmail_msg">
> # lvchange -ay vg/test -v<br class="gmail_msg">
> Activating logical volume "test" exclusively.<br class="gmail_msg">
> activation/volume_list configuration setting not defined: Checking<br class="gmail_msg">
> only host tags for vg/test.<br class="gmail_msg">
> Loading vg-test_rmeta_0 table (253:35)<br class="gmail_msg">
> Suppressed vg-test_rmeta_0 (253:35) identical table reload.<br class="gmail_msg">
> Loading vg-test_rimage_0 table (253:36)<br class="gmail_msg">
> Suppressed vg-test_rimage_0 (253:36) identical table reload.<br class="gmail_msg">
> Loading vg-test_rmeta_1 table (253:37)<br class="gmail_msg">
> Suppressed vg-test_rmeta_1 (253:37) identical table reload.<br class="gmail_msg">
> Loading vg-test_rimage_1 table (253:38)<br class="gmail_msg">
> Suppressed vg-test_rimage_1 (253:38) identical table reload.<br class="gmail_msg">
> Loading vg-test_rmeta_2 table (253:39)<br class="gmail_msg">
> Suppressed vg-test_rmeta_2 (253:39) identical table reload.<br class="gmail_msg">
> Loading vg-test_rimage_2 table (253:40)<br class="gmail_msg">
> Suppressed vg-test_rimage_2 (253:40) identical table reload.<br class="gmail_msg">
> Creating vg-test<br class="gmail_msg">
> Loading vg-test table (253:87)<br class="gmail_msg">
> device-mapper: reload ioctl on (253:87) failed: Invalid argument<br class="gmail_msg">
> Removing vg-test (253:87)<br class="gmail_msg">
><br class="gmail_msg">
> While trying to activate I'm getting the following in the dmesg:<br class="gmail_msg">
><br class="gmail_msg">
> device-mapper: table: 253:87: raid: Cannot change device positions in<br class="gmail_msg">
> RAID array<br class="gmail_msg">
> device-mapper: ioctl: error adding target to table<br class="gmail_msg">
<br class="gmail_msg">
That's a new error message to me. I would try clearing out the dm<br class="gmail_msg">
table (dmsetup remove /dev/vg/test_*) before trying again (-v -t,<br class="gmail_msg">
first).<br class="gmail_msg"></blockquote><div>After cleaning the dmsetup table of test_* and trying to lvchange -ay I get practically the same:</div><div># lvchange -ay vg/test -v </div><div> Activating logical volume vg/test exclusively.</div><div> activation/volume_list configuration setting not defined: Checking only host tags for vg/test.</div><div> Creating vg-test_rmeta_0</div><div> Loading vg-test_rmeta_0 table (253:35)</div><div> Resuming vg-test_rmeta_0 (253:35)</div><div> Creating vg-test_rimage_0</div><div> Loading vg-test_rimage_0 table (253:36)</div><div> Resuming vg-test_rimage_0 (253:36)</div><div> Creating vg-test_rmeta_1</div><div> Loading vg-test_rmeta_1 table (253:37)</div><div> Resuming vg-test_rmeta_1 (253:37)</div><div> Creating vg-test_rimage_1</div><div> Loading vg-test_rimage_1 table (253:38)</div><div> Resuming vg-test_rimage_1 (253:38)</div><div> Creating vg-test_rmeta_2</div><div> Loading vg-test_rmeta_2 table (253:39)</div><div> Resuming vg-test_rmeta_2 (253:39)</div><div> Creating vg-test_rimage_2</div><div> Loading vg-test_rimage_2 table (253:40)</div><div> Resuming vg-test_rimage_2 (253:40)</div><div> Creating vg-test</div><div> Loading vg-test table (253:87)</div><div> device-mapper: reload ioctl on (253:87) failed: Invalid argument</div><div> Removing vg-test (253:87) <br></div><div><br></div><div><div>device-mapper: table: 253:87: raid: Cannot change device positions in RAID array</div><div>device-mapper: ioctl: error adding target to table</div></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
> lvconvert only works on active LVs:<br class="gmail_msg">
> # lvconvert --repair vg/test<br class="gmail_msg">
> vg/test must be active to perform this operation.<br class="gmail_msg">
<br class="gmail_msg">
And it requires new PVs ("replacement drives") to put the subLVs on.<br class="gmail_msg">
It's probably not what you want.<br class="gmail_msg">
<br class="gmail_msg">
> I have the following LVM version:<br class="gmail_msg">
><br class="gmail_msg">
> # lvm version<br class="gmail_msg">
> LVM version: 2.02.145(2) (2016-03-04)<br class="gmail_msg">
> Library version: 1.02.119 (2016-03-04)<br class="gmail_msg">
> Driver version: 4.34.0<br class="gmail_msg">
<br class="gmail_msg">
I would update LVM to whatever is in Debian testing as there has been<br class="gmail_msg">
a fair bit of change this year.<br class="gmail_msg"></blockquote><div>I've updated to the 2.02.166 (the latest version):</div><div><pre style="top: -99px;"># lvm version
LVM version: 2.02.166(2) (2016-09-26)
Library version: 1.02.135 (2016-09-26)
Driver version: 4.34.0</pre></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
> And the following kernel version:<br class="gmail_msg">
><br class="gmail_msg">
> Linux server 4.4.8-hardened-r1-1 #1 SMP<br class="gmail_msg">
<br class="gmail_msg">
More useful would be the contents of /etc/lvm/backup/vg and the output<br class="gmail_msg">
of vgs and pvs.<br class="gmail_msg"></blockquote><div><pre style="top: -99px;"># pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg lvm2 a-- 1.82t 1.10t
/dev/sdb2 vg lvm2 a-- 3.64t 1.42t
/dev/sdc2 vg lvm2 a-- 931.51g 195.18g</pre><pre style="top: -99px;"><pre style="top: -99px;"># vgs
VG #PV #LV #SN Attr VSize VFree
vg 3 18 0 wz--n- 6.37t 2.71t</pre><pre style="top: -99px;">Here is the relevant /etc/lvm/archive (archive is more recent that backup) content:<br><span style="font-family:sans-serif">test {</span><br></pre> id = "JjiPmi-esfx-vdeF-5zMv-TsJC-6vFf-qNgNnZ"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 16 # 64 Megabytes
type = "raid5"
device_count = 3
stripe_size = 128
region_size = 1024
raids = [
"test_rmeta_0", "test_rimage_0",
"test_rmeta_1", "test_rimage_1",
"test_rmeta_2", "test_rimage_2"
]
}
}
</pre><pre style="top: -99px;"> test_rmeta_0 {
id = "WE3CUg-ayo8-lp1Y-9S2v-zRGi-mV1s-DWYoST"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 1 # 4 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv0", 0
]
}
}
test_rmeta_1 {
id = "Apk3mc-zy4q-c05I-hiIO-1Kae-9yB6-Cl5lfJ"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 1 # 4 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 238243
]
}
}
test_rmeta_2 {
id = "j2Waf3-A77y-pvfd-foGK-Hq7B-rHe8-YKzQY0"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 1 # 4 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv2", 148611
]
}
}
test_rimage_0 {
id = "zaGgJx-YSIl-o2oq-UN9l-02Q8-IS5u-sz4RhQ"
status = ["READ", "WRITE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 8 # 32 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv0", 1
]
}
}
test_rimage_1 {
id = "0mD5AL-GKj3-siFz-xQmO-ZtQo-L3MM-Ro2SG2"
status = ["READ", "WRITE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 8 # 32 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv1", 238244
]
}
}
test_rimage_2 {
id = "4FxiHV-j637-ENml-Okm3-uL1p-fuZ0-y9dE8Y"
status = ["READ", "WRITE"]
flags = []
creation_time = 18446744073709551615 # 1970-01-01 02:59:59 +0300
creation_host = "server"
segment_count = 1
segment1 {
start_extent = 0
extent_count = 8 # 32 Megabytes
type = "striped"
stripe_count = 1 # linear
stripes = [
"pv2", 148612
]
}
}<br></pre>--<br>Best regards,<br>Slava Prisivko.<br></div><div> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
linux-lvm mailing list<br class="gmail_msg">
<a href="mailto:linux-lvm@redhat.com" class="gmail_msg" target="_blank">linux-lvm@redhat.com</a><br class="gmail_msg">
<a href="https://www.redhat.com/mailman/listinfo/linux-lvm" rel="noreferrer" class="gmail_msg" target="_blank">https://www.redhat.com/mailman/listinfo/linux-lvm</a><br class="gmail_msg">
read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/" rel="noreferrer" class="gmail_msg" target="_blank">http://tldp.org/HOWTO/LVM-HOWTO/</a><br class="gmail_msg">
</blockquote></div></div></div></div>