<div dir="ltr">Hi there,<div><br></div><div>I have set up a RAID-1 between two PVs on two different hard disks in a RHEL-6 environment.</div><div>I am having a problem restoring the broken mirror without rebooting the OS.</div>
<div><br></div><div>Here is how to reproduce my problem.</div><div>1. First of all, we have set up a raid-1 LV that looks like this:</div><div><div>[root@shi-rhel63 home]# lvs -a -o +seg_pe_ranges|grep home</div><div> lv_home vg_root rwi-aom- 1.50g 100.00 lv_home_rimage_0:0-47 lv_home_rimage_1:0-47 </div>
<div> [lv_home_rimage_0] vg_root iwi-aor- 1.50g /dev/sda2:192-239 </div><div> [lv_home_rimage_1] vg_root iwi-aor- 1.50g /dev/sdb2:34-81 </div>
<div> [lv_home_rmeta_0] vg_root ewi-aor- 32.00m /dev/sda2:177-177 </div><div> [lv_home_rmeta_1] vg_root ewi-aor- 32.00m /dev/sdb2:33-33 </div>
<div>[root@shi-rhel63 home]# pvs</div><div> PV VG Fmt Attr PSize PFree </div><div> /dev/sda2 vg_root lvm2 a-- 17.84g 224.00m</div><div> /dev/sdb2 vg_root lvm2 a-- 18.84g 1.22g</div><div><br></div>
<div style>I can test the mirror by performing a dd write and watch the iostat on both physical disks:</div><div style><div>[root@shi-rhel63 home]# dd if=/dev/zero of=test bs=1M count=10000 &</div><div>[1] 23388</div>
<div>[root@shi-rhel63 home]# iostat -x 1 sda sdb -m</div><div>Linux 2.6.32-279.el6.x86_64 (shi-rhel63) <span class="" style="white-space:pre"> </span>06/11/13 <span class="" style="white-space:pre"> </span>_x86_64_<span class="" style="white-space:pre"> </span>(1 CPU)</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.06 0.07 0.35 0.32 0.00 99.20</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.05 0.78 0.38 0.14 0.00 0.00 22.14 0.01 11.07 5.63 0.29</div><div>sdb 0.11 0.79 0.14 0.23 0.00 0.00 27.94 0.00 5.60 3.71 0.14</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.00 0.00 7.14 92.86 0.00 0.00</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.00 5702.04 1.02 56.12 0.01 22.46 805.25 50.75 911.66 17.86 102.04</div><div>sdb 0.00 5173.47 0.00 71.43 0.00 20.42 585.43 1.21 16.97 3.44 24.59</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.00 0.00 8.08 91.92 0.00 0.00</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.00 5131.31 0.00 52.53 0.00 19.71 768.50 51.39 894.83 19.23 101.01</div><div>sdb 0.00 5121.21 0.00 69.70 0.00 18.20 534.70 1.15 15.96 3.41 23.74</div>
<div><br></div><div style>As you may see, both sda and sdb gets similar write IOs so the miror is in fact working.</div><div style><br></div><div style>2. Now I am going to simulate a disk failure on sdb by removing the disk (I use VMware so it is very easy). Of course, the mirror is now broken as show below:</div>
<div style><br></div><div style><div>[root@shi-rhel63 home]# lvs -a -o +seg_pe_ranges|grep home</div><div> Couldn't find device with uuid pnsMYs-Ce4t-9KYR-3Zfs-GItC-k5SZ-VidQ30.</div><div> lv_home vg_root rwi-aom- 1.50g 100.00 lv_home_rimage_0:0-47 lv_home_rimage_1:0-47 </div>
<div> [lv_home_rimage_0] vg_root iwi-aor- 1.50g /dev/sda2:192-239 </div><div> [lv_home_rimage_1] vg_root iwi-aor- 1.50g unknown device:34-81 </div>
<div> [lv_home_rmeta_0] vg_root ewi-aor- 32.00m /dev/sda2:177-177 </div><div> [lv_home_rmeta_1] vg_root ewi-aor- 32.00m unknown device:33-33 </div>
<div>[root@shi-rhel63 home]# pvs</div><div> Couldn't find device with uuid pnsMYs-Ce4t-9KYR-3Zfs-GItC-k5SZ-VidQ30.</div><div> PV VG Fmt Attr PSize PFree </div><div> /dev/sda2 vg_root lvm2 a-- 17.84g 224.00m</div>
<div> unknown device vg_root lvm2 a-m 18.84g 1.22g</div><div><br></div><div style>I already have a slight problem here since the mirror status above still shows 100% Copy% but I accept it as a minor presentation issue.</div>
<div style><br></div><div style>3. Now I put the moved disk back and I would like to have a way to incrementally resync the difference from the point where the mirror is broken. Note that the same disk now shows up as sdc</div>
<div style><br></div><div style><div>[root@shi-rhel63 home]# pvs</div><div> PV VG Fmt Attr PSize PFree </div><div> /dev/sda2 vg_root lvm2 a-- 17.84g 224.00m</div><div> /dev/sdc2 vg_root lvm2 a-- 18.84g 1.22g</div>
<div>[root@shi-rhel63 home]# lvs -a -o +seg_pe_ranges|grep home</div><div> lv_home vg_root rwi-aom- 1.50g 100.00 lv_home_rimage_0:0-47 lv_home_rimage_1:0-47 </div><div> [lv_home_rimage_0] vg_root iwi-aor- 1.50g /dev/sda2:192-239 </div>
<div> [lv_home_rimage_1] vg_root iwi-aor- 1.50g /dev/sdc2:34-81 </div><div> [lv_home_rmeta_0] vg_root ewi-aor- 32.00m /dev/sda2:177-177 </div>
<div> [lv_home_rmeta_1] vg_root ewi-aor- 32.00m /dev/sdc2:33-33 </div><div><br></div><div style>So everything looks perfect but if I perform the same dd write test, here is what I got:</div>
<div style><div>[root@shi-rhel63 home]# iostat -x 1 sda sdc -m</div><div>Linux 2.6.32-279.el6.x86_64 (shi-rhel63) <span class="" style="white-space:pre"> </span>06/11/13 <span class="" style="white-space:pre"> </span>_x86_64_<span class="" style="white-space:pre"> </span>(1 CPU)</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.06 0.07 0.36 0.42 0.00 99.10</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.05 7.70 0.39 0.21 0.00 0.03 109.24 0.11 174.10 6.77 0.40</div><div>sdc 0.02 0.00 0.01 0.00 0.00 0.00 8.30 0.00 3.07 2.85 0.00</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.00 0.00 5.15 94.85 0.00 0.00</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.00 6281.44 0.00 49.48 0.00 22.14 916.38 138.42 2932.19 20.83 103.09</div><div>sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.00 0.00 5.10 94.90 0.00 0.00</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.00 6213.27 0.00 54.08 0.00 23.99 908.30 136.34 2812.40 18.87 102.04</div><div>sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00</div>
<div><br></div><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 0.00 0.00 6.06 93.94 0.00 0.00</div><div><br></div><div>Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util</div>
<div>sda 0.00 5141.41 0.00 50.51 0.00 22.23 901.36 133.47 2612.74 20.00 101.01</div><div>sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00</div>
<div><br></div><div style>So it is clear that the newly added mirror is not being written at all.</div><div style><br></div><div style>What is really interesting is that if I reboot, it will work properly. But is there a way not to reboot?</div>
<div style><br></div><div style>Thanks a lot,</div><div style>Shi</div><div style>PS. My OS info</div><div style><br></div><div style><div>[root@shi-rhel63 home]# uname -a</div><div>Linux shi-rhel63 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux</div>
<div>[root@shi-rhel63 home]# lvm version</div><div> LVM version: 2.02.95(2)-RHEL6 (2012-05-16)</div><div> Library version: 1.02.74-RHEL6 (2012-05-16)</div><div> Driver version: 4.22.6</div><div><br></div></div></div>
</div></div></div></div></div>