<div dir="ltr"><div><br></div>That's pretty much it. Whenever any app attempts to read a block from the missing drive, I get the "Buffer I/O error" message. So, even though my recovery apps can scan the LV, marking blocks on the last drive as missing/unknown/etc., they can't display any recovered data - which I know does exist. Looking at raw data from the apps' scans, I can see directory entries, as well as files. I'm sure the inodes and bitmaps are still there for some of these, I just can't really reverse engineer and follow them through. But isn't that what the apps are supposed to do?<div><br></div><div>As for debugfs: pretty much the same issue: in order to use it, I need to open the fs. But that (in debugfs) fails as well. So it can't help much. Unless I'm missing something about debugfs.</div><div><br></div><div>The one thing I haven't tried is to use vgreduce to remove the missing PV; but that will also remove the LV as well, which is why I haven't tried it yet.<br></div><div><br></div><div>Sorry I haven't replied sooner, but it takes a long time (days) to clone, then scan 16Tb...</div><div><br></div><div>So, please any suggestions are greatly appreciated, as well as needed.</div><div><br></div><div>ken</div><div><br></div><div><div>(I know: No backup; got burned; it hurts; and I will now always have backups. 'Nuf said.)</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 28, 2022 at 3:12 AM Roger James <<a href="mailto:roger@beardandsandals.co.uk">roger@beardandsandals.co.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u> <div> <div dir="auto"> <div dir="auto">The procedure outlined should at least get you back to a state where the lv is consistent but with blank sectors where the data is missing. I would suggest using dd to make a backup partition image. Then you can either work on that or the original to mend the fs.</div><div dir="auto"><br></div> <div id="gmail-m_-8979662971263915469aqm-original" style="color:black"> <div dir="auto">On 27 July 2022 11:50:07 Roger Heflin <<a href="mailto:rogerheflin@gmail.com" target="_blank">rogerheflin@gmail.com</a>> wrote:</div> <div><br></div> <blockquote type="cite" class="gmail_quote" style="margin:0px 0px 0px 0.75ex;border-left:1px solid rgb(128,128,128);padding-left:0.75ex"> <div dir="auto">I don't believe that is going to work.</div> <div dir="auto"><br></div> <div dir="auto">His issue is that the filesystem is refusing to work because of the</div> <div dir="auto">missing data.</div> <div dir="auto"><br></div> <div dir="auto">man debugfs</div> <div dir="auto"><br></div> <div dir="auto">It will let you manually look at the metadata and structures of the</div> <div dir="auto">ext2/3/4 fs. You will likely need to use the "-c" option.</div> <div dir="auto"><br></div> <div dir="auto">It will be very manual and you should probably read up on the fs</div> <div dir="auto">structure a bit.</div> <div dir="auto"><br></div> <div dir="auto">A data recovery company could get most of the data back, but they</div> <div dir="auto">charge 5k-10k per TB, so likely close to 100k US$.</div> <div dir="auto"><br></div> <div dir="auto">And the issues will be that 1/3 of the metadata was on the missing</div> <div dir="auto">disk, and some of the data was on the missing disk.</div> <div dir="auto"><br></div> <div dir="auto">I was able to do debugfs /dev/sda2 (my /boot) and do an ls and list</div> <div dir="auto">out the files and then do a dump <filename> /tmp/junk.out and copy out</div> <div dir="auto">that file.</div> <div dir="auto"><br></div> <div dir="auto">So the issue will be writing up a script to do lses and find all of</div> <div dir="auto">the files and dump all of the files to someplace else.</div> <div dir="auto"><br></div> <div dir="auto">On Wed, Jul 27, 2022 at 2:39 AM Roger James <<a href="mailto:roger@beardandsandals.co.uk" target="_blank">roger@beardandsandals.co.uk</a>> wrote:</div> <blockquote type="cite" class="gmail_quote" style="margin:0px 0px 0px 0.75ex;border-left:1px solid rgb(0,153,204);padding-left:0.75ex"> <div dir="auto"><br></div> <div dir="auto">Try <a href="https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp" target="_blank">https://www.linuxsysadmins.com/recover-a-deleted-physical-volume/?amp</a></div> <div dir="auto"><br></div> <div dir="auto">On 26 July 2022 09:16:32 Ken Bass <<a href="mailto:daytooner@gmail.com" target="_blank">daytooner@gmail.com</a>> wrote:</div> <blockquote type="cite" class="gmail_quote" style="margin:0px 0px 0px 0.75ex;border-left:1px solid rgb(153,51,204);padding-left:0.75ex"> <div dir="auto"><br></div> <div dir="auto">(fwiw: I am new to this list, so please bear with me.)</div> <div dir="auto"><br></div> <div dir="auto">Background: I have a very large (20TB) logical volume consisting of 3 drives. One of those drives unexpectedloy died (isn't that always the case :-)). The drive that failed happened to be the last PV. So I am assuming that there is still 2/3 of the data still intact and, to some extent, recoverable. Although, apparently the ext4 fs is not recognised.</div> <div dir="auto"><br></div> <div dir="auto">I activated the LV partially (via -P). But running any utility on that (eg: dumpe2fs, e2fsck, ...) I get many of these in dmesg:</div> <div dir="auto"><br></div> <div dir="auto">"Buffer I/O error on dev dm-0, logical block xxxxxxx, async page read." The thing is, the xxxxxxx block is on the missing drive/pv.</div> <div dir="auto"><br></div> <div dir="auto">I have also tried some recovery software, but eventually get these same messages, and the data recovered is not really useful.</div> <div dir="auto"><br></div> <div dir="auto">Please help! How can I get passed that dmesg error, and move on. 14TB recovered is better than 0.</div> <div dir="auto"><br></div> <div dir="auto">TIA</div> <div dir="auto">ken</div> <div dir="auto"><br></div> <div dir="auto"><br></div> <div dir="auto">_______________________________________________</div> <div dir="auto">linux-lvm mailing list</div> <div dir="auto"><a href="mailto:linux-lvm@redhat.com" target="_blank">linux-lvm@redhat.com</a></div> <div dir="auto"><a href="https://listman.redhat.com/mailman/listinfo/linux-lvm" target="_blank">https://listman.redhat.com/mailman/listinfo/linux-lvm</a></div> <div dir="auto">read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/" target="_blank">http://tldp.org/HOWTO/LVM-HOWTO/</a></div> <div dir="auto"><br></div> </blockquote> <div dir="auto"><br></div> <div dir="auto">_______________________________________________</div> <div dir="auto">linux-lvm mailing list</div> <div dir="auto"><a href="mailto:linux-lvm@redhat.com" target="_blank">linux-lvm@redhat.com</a></div> <div dir="auto"><a href="https://listman.redhat.com/mailman/listinfo/linux-lvm" target="_blank">https://listman.redhat.com/mailman/listinfo/linux-lvm</a></div> <div dir="auto">read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/" target="_blank">http://tldp.org/HOWTO/LVM-HOWTO/</a></div> </blockquote> <div dir="auto"><br></div> <div dir="auto">_______________________________________________</div> <div dir="auto">linux-lvm mailing list</div> <div dir="auto"><a href="mailto:linux-lvm@redhat.com" target="_blank">linux-lvm@redhat.com</a></div> <div dir="auto"><a href="https://listman.redhat.com/mailman/listinfo/linux-lvm" target="_blank">https://listman.redhat.com/mailman/listinfo/linux-lvm</a></div> <div dir="auto">read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/" target="_blank">http://tldp.org/HOWTO/LVM-HOWTO/</a></div> </blockquote> </div><div dir="auto"><br></div> </div></div> _______________________________________________<br> linux-lvm mailing list<br> <a href="mailto:linux-lvm@redhat.com" target="_blank">linux-lvm@redhat.com</a><br> <a href="https://listman.redhat.com/mailman/listinfo/linux-lvm" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/linux-lvm</a><br> read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/" rel="noreferrer" target="_blank">http://tldp.org/HOWTO/LVM-HOWTO/</a><br> </blockquote></div>