<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">I would happily send you the diagnostic logs - if you could specify what needs to be collected.</div></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; "><br></div></span></div><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">-- </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">----------------------------------------------</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; ">Andres Toomsalu, <a href="mailto:andres@active.ee">andres@active.ee</a></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Helvetica; min-height: 14px; "><br></div></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On 18.10.2012, at 13:42, Andres Toomsalu wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Yes I know that its not currently possible to recover this pool anymore - and we got data copied from read-only mode anyway - but bigger problem is that there is even no known way to reset/recreate pool - as I cant delete/remove the pool - which means that only erasing all PV will help probably. Im just looking for a way to erase faulty pool (with erasing data) and to be able recreate it without full OS reinstall. Is there a way to do it - DM mapper low level commands perhaps?<br><br>Cheers,<br>-- <br>----------------------------------------------<br>Andres Toomsalu, <a href="mailto:andres@active.ee">andres@active.ee</a><br><br><br><br><br><br><br>On 18.10.2012, at 13:30, Zdenek Kabelac wrote:<br><br><blockquote type="cite">Dne 17.10.2012 22:21, Andres Toomsalu napsal(a):<br></blockquote><blockquote type="cite"><blockquote type="cite">Hi,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I'm aware that thin provisioning is not yet production ready (no metadata resize) - but is there a way to recover from thin pool failure when pool metadata was filled up?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I did setup 1.95T thin pool and after some usage pool metadata (128MB) was filling up to 99,08% - so all pool thin volumes went into read-only state.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Problem is that I cannot find a way in order to recover from this failure - eg also unable to delete/erase thin volumes and pool - only option seems to be full disk PV re-creation (eg OS re-install).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Is there a way to recover or delete thin pool/volumes - without erasing other (normal) LVs in this Volume Group?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">For example dmsetup remove didn't help.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Some diagnostic output:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">lvs -a -o+metadata_percent<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> dm_report_object: report function failed for field data_percent<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> --- REPEATABLE MSG ---<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> dm_report_object: report function failed for field data_percent<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert Meta%<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> pool VolGroupL0 twi-i-tz 1,95t 75,28 99,08<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> [pool_tdata] VolGroupL0 Twi-aot- 1,95t<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> [pool_tmeta] VolGroupL0 ewi-aot- 128,00m<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> root VolGroupL0 -wi-ao-- 10,00g<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> swap VolGroupL0 -wi-ao-- 16,00g<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_backup VolGroupL0 Vwi-i-tz 700,00g pool<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage VolGroupL0 Vwi---tz 900,00g pool<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot1 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot106 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot130 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot154 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot178 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot2 VolGroupL0 Vwi-i-tz 700,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> thin_storage-snapshot202 VolGroupL0 Vwi-i-tz 900,00g pool thin_storage<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">dmsetup table<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot2:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot178:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-swap: 0 33554432 linear 8:2 41945088<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot1:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-root: 0 20971520 linear 8:2 20973568<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot130:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-pool:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_backup:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot106:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot154:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-pool-tpool: 0 4194304000 thin-pool 253:2 253:3 1024 0 0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-pool_tdata: 0 2097152000 linear 8:2 75499520<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-pool_tdata: 2097152000 2097152000 linear 8:2 2172913664<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-pool_tmeta: 0 262144 linear 8:2 2172651520<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VolGroupL0-thin_storage--snapshot202:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">lvremove -f VolGroupL0/pool<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Thin pool transaction_id=640, while expected: 643.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Unable to deactivate open VolGroupL0-pool_tdata (253:3)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Unable to deactivate open VolGroupL0-pool_tmeta (253:2)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Failed to deactivate VolGroupL0-pool-tpool<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Failed to resume pool.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Failed to update thin pool pool.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Unfortunately there is no 'easy' advice for now yet - you hit current Achilles heel of thinp support in lvm2 - we are thinking how to make recovery usable for user - but it's not easy task since many things are making it very complex - so it still needs some month of work.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">As for now - you would need to download git source and disable certain security check directly in the source to allow activation of damaged pool.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I guess I could provide you some extra patch that would allow you to active thin pool in 'read-only' mode (but it's not yet ready for upstream).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thus you should be able access 'some' data in this mode with a big note - since devices would be in read-only mode - you cannot even run fsck on such partition - and since you have transaction_id mismatch - it would need<br></blockquote><blockquote type="cite">some analysis to see what actually happened - are you able to provide me<br></blockquote><blockquote type="cite">archive files for the history of your recent lvm commands -<br></blockquote><blockquote type="cite">it should not allow you to have difference bigger then 1 - so there is probably some bug (unless you are using some old version of lvm2 with initial thinp support)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Zdenek<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">linux-lvm mailing list<br></blockquote><blockquote type="cite"><a href="mailto:linux-lvm@redhat.com">linux-lvm@redhat.com</a><br></blockquote><blockquote type="cite"><a href="https://www.redhat.com/mailman/listinfo/linux-lvm">https://www.redhat.com/mailman/listinfo/linux-lvm</a><br></blockquote><blockquote type="cite">read the LVM HOW-TO at <a href="http://tldp.org/HOWTO/LVM-HOWTO/">http://tldp.org/HOWTO/LVM-HOWTO/</a><br></blockquote><blockquote type="cite"><br></blockquote><br><br>_______________________________________________<br>linux-lvm mailing list<br><a href="mailto:linux-lvm@redhat.com">linux-lvm@redhat.com</a><br>https://www.redhat.com/mailman/listinfo/linux-lvm<br>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/<br><br></div></blockquote></div><br></body></html>