<div dir="ltr">Thanks for the response. <div><br></div><div>><span style="color:rgb(33,33,33)">Is this everything?</span></div><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">Yes, that is everything in the metadata xml dump. I just removed all of the *_mapping entries for brevity. For the lvs output I removed other logical volumes that aren't related to this pool.</span></div><div dir="ltr"><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">></span><span style="color:rgb(33,33,33)">Is this a pool used by docker, which does not (did</span><span style="color:rgb(33,33,33)"> </span><span style="color:rgb(33,33,33)">not) use LVM to manage thin-volumes?</span></div></div><div dir="ltr"><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">It's not docker, but it is an application called serviced that uses docker's library for managing the volumes</span></div></div><div dir="ltr"><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">></span><span style="color:rgb(33,33,33)">LVM just queries DM, and displays whatever that provides</span></div></div><div dir="ltr"><div><span style="color:rgb(33,33,33)"><br></span></div><div><span style="color:rgb(33,33,33)">Yeah, it looks like dmsetup status output matches lvs:</span></div><div><pre style="padding:6px 10px;font-family:Consolas,"Liberation Mono",Menlo,"Bitstream Vera Sans Mono",Courier,monospace;font-size:12.025px;color:rgb(51,51,51);border-radius:3px;margin-top:15px;margin-bottom:15px;line-height:19px;word-break:break-all;word-wrap:break-word;white-space:pre-line;background:rgb(248,248,248);border:1px solid rgb(204,204,204);outline:0px;vertical-align:baseline"><code style="padding:0px;font-family:Consolas,"Liberation Mono",Menlo,"Bitstream Vera Sans Mono",Courier,monospace;font-size:12px;color:inherit;border-radius:3px;background:none;border:0px;margin:0px;outline:0px;vertical-align:baseline;white-space:pre-wrap;word-break:break-word">myvg-my--pool: 0 5242880000 thin-pool 70 207941/4145152 29018611/40960000 - rw discard_passdown queue_if_no_space -
myvg-my--pool_tdata: 0 <a href="tel:(419)%20430-4000" value="+14194304000" target="_blank">4194304000</a> linear
myvg-my--pool_tdata: 4194304000 1048576000 linear
myvg-my--pool_tmeta: 0 33161216 linear</code></pre></div><div>><span style="color:rgb(33,33,33)">What is kernel/lvm version?</span></div><div><pre style="padding:6px 10px;font-family:Consolas,"Liberation Mono",Menlo,"Bitstream Vera Sans Mono",Courier,monospace;font-size:12.025px;color:rgb(51,51,51);border-radius:3px;margin-top:15px;margin-bottom:15px;line-height:19px;word-break:break-all;word-wrap:break-word;white-space:pre-line;background:rgb(248,248,248);border:1px solid rgb(204,204,204);outline:0px;vertical-align:baseline"><code style="padding:0px;font-family:Consolas,"Liberation Mono",Menlo,"Bitstream Vera Sans Mono",Courier,monospace;font-size:12px;color:inherit;border-radius:3px;background:none;border:0px;margin:0px;outline:0px;vertical-align:baseline;white-space:pre-wrap;word-break:break-word"># uname -r
3.10.0-693.21.1.el7.x86_64</code></pre></div></div><div><pre style="padding:6px 10px;font-family:Consolas,"Liberation Mono",Menlo,"Bitstream Vera Sans Mono",Courier,monospace;font-size:12.025px;color:rgb(51,51,51);border-radius:3px;margin-top:15px;margin-bottom:15px;line-height:19px;word-break:break-all;word-wrap:break-word;white-space:pre-line;background:rgb(248,248,248);border:1px solid rgb(204,204,204);outline:0px;vertical-align:baseline"><code style="padding:0px;font-family:Consolas,"Liberation Mono",Menlo,"Bitstream Vera Sans Mono",Courier,monospace;font-size:12px;color:inherit;border-radius:3px;background:none;border:0px;margin:0px;outline:0px;vertical-align:baseline;white-space:pre-wrap;word-break:break-word"># lvm version
LVM version: 2.02.171(2)-RHEL7 (2017-05-03)
Library version: 1.02.140-RHEL7 (2017-05-03)
Driver version: 4.35.0
Configuration: ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-dmfilemapd</code></pre></div><div>><span style="color:rgb(33,33,33);font-size:13px">Is thin_check_executable configured in lvm.conf?</span></div><div><span style="color:rgb(33,33,33);font-size:13px"><br></span></div><div><span style="color:rgb(33,33,33);font-size:13px">Yes</span></div><div><span style="color:rgb(33,33,33);font-size:13px"><br></span></div><div>I also just found out that they apparently ran thin_check recently and got a message about a corrupt superblock, but didn't repair it. They were still able to re-activate the pool though. We'll run a repair as soon as we get a chance and see if that fixes it.<br></div><div><br></div><div>Thanks,</div><div><br></div><div>John</div><br><div class="gmail_quote"><div dir="ltr">On Fri, May 11, 2018 at 3:54 AM Marian Csontos <<a href="mailto:mcsontos@redhat.com" target="_blank">mcsontos@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 05/11/2018 10:21 AM, Joe Thornber wrote:<br>
> On Thu, May 10, 2018 at 07:30:09PM +0000, John Hamilton wrote:<br>
>> I saw something today that I don't understand and I'm hoping somebody can<br>
>> help. We had a ~2.5TB thin pool that was showing 69% data utilization in<br>
>> lvs:<br>
>><br>
>> # lvs -a<br>
>> LV VG Attr LSize Pool Origin Data%<br>
>> Meta% Move Log Cpy%Sync Convert<br>
>> my-pool myvg twi-aotz-- 2.44t 69.04 4.90<br>
>> [my-pool_tdata] myvg Twi-ao---- 2.44t<br>
>> [my-pool_tmeta] myvg ewi-ao---- 15.81g<br>
<br>
Is this everything? Is this a pool used by docker, which does not (did <br>
not) use LVM to manage thin-volumes?<br>
<br>
>> However, when I dump the thin pool metadata and look at the mapped_blocks<br>
>> for the 2 devices in the pool, I can only account for about 950GB. Here is<br>
>> the superblock and device entries from the metadata xml. There are no<br>
>> other devices listed in the metadata:<br>
>><br>
>> <superblock uuid="" time="34" transaction="68" flags="0" version="2"<br>
>> data_block_size="128" nr_data_blocks="0"><br>
>> <device dev_id="1" mapped_blocks="258767" transaction="0"<br>
>> creation_time="0" snap_time="14"><br>
>> <device dev_id="8" mapped_blocks="15616093" transaction="27"<br>
>> creation_time="15" snap_time="34"><br>
>><br>
>> That first device looks like it has about 16GB allocated to it and the<br>
>> second device about 950GB. So, I would expect lvs to show somewhere<br>
>> between 950G-966G Is something wrong, or am I misunderstanding how to read<br>
>> the metadata dump? Where is the other 700 or so GB that lvs is showing<br>
>> used?<br>
> <br>
> The non zero snap_time suggests that you're using snapshots. I which case it<br>
> could just be there is common data shared between volumes that is getting counted<br>
> more than once.<br>
> <br>
> You can confirm this using the thin_ls tool and specifying a format line that<br>
> includes EXCLUSIVE_BLOCKS, or SHARED_BLOCKS. Lvm doesn't take shared blocks into<br>
> account because it has to scan all the metadata to calculate what's shared.<br>
<br>
LVM just queries DM, and displays whatever that provides. You could see <br>
that in `dmsetup status` output, there are two pairs of '/' separated <br>
entries - first is metadata usage (USED_BLOCKS/ALL_BLOCKS), second data <br>
usage (USED_CHUNKS/ALL_CHUNKS).<br>
<br>
So the error lies somewhere between dmsetup and kernel.<br>
<br>
What is kernel/lvm version?<br>
Is thin_check_executable configured in lvm.conf?<br>
<br>
-- Martian<br>
</blockquote></div></div>