<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hi Zdenek Kabelac, Thanks for your quick reply and suggestions. We conducted couple of tests on Ubuntu 22.04 and observed similar performance behavior post thin snapshot deletion without writing any data anywhere. <o:p> </o:p> Commands used to create Thin LVM volume: - lvcreate -L 480G --poolmetadataspare n --poolmetadatasize 16G --chunksize=64K --thinpool ThinDataLV ThinVolGrp - lvcreate -n ext4.ThinLV -V 100G --thinpool ThinDataLV ThinVolGrp<o:p></o:p> <o:p> </o:p> ThinLV display: --- Logical volume --- LV Path /dev/ThinVolGrp/ext4.ThinLV LV Name ext4.ThinLV VG Name ThinVolGrp LV UUID sRcj9L-Ili4-dR3I-MeJI-3KLv-xPUP-VMd1LC LV Write Access read/write LV Creation host, time kapil-upstream, 2022-10-14 18:05:22 +0000 LV Pool name ThinDataLV LV Status available # open 1 LV Size 100.00 GiB Mapped size 21.51% Current LE 25600 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4<o:p></o:p> <o:p> </o:p> dmsetup table output after creation of thin lvm volume:<o:p></o:p> ThinVolGrp-ThinDataLV: 0 1006632960 linear 253:2 0<o:p></o:p> ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 253:0 253:1 128 0 0 <o:p></o:p> ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear 8:32 2048<o:p></o:p> ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear 8:32 1006635008<o:p></o:p> ThinVolGrp-ext4.ThinLV: 0 209715200 thin 253:2 1<o:p></o:p> <o:p> </o:p> dmsetup status after creation of thin lvm volume:<o:p></o:p> ThinVolGrp-ThinDataLV: 0 1006632960 linear <o:p></o:p> ThinVolGrp-ThinDataLV-tpool: 0 1006632960 thin-pool 1 4878/4145152 8325/7864320 - rw discard_passdown queue_if_no_space - 1024 <o:p></o:p> ThinVolGrp-ThinDataLV_tdata: 0 1006632960 linear <o:p></o:p> ThinVolGrp-ThinDataLV_tmeta: 0 33161216 linear <o:p></o:p> ThinVolGrp-ext4.ThinLV: 0 209715200 thin 1065600 209715199 <o:p></o:p> <o:p> </o:p> FIO Command to dump some data: fio --filename=/dev/ThinVolGrp/ext4.ThinLV --name=<name> --time_based --group_reporting --ioengine=libaio --direct=1 --size=20G --runtime=120 --rw=randrw --rwmixread=50 --bs=16K --iodepth=8 --numjobs=3 --randrepeat=0 --randseed=$(date +%s) <o:p> </o:p> For detailed information of fio and dmsetup results click <a href="https://gist.github.com/mittachaitu/17006a1e3f08ad0c17add7ea007e63b6#file-lvm_snapshot_performance"> here</a>. Note: All the tests are conducted using thin lvm volume and thin snapshot Environment details: <div> Kernel Version: 5.15.0-1021-azure<o:p></o:p> LVM Version: LVM version: 2.03.11(2) (2021-01-08) Library version: 1.02.175 (2021-01-08) Driver version: 4.45.0<o:p></o:p> FIO Version: 3.28<o:p></o:p> Please let us know if you required more information and tests that needs to be run. </div> <o:p> </o:p> Thanks & Regards mittachaitu (Sai) <o:p> </o:p> <div style="mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: <a href="mailto:zdenek.kabelac@gmail.com">Zdenek Kabelac</a> Sent: Thursday, October 13, 2022 4:20 PM To: <a href="mailto:lvm-devel@redhat.com">LVM2 development</a>; <a href="mailto:sharmapawan@microsoft.com"> Pawan Sharma</a>; <a href="mailto:linux-lvm@redhat.com">linux-lvm@redhat.com</a> Cc: <a href="mailto:kupadhayay@microsoft.com">Kapil Upadhayay</a>; <a href="mailto:mittas@microsoft.com"> Mitta Sai Chaithanya</a> Subject: [EXTERNAL] Re: LVM2 : performance drop even after deleting the snapshot </div> <o:p> </o:p> [Some people who received this message don't often get email from zdenek.kabelac@gmail.com. Learn why this is important at <a href="https://aka.ms/LearnAboutSenderIdentification">https://aka.ms/LearnAboutSenderIdentification</a> ] Dne 13. 10. 22 v 8:53 Pawan Sharma napsal(a): > adding this to lvm-devel mailing list also. > > Regards, > Pawan > ------------------------------------------------------------------------------ > *From:* Pawan Sharma > *Sent:* Wednesday, October 12, 2022 10:42 PM > *To:* linux-lvm@redhat.com <linux-lvm@redhat.com> > *Cc:* Mitta Sai Chaithanya <mittas@microsoft.com>; Kapil Upadhayay > <kupadhayay@microsoft.com> > *Subject:* LVM2 : performance drop even after deleting the snapshot > Hi Everyone, > > > We are evaluating lvm2 snapshots and doing performance testing on it. This is > what we are doing : > > 1. dump some data to lvm2 volume (using fio) > 2. take the snapshot > 3. delete the snapshot (no IOs anywhere after creating the snapshot) > 4. run the fio on lvm2 volume > > Here as you can see, we are just creating the snapshot and immediately > deleting it. There are no IOs to the main volume or anywhere. When we run the > fio after this (step 4) and we see around 50% drop in performance with > reference to the number we get in step 1. > > It is expected to see a performance drop if there is a snapshot because of the > COW. But here we deleted the snapshot, and it is not referring to any data > also. We should not see any performance drop here. > > Could someone please help me understand this behavior. Why are we seeing the > performance drop in this case? It seems like we deleted the snapshot but still > it is not deleted, and we are paying the COW penalty. > > System Info: > > OS : ubuntu 18.04 > Kernel : 5.4.0 > > # lvm version > LVM version:2.02.176(2) (2017-11-03) > Library version: 1.02.145 (2017-11-03) > Driver version:4.41.0 > > We also tried on latest ubuntu with newer version of LVM. We got the same > behavior. > > Hi Debugging 5 year old software is likely not going to get lot of attention from upstream. So please: a) reproduce the issue with some recent kernel & lvm2 b) take 'dmsetup table && dmsetup status' before you run every 'fio' test and present here your result in some form - otherwise we can hardly see what is the problem. What should be expected - if you use old/thick snapshots - when you 'drop' snapshot - you have your original intact LV - so results should mostly match results before you take the snapshot - but you clearly have to take into account if you use some 'SSD/NVMe' discarding and other things - so always run series of tests and average your results. If you use thin snapshot - that you can get various results depending on your settings of thin chunks, discard usage. Also maybe try your benchmark with different filesystems... Regards Zdenek<o:p></o:p> <o:p> </o:p> </div> </body> </html>