[libvirt] Found mem leak in livirt, need help to debug

[Piotr Rybicki]

W dniu 2015-11-19 o 11:07, Michal Privoznik pisze:
> On 18.11.2015 15:33, Piotr Rybicki wrote:
>> Hi.
>>
>> There is a mem leak in libvirt, when doing external snapshot (for backup
>> purposes). My KVM domain uses raw storage images via libgfapi. I'm using
>> latest 1.2.21 libvirt (although previous versions act the same).
>>
>> My bash script for snapshot backup uses series of shell commands (virsh
>> connect to a remote libvirt host):
>>
>> * virsh domblklist KVM
>> * qemu-img create -f qcow2 -o backing_file=gluster(...) - precreate
>> backing file
>> * virsh snapshot-create KVM SNAP.xml (...) - create snapshot from
>> precreated XML snapshot file
>> * cp main img file
>> * virsh blockcommit KVM disk (...)
>>
>> Backup script works fine, however libvirtd process gets bigger and
>> bigger each time I run this script.
>>
>> Some proof of memleak:
>>
>> 32017 - libvirtd pid
>>
>> When libvirt started:
>> # ps p 32017 o vsz,rss
>>     VSZ   RSS
>> 585736 15220
>>
>> When I start KVM via virsh start KVM
>> # ps p 32017 o vsz,rss
>>     VSZ   RSS
>> 1327968 125956
>>
>> When i start backup script, after snapshot is created (lots of mem
>> allocated)
>> # ps p 32017 o vsz,rss
>>     VSZ   RSS
>> 3264544 537632
>>
>> After backup script finished
>> # ps p 32017 o vsz,rss
>>     VSZ   RSS
>> 3715920 644940
>>
>> When i start backup script for a second time, after snapshot is created
>> # ps p 32017 o vsz,rss
>>     VSZ   RSS
>> 5521424 1056352
>>
>> And so on, until libvirt spills 'Out of memory' when connecting, ane
>> being really huge process.
>>
>> Now, I would like to diagnose it further, to provide detailed
>> information about memleak. I tried to use valgrind, but unfortunatelly
>> I'm on Opteron 6380 platform, and valgrind doesn't support XOP quitting
>> witch SIGILL.
>>
>> If someone could provide me with detailed information on how to get some
>> usefull debug info about this memleak, i'll be more than happy to do it,
>> and share results here.
>
> You can run libvirtd under valgrind (be aware that it will be slow as
> snail), then run the reproducer and then just terminate the daemon
> (CTRL+C). Valgrind will then report on all the leaks. When doing this I
> usually use:
>
>    # valgrind --leak-check=full --show-reachable=yes \
>    --child-silent-after-fork=yes libvirtd
>
> Remember to terminate the system-wide daemon firstly as the one started
> under valgrind will die early since you can only have one deamon running
> at the time.
>
> If you are unfamiliar with the output, share it somewhere and I will
> take a look.
>

Thank You Michal.

I finally managed to have valgrind running on Opteron 6380. I recomplied 
with -mno-xop glibc, libvirt and other revelant libs (just for others 
looking for solution for valgrind on Opteron).

Gluster is at 3.5.4

procedure is:
start libvirtd
start kvm
run backup script (with external snapshot)
stop kvm
stop libvirtd

Valgrind output:
http://wikisend.com/download/457046/valgrind.log

I' happy to test patches against libvirt 1.2.21 and retest.

Best regards
Piotr Rybicki