[libvirt] Found mem leak in livirt, need help to debug

[Piotr Rybicki]

W dniu 2015-11-19 o 14:36, Piotr Rybicki pisze:
>
>
> W dniu 2015-11-19 o 11:07, Michal Privoznik pisze:
>> On 18.11.2015 15:33, Piotr Rybicki wrote:
>>> Hi.
>>>
>>> There is a mem leak in libvirt, when doing external snapshot (for backup
>>> purposes). My KVM domain uses raw storage images via libgfapi. I'm using
>>> latest 1.2.21 libvirt (although previous versions act the same).
>>>
>>> My bash script for snapshot backup uses series of shell commands (virsh
>>> connect to a remote libvirt host):
>>>
>>> * virsh domblklist KVM
>>> * qemu-img create -f qcow2 -o backing_file=gluster(...) - precreate
>>> backing file
>>> * virsh snapshot-create KVM SNAP.xml (...) - create snapshot from
>>> precreated XML snapshot file
>>> * cp main img file
>>> * virsh blockcommit KVM disk (...)
>>>
>>> Backup script works fine, however libvirtd process gets bigger and
>>> bigger each time I run this script.
>>>
>>> Some proof of memleak:
>>>
>>> 32017 - libvirtd pid
>>>
>>> When libvirt started:
>>> # ps p 32017 o vsz,rss
>>>     VSZ   RSS
>>> 585736 15220
>>>
>>> When I start KVM via virsh start KVM
>>> # ps p 32017 o vsz,rss
>>>     VSZ   RSS
>>> 1327968 125956
>>>
>>> When i start backup script, after snapshot is created (lots of mem
>>> allocated)
>>> # ps p 32017 o vsz,rss
>>>     VSZ   RSS
>>> 3264544 537632
>>>
>>> After backup script finished
>>> # ps p 32017 o vsz,rss
>>>     VSZ   RSS
>>> 3715920 644940
>>>
>>> When i start backup script for a second time, after snapshot is created
>>> # ps p 32017 o vsz,rss
>>>     VSZ   RSS
>>> 5521424 1056352
>>>
>>> And so on, until libvirt spills 'Out of memory' when connecting, ane
>>> being really huge process.
>>>
>>> Now, I would like to diagnose it further, to provide detailed
>>> information about memleak. I tried to use valgrind, but unfortunatelly
>>> I'm on Opteron 6380 platform, and valgrind doesn't support XOP quitting
>>> witch SIGILL.
>>>
>>> If someone could provide me with detailed information on how to get some
>>> usefull debug info about this memleak, i'll be more than happy to do it,
>>> and share results here.
>>
>> You can run libvirtd under valgrind (be aware that it will be slow as
>> snail), then run the reproducer and then just terminate the daemon
>> (CTRL+C). Valgrind will then report on all the leaks. When doing this I
>> usually use:
>>
>>    # valgrind --leak-check=full --show-reachable=yes \
>>    --child-silent-after-fork=yes libvirtd
>>
>> Remember to terminate the system-wide daemon firstly as the one started
>> under valgrind will die early since you can only have one deamon running
>> at the time.
>>
>> If you are unfamiliar with the output, share it somewhere and I will
>> take a look.
>>
>
> Thank You Michal.
>
> I finally managed to have valgrind running on Opteron 6380. I recomplied
> with -mno-xop glibc, libvirt and other revelant libs (just for others
> looking for solution for valgrind on Opteron).
>
> Gluster is at 3.5.4
>
> procedure is:
> start libvirtd
> start kvm
> run backup script (with external snapshot)
> stop kvm
> stop libvirtd
>
> Valgrind output:

Sorry, better valgrind output - showing problem:

valgrind --leak-check=full --show-reachable=yes 
--child-silent-after-fork=yes /usr/sbin/libvirtd --listen 2> valgrind.log

http://wikisend.com/download/314166/valgrind.log

Best regards
Piotr Rybicki