[libvirt] Found mem leak in livirt, need help to debug

[Michal Privoznik]

On 18.11.2015 15:33, Piotr Rybicki wrote:
> Hi.
> 
> There is a mem leak in libvirt, when doing external snapshot (for backup
> purposes). My KVM domain uses raw storage images via libgfapi. I'm using
> latest 1.2.21 libvirt (although previous versions act the same).
> 
> My bash script for snapshot backup uses series of shell commands (virsh
> connect to a remote libvirt host):
> 
> * virsh domblklist KVM
> * qemu-img create -f qcow2 -o backing_file=gluster(...) - precreate
> backing file
> * virsh snapshot-create KVM SNAP.xml (...) - create snapshot from
> precreated XML snapshot file
> * cp main img file
> * virsh blockcommit KVM disk (...)
> 
> Backup script works fine, however libvirtd process gets bigger and
> bigger each time I run this script.
> 
> Some proof of memleak:
> 
> 32017 - libvirtd pid
> 
> When libvirt started:
> # ps p 32017 o vsz,rss
>    VSZ   RSS
> 585736 15220
> 
> When I start KVM via virsh start KVM
> # ps p 32017 o vsz,rss
>    VSZ   RSS
> 1327968 125956
> 
> When i start backup script, after snapshot is created (lots of mem
> allocated)
> # ps p 32017 o vsz,rss
>    VSZ   RSS
> 3264544 537632
> 
> After backup script finished
> # ps p 32017 o vsz,rss
>    VSZ   RSS
> 3715920 644940
> 
> When i start backup script for a second time, after snapshot is created
> # ps p 32017 o vsz,rss
>    VSZ   RSS
> 5521424 1056352
> 
> And so on, until libvirt spills 'Out of memory' when connecting, ane
> being really huge process.
> 
> Now, I would like to diagnose it further, to provide detailed
> information about memleak. I tried to use valgrind, but unfortunatelly
> I'm on Opteron 6380 platform, and valgrind doesn't support XOP quitting
> witch SIGILL.
> 
> If someone could provide me with detailed information on how to get some
> usefull debug info about this memleak, i'll be more than happy to do it,
> and share results here.

You can run libvirtd under valgrind (be aware that it will be slow as
snail), then run the reproducer and then just terminate the daemon
(CTRL+C). Valgrind will then report on all the leaks. When doing this I
usually use:

  # valgrind --leak-check=full --show-reachable=yes \
  --child-silent-after-fork=yes libvirtd

Remember to terminate the system-wide daemon firstly as the one started
under valgrind will die early since you can only have one deamon running
at the time.

If you are unfamiliar with the output, share it somewhere and I will
take a look.

Michal