[libvirt] Found mem leak in livirt, need help to debug

[Piotr Rybicki]

W dniu 2015-11-19 o 17:31, Michal Privoznik pisze:
>
> procedure is:
> start libvirtd
> start kvm
> run backup script (with external snapshot)
> stop kvm
> stop libvirtd
>
> Valgrind output:
>> Sorry, better valgrind output - showing problem:
>>
>> valgrind --leak-check=full --show-reachable=yes
>> --child-silent-after-fork=yes /usr/sbin/libvirtd --listen 2> valgrind.log
>>
>> http://wikisend.com/download/314166/valgrind.log
> Interesting. I'm gonna post a couple of errors here, so that they don't get lost meanwhile:
>
> ==2650== 7,692,288 bytes in 2 blocks are still reachable in loss record 1,444 of 1,452
> ==2650==    at 0x4C2BFC8: calloc (vg_replace_malloc.c:711)
> ==2650==    by 0x1061335C: __gf_default_calloc (mem-pool.h:75)
> ==2650==    by 0x106137D2: __gf_calloc (mem-pool.c:104)
> ==2650==    by 0x1061419D: mem_pool_new_fn (mem-pool.c:316)
> ==2650==    by 0xFD69DDA: glusterfs_ctx_defaults_init (glfs.c:110)
> ==2650==    by 0xFD6AC31: glfs_new@@GFAPI_3.4.0 (glfs.c:558)
> ==2650==    by 0xF90321E: virStorageFileBackendGlusterInit (storage_backend_gluster.c:611)
> ==2650==    by 0xF8F43AF: virStorageFileInitAs (storage_driver.c:2736)
> ==2650==    by 0x115AE41A: qemuDomainStorageFileInit (qemu_domain.c:2929)
> ==2650==    by 0x1163DE5A: qemuDomainSnapshotCreateSingleDiskActive (qemu_driver.c:14201)
> ==2650==    by 0x1163E604: qemuDomainSnapshotCreateDiskActive (qemu_driver.c:14371)
> ==2650==    by 0x1163ED27: qemuDomainSnapshotCreateActiveExternal (qemu_driver.c:14559)
> ==2650==
> ==2650== 7,692,288 bytes in 2 blocks are still reachable in loss record 1,445 of 1,452
> ==2650==    at 0x4C2BFC8: calloc (vg_replace_malloc.c:711)
> ==2650==    by 0x1061335C: __gf_default_calloc (mem-pool.h:75)
> ==2650==    by 0x106137D2: __gf_calloc (mem-pool.c:104)
> ==2650==    by 0x1061419D: mem_pool_new_fn (mem-pool.c:316)
> ==2650==    by 0xFD69DDA: glusterfs_ctx_defaults_init (glfs.c:110)
> ==2650==    by 0xFD6AC31: glfs_new@@GFAPI_3.4.0 (glfs.c:558)
> ==2650==    by 0xF90321E: virStorageFileBackendGlusterInit (storage_backend_gluster.c:611)
> ==2650==    by 0xF8F43AF: virStorageFileInitAs (storage_driver.c:2736)
> ==2650==    by 0xF8F4B0A: virStorageFileGetMetadataRecurse (storage_driver.c:2996)
> ==2650==    by 0xF8F4F66: virStorageFileGetMetadata (storage_driver.c:3119)
> ==2650==    by 0x115AE629: qemuDomainDetermineDiskChain (qemu_domain.c:2980)
> ==2650==    by 0x1163E843: qemuDomainSnapshotCreateDiskActive (qemu_driver.c:14421)
> ==2650==
> ==2650== 7,692,288 bytes in 2 blocks are still reachable in loss record 1,446 of 1,452
> ==2650==    at 0x4C2BFC8: calloc (vg_replace_malloc.c:711)
> ==2650==    by 0x1061335C: __gf_default_calloc (mem-pool.h:75)
> ==2650==    by 0x106137D2: __gf_calloc (mem-pool.c:104)
> ==2650==    by 0x1061419D: mem_pool_new_fn (mem-pool.c:316)
> ==2650==    by 0xFD69DDA: glusterfs_ctx_defaults_init (glfs.c:110)
> ==2650==    by 0xFD6AC31: glfs_new@@GFAPI_3.4.0 (glfs.c:558)
> ==2650==    by 0xF90321E: virStorageFileBackendGlusterInit (storage_backend_gluster.c:611)
> ==2650==    by 0xF8F43AF: virStorageFileInitAs (storage_driver.c:2736)
> ==2650==    by 0xF8F4B0A: virStorageFileGetMetadataRecurse (storage_driver.c:2996)
> ==2650==    by 0xF8F4DC5: virStorageFileGetMetadataRecurse (storage_driver.c:3054)
> ==2650==    by 0xF8F4F66: virStorageFileGetMetadata (storage_driver.c:3119)
> ==2650==    by 0x115AE629: qemuDomainDetermineDiskChain (qemu_domain.c:2980)
>
>
> So, I think that we are missing few virStorageFileDeinit() calls somewhere. This is a very basic scratch:
>
> diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
> index f0ce78b..bdb511f 100644
> --- a/src/qemu/qemu_domain.c
> +++ b/src/qemu/qemu_domain.c
> @@ -2970,9 +2970,10 @@ qemuDomainDetermineDiskChain(virQEMUDriverPtr driver,
>           goto cleanup;
>   
>       if (disk->src->backingStore) {
> -        if (force_probe)
> +        if (force_probe) {
> +            virStorageFileDeinit(disk->src);
>               virStorageSourceBackingStoreClear(disk->src);
> -        else
> +        } else
>               goto cleanup;
>       }
>   
> diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
> index 2192ad8..dd9a89a 100644
> --- a/src/qemu/qemu_process.c
> +++ b/src/qemu/qemu_process.c
> @@ -5256,6 +5256,7 @@ void qemuProcessStop(virQEMUDriverPtr driver,
>           dev.type = VIR_DOMAIN_DEVICE_DISK;
>           dev.data.disk = disk;
>           ignore_value(qemuRemoveSharedDevice(driver, &dev, vm->def->name));
> +        virStorageFileDeinit(disk->src);
>       }
>   
>       /* Clear out dynamically assigned labels */
>
>
> Can you apply it, build libvirt and give it a try? valgrind should report much fewer leaks.
>
Looks like it doesn't make much of a difference :(
http://wikisend.com/download/158168/valgrind2.log

Best regards
Piotr Rybicki