[libvirt] Stored secrets seem to get corrupted

[Wido den Hollander]

On 03-07-12 16:54, Daniel P. Berrange wrote:
> On Tue, Jul 03, 2012 at 04:42:54PM +0200, Wido den Hollander wrote:
>>
>>
>> On 03-07-12 15:13, Daniel P. Berrange wrote:
>>>>>
>>>>> That is really bizarre. Can you look at what is actually stored
>>>>> in the .base64 file each time ? And what 'secret-get-value'
>>>>> replies with ?
>>>>
>>>> I haven't been able to look into this any further, however: I just
>>>> downloaded 0.9.13 from the libvirt website and installed it on a
>>>> totally different host which is also running Ubuntu 12.04
>>>>
>>>> I wanted to start a virtual machine with RBD storage and that
>>>> failed, the secret was corrupted...
>>>>
>>>> The symptoms on this machine are exactly the same, the secret file
>>>> is just 2 bytes big.
>>>>
>>>> root at amd:~# ls -al /etc/libvirt/secrets/*.base64
>>>> -rw------- 1 root root 2 Jul  3 15:02
>>>> /etc/libvirt/secrets/69f9540e-f0ce-4184-8254-9b22efade5f2.base64
>>>> root at amd:~#
>>>>
>>>>>
>>>>> This is the correct behaviour tht I see myself too.
>>>>>
>>>>>> I verified that stack01 isn't out of disk space or out of inodes,
>>>>>> those are in the acceptable values range.
>>>>>>
>>>>>> Any suggestions?
>>>>>
>>>>> I think you'll probably need to add some more  VIR_DEBUG lines
>>>>> to secret_driver.c to see where in the process it is going
>>>>> wrong. Or perhaps strace libvirtd to see what it thinks it
>>>>> is writing out & whether any errors appear.
>>>>>
>>>>
>>>> I haven't added any VIR_DEBUG lines yet, but stracing the libvirtd
>>>> process doesn't show any fopen() nor fwrites() to any *.base64
>>>> files.
>>
>> I just added a couple of VIR_DEBUG lines to secret_driver.c and
>> found out that the base64 encoding is actually the problem.
>>
>> In secretSaveValue
>>
>>      VIR_DEBUG("WIDO Secret value: %s, size %lu", secret->value,
>> secret->value_size);
>>
>>      filename = secretBase64Path(driver, secret);
>>      if (filename == NULL)
>>          goto cleanup;
>>      base64_encode_alloc((char *)secret->value, secret->value_size,
>>                          &base64);
>>      if (base64 == NULL) {
>>          virReportOOMError();
>>          goto cleanup;
>>      }
>>
>>      VIR_DEBUG("WIDO Writing %s to %s with a length of %lu", base64,
>> filename, strlen(base64));
>>      if (replaceFile(filename, base64, strlen(base64)) < 0)
>>          goto cleanup;
>>
>>
>> The results I get back:
>>
>> $ virsh secret-set-value 322bccea-f2ed-4eae-a7e5-d0793ffb162d d2lkbw==
>>
>> 2012-07-03 14:02:57.065+0000: 4593: debug : secretSaveValue:297 :
>> WIDO Secret value: wido, size 4
>> 2012-07-03 14:02:57.065+0000: 4593: debug : secretSaveValue:309 :
>> WIDO Writing ���
>>
>>              to
>> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
>> with a length of 6
>>
>> Here you can see the secret value arrives at the secret driver in
>> tact, but the base64_encode_alloc seems to scramble the data.
>>
>> It should display base64 encoded data to write to 'filename', but
>> it's showing some binary stuff.
>
> Yeah, I'm damned I can understand what's broken at that point.
> The logs show the input is sensible, and we're calling the APIs
> the right way.
>
> Can you try to run libvirtd under valgrind eg, just run
>
>    valgrind /usr/sbin/libvirtd
>
>
> and then try to reproduce it. This would show if there is
> memory corruption happening somewhere

Yes, there is memory corruption somewhere. I never used valgrind before, 
but the output seems to show.

I ran libvirtd inside a screen, I've attached the screenlog with all the 
output.

At the end you'll see there is a misread. I was storing the base64 
encoded value of "wido".

Wido
-------------- next part --------------
A non-text attachment was scrubbed...
Name: screenlog.0.gz
Type: application/x-gzip
Size: 5908 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20120703/bdbe048c/attachment-0001.bin>