[libvirt] Stored secrets seem to get corrupted

[Wido den Hollander]

On 06/25/2012 04:54 PM, Daniel P. Berrange wrote:
> On Mon, Jun 25, 2012 at 04:37:48PM +0200, Wido den Hollander wrote:
>> Hi,
>>
>> On one of my systems I'm having troubles with my RBD storage backend.
>>
>> At first I thought it was a problem with my code, but after trying
>> the same code on a second machine I'm a bit confused.
>>
>> The problem is that the storage backend tries to retrieve the value
>> of a secret and base64 decode it, that fails.
>>
>> My debug log shows:
>
>> Notice this behavior:
>>
>> root at stack01:~# virsh secret-set-value
>> 322bccea-f2ed-4eae-a7e5-d0793ffb162d
>> AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
>> Secret value set
>>
>> root at stack01:~# md5sum
>> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
>> b4b147bc522828731f1a016bfa72c073
>> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
>> root at stack01:~# virsh secret-set-value
>> 322bccea-f2ed-4eae-a7e5-d0793ffb162d
>> AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
>> Secret value set
>>
>> root at stack01:~# md5sum
>> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
>> 927e2458c32cc3f6754d91694e41333f
>> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
>> root at stack01:~#
>>
>> As you can see, the md5sum of the file changes when I set the value
>> of the secret to the same.
>
> That is really bizarre. Can you look at what is actually stored
> in the .base64 file each time ? And what 'secret-get-value'
> replies with ?
>

The content of the .base64 is pure garbage, my terminal can't make 
anything of it.

What I do notice is that the .base64 file is only 2 bytes big, while it 
should be 40 bytes.

"secret-get-value" returns the correct data, but I think that is due to 
it being in memory. That also tells me that the writing to disk fails, 
in memory it is still fine.

When I restart libvirt I see:

secretLoadValue:406 : internal error invalid base64 in 
'/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64'

I checked my disk-space and inodes again, but those are all fine. I can 
write other files on the same FS without any problem.

I also made sure that AppArmor (Ubuntu) was turned off.

>> I verified that stack01 isn't out of disk space or out of inodes,
>> those are in the acceptable values range.
>>
>> Any suggestions?
>
> I think you'll probably need to add some more  VIR_DEBUG lines
> to secret_driver.c to see where in the process it is going
> wrong. Or perhaps strace libvirtd to see what it thinks it
> is writing out & whether any errors appear.
>

I'll try that.

Wido