[libvirt] [PATCH 1/1] qemu: add support for multiple gluster hosts

[Deepak C Shetty]

On 10/06/2015 02:53 PM, Peter Krempa wrote:
> On Tue, Oct 06, 2015 at 10:38:23 +0530, Deepak C Shetty wrote:
>> On 10/05/2015 07:33 PM, Peter Krempa wrote:
>>> On Mon, Oct 05, 2015 at 18:59:54 +0530, Deepak C Shetty wrote:
>>> [Note: Formatting of this mail is broken. Please configure your mail
>>> client to properly flow text in the plaintext mode.]
>> Sorry about that. I configured plain text format for libvir-list at redhat.com
>> domain in my thunderbird client and selected plain text as delivery format
>> for this specificmail.
> Looks better now, but you really should use plain text for any technical
> stuff.

Yup, willl do.

>
>>>> On 10/05/2015 04:31 PM, Peter Krempa wrote:
>>>>> On Mon, Oct 05, 2015 at 15:30:42 +0530, Prasanna Kumar Kalever wrote:
> [snip]
>
>>>>
>>>> * Create a qcow2 file for use with external snapshot
>>>>
>>>> qemu-img create -f qcow2 -b gluster://10.70.1.86/sample/CentOs7.qcow2
>>>> gluster://10.70.1.86/sample/newsnap.qcow2
>>> If you pre-create backing files to use with --reuse-external, you have
>>> to use the json syntax here to express the backing file.
>> We tried that, but qemu-img says invalid protocol/format when given a
>> json:"{...}" string in the above cmd, for backing and new file names.
>>
>> Qemu being used is built from source using the qemu patch thats
>> referenced with the link below, that has json support for gluster network
>> storage.
> Well, that probably means that the QEMU patch isn't entirely right in
> that case.
We just figured ....

1) qemu-img create -f qcow2 -b json:"{...}" 
gluster://hostname/volname/snapfile.qcow2 - works
2) qemu-img create -f qcow2 -b json:"{.../base.img}" 
json:"{.../snapfile.qcow2}"  - fails saying json: protocol unsupported

This is probably bcos qemu-img utility doesn't have support for json: 
syntax and IIRC in the qemu thread it was said
that we don't need to add this to the qemu-img utility as it adds complexity

So using #1 we are able to pre-create snapfiles that have backing-file: 
json string

>
>>>> * Create a domainsnapshot XML (snap.xml)
>>>>
>>>> <domainsnapshot>
>>>>        <disks>
>>>>            <disk name="vda" snapshot="external" type="network">
>>>>                <source protocol="gluster" name="sample/newsnap.qcow2">
>>>>                    <host name="10.70.1.86" port="24007"/>
>>>>                    <host name="10.70.1.88" port="24007"/>
>>>>                    <host name="10.70.1.87" port="24007"/>
>>>>                </source>
>>>>            </disk>
>>>>        </disks>
>>>> </domainsnapshot>
>>> You are doing it the other way around. I was speaking of doing a VM
>>> started with a multi-server gluster volume and then doing a snapshot
>>> wherever you like. This should result in a snapshot that has json
>>> definition of the backing file.
>> Not clear what you mean by 'other way around' ?
>>
>> A VM was created (HFM seen above) using libvirt with this patch. The HFM's
>> domain xml disk element is of type network & has 3 host's (hence
>> multi-server) entries.
>>
>> Also not clear on what you mean by 'whereeve u like' :) Can you pls
>> elaborate or provide
>> the high level virsh steps that we need to use ?
> [1]
>
> So, as you've stated above you already have a VM that is using a gluster
> disk, so if you use that one and:
>
> virsh snapshot-create-as VMNAME --disk-only --diskspec vda,snapshot=external,file=/tmp/snapshot
>
> virsh shutdown or virsh destroy VMNAME
>
> virsh start VMNAME
>
> So this basically creates a local image file that has a multi-host
> gluster volume as backing. libvirt needs to be able to successfuly start
> such VM, since we'd create them. Please also see [2]. Note that the
> above step modifies the configuration of the VM, so you might want to
> back it up prior to the steps above.
>
>> We are using virsh which is again built as part of the libvirt src build
>> with this patch
> This doesn't matter, virsh just forwards everything to the daemon. The
> daemon has to have the right support.
>
>>>> * Run virsh snap cmd (error in bold)
>>> [Note: using HTML mails on technical lists is not a good idea.]
>> Sorry again, this mail should be in plain text
>>
>>>> 0 :) dhcp1-86 ~/deepakcs $ virsh snapshot-create HFM --xmlfile
>>>> ./snap.xml --disk-only --reuse-external
>>>> *error: internal error: missing storage backend for network files using
>>>> gluster protocol*
>>> Erm, how are you attempting to test this if you don't even have libvirt
>>> with gluster support installed?
>> Hmm, what makes u think we don't have libvirt with gluster support
>> installed ?
>> libvirtd is running from source with this patch included.
> So your configure options or build environment is wrong then. The hint
> is the error message you've posted.
>
> error: internal error: missing storage backend for network files using gluster protocol

We configured it correctly now --with-gluster and --with-storage-gluster

>
> That is a generic message from the storage driver that is used in case
> when the appropriate backend is not available.
>
> As I've pointed out, it's the part of the storage driver that assists
> the qemu driver in regular operations like reading the image header or
> setting permissions on image files, not the actual pool/volume storage
> driver.

So just to understand this correctly, the storage driver is called as part
of parsing the backing file and yes we do see that it fails when backing 
file
is of json: syntax but works when its URI syntax.

This also means that the snapshot operation (of having the VM switch to the
new active file) is done on qemu side and when it returns back to libvirt,
libvirt is unable to parse as the backing file is json format, so fixing
the parse URI to support parseJSON needs to be done.

What happens when we don't use --reuse-external. IIUC the newfile is
created by libvirt so which part of libvirt needs to be fixed so ensure that
libvirt creates the file using json: syntax and not URI syntax ? The above
qemu-img snip shows that json is only supported as part of -b option
only, so will that be a inhibitor for libvirt when creating new files ?

>
> [2]
>
> Additionally this might be the reason you are not seeing any failures
> right now. The code that uses the storage backend queries whether the
> storage backend is actually supported before attempting to use it, so
> please make sure that your daemon is properly built with gluster
> support.

pkalever is now working with libvirtd having gluster support
We will know more as we make more experiments... the whole gamut
of network disk support in qemu/libvirt is confusing esp with the addn
of json and URI syntax both :) So pls bear with our questions above :)

thanx,
deepak