[libvirt-users] Using libvirt to access Ceph RBDs with Xen

[Jim Fehlig]

On 05/27/2018 05:42 AM, thg wrote:
> Hi everybody,
> 
> my background: I'm doing Xen since 10++ years, many years with DRBD for
> high availability, since some time I'm using preferable GlusterFS with
> FUSE as replicated storage, where I place the image-files for the vms.
> 
> In my current project we started (successfully) with Xen/GlusterFS too,
> but because the provider, where we placed the servers, uses widely CEPH,
> we decided to switch too.
> 
> Unfortunately I'm new to CEPH, but with help of a technician, we have
> running a 3 node CEPH-cluster now, that seems to work fine.
> 
> Hardware:
> - Xeons, 24 Cores, 256 GB RAM,
>    2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID)
> 
> Software we are using:
> - CentOS 7.5.1804
> - Kernel: 4.9.86-30.el7             @centos-virt-xen-48
> - Xen: 4.8.3-5.el7                  @centos-virt-xen-48
> - libvirt-xen: 4.1.0-2.xen48.el7    @centos-virt-xen-48
> - Ceph: 2:12.2.5-0.el7              @Ceph
> 
> 
> CEPH itself works, Xen too and even libvirt (I'm quite new in this too,
> normally I use the XL-tools to manage my VMs) ;-) I can manually map an
> RBD and start a PV-VM via libvirt, after modifying the XML-config and
> defined the VM with libvirt.
> 
> 
> That is what happens:
> 
> # virsh define xml/testvm3.xml
> Domain testvm3 defined from xml/testvm3.xml
> 
> # virsh start --console testvm3
> error: Failed to start domain testvm3
> error: internal error: libxenlight failed to create new domain 'testvm3'
> 
> xml/testvm3.xml:
> -> ...
>    <devices>
>      <disk type='network' device='disk'>
>        <source protocol='rbd' name='devel-pool/testvm3.rbd'>
>          <host name="10.20.30.1" port="6789"/>
>          <host name="10.20.30.2" port="6789"/>
>          <host name="10.20.30.3" port="6789"/>
>        </source>
>        <auth username='libvirt'>
>          <secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/>
>        </auth>
>        <target dev='xvda' bus='xen'/>
>      </disk>
> ...
> 
> Logs:
> 
> /var/log/libvirt/libxl/libxl-driver.log:
> -> ...
> 2018-05-27 11:24:36.666+0000: libxl:
> libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
> logfile /var/log/xen/bootloader.14.log
> 2018-05-27 11:24:36.666+0000: libxl:
> libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216]
> exited with error status 1
> 2018-05-27 11:24:36.667+0000: libxl:
> libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3
> 
> /var/log/xen/bootloader.14.log:
> ->
> Traceback (most recent call last):
>    File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
>      part_offs = get_partition_offsets(file)
>    File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
>      image_type = identify_disk_image(file)
>    File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
>      fd = os.open(file, os.O_RDONLY)
> OSError: [Errno 2] No such file or directory:
> 'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'

pygrub doesn't work with rbd. But why not use pvgrub instead, which will work 
with rbd, and doesn't require mounting foreign (possibly hostile) images in dom0?

Regards,
Jim