[libvirt-users] Using libvirt to access Ceph RBDs with Xen

[thg]

Hi everybody,

my background: I'm doing Xen since 10++ years, many years with DRBD for
high availability, since some time I'm using preferable GlusterFS with
FUSE as replicated storage, where I place the image-files for the vms.

In my current project we started (successfully) with Xen/GlusterFS too,
but because the provider, where we placed the servers, uses widely CEPH,
we decided to switch too.

Unfortunately I'm new to CEPH, but with help of a technician, we have
running a 3 node CEPH-cluster now, that seems to work fine.

Hardware:
- Xeons, 24 Cores, 256 GB RAM,
  2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID)

Software we are using:
- CentOS 7.5.1804
- Kernel: 4.9.86-30.el7             @centos-virt-xen-48
- Xen: 4.8.3-5.el7                  @centos-virt-xen-48
- libvirt-xen: 4.1.0-2.xen48.el7    @centos-virt-xen-48
- Ceph: 2:12.2.5-0.el7              @Ceph


CEPH itself works, Xen too and even libvirt (I'm quite new in this too,
normally I use the XL-tools to manage my VMs) ;-) I can manually map an
RBD and start a PV-VM via libvirt, after modifying the XML-config and
defined the VM with libvirt.


That is what happens:

# virsh define xml/testvm3.xml
Domain testvm3 defined from xml/testvm3.xml

# virsh start --console testvm3
error: Failed to start domain testvm3
error: internal error: libxenlight failed to create new domain 'testvm3'

xml/testvm3.xml:
-> ...
  <devices>
    <disk type='network' device='disk'>
      <source protocol='rbd' name='devel-pool/testvm3.rbd'>
        <host name="10.20.30.1" port="6789"/>
        <host name="10.20.30.2" port="6789"/>
        <host name="10.20.30.3" port="6789"/>
      </source>
      <auth username='libvirt'>
        <secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/>
      </auth>
      <target dev='xvda' bus='xen'/>
    </disk>
...

Logs:

/var/log/libvirt/libxl/libxl-driver.log:
-> ...
2018-05-27 11:24:36.666+0000: libxl:
libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult
logfile /var/log/xen/bootloader.14.log
2018-05-27 11:24:36.666+0000: libxl:
libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216]
exited with error status 1
2018-05-27 11:24:36.667+0000: libxl:
libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3

/var/log/xen/bootloader.14.log:
->
Traceback (most recent call last):
  File "/usr/lib64/xen/bin/pygrub", line 896, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory:
'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789'


So the Xen-Python-script, that searches for the kernel/bootloader of the
VM, does not find the imagefile. But if I grab the above output and use
it with qemu, it does find the blockdevice:

# qemu-img info
"rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789"
->
image:
rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\;none:mon_host=10.20.30.1\:6789\;10.20.30.2\:6789\;10.20.30.3\:6789
file format: raw
virtual size: 16G (17179869184 bytes)
disk size: unavailable


Thus it seems, that CEPH-auth works and the VM-XML-config is interpreted
correct by libvirt. But the next step fails, when libvirt should pass
the blockdevice to Xen?


Anybody out there who uses a similar configuration? Any hints how to
debug this? Unfortunately I do not have anymore logs to look at.


Thank you very much,
-- 

Kind regards,

thg