RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)

[Marcel Juffermans]

Hi there,

Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the 
virtual guests don't get their RBD disks made available to them. On 
openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The XML 
is as follows:

<domain type='xen' id='7'>
   <name>mytwotel-a</name>
   <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid>
   <description>mytwotel-a</description>
   <memory unit='KiB'>1048576</memory>
   <currentMemory unit='KiB'>1048576</currentMemory>
   <vcpu placement='static'>1</vcpu>
   <cputune>
     <vcpupin vcpu='0' cpuset='8-23'/>
   </cputune>
   <os>
     <type arch='x86_64' machine='xenpv'>linux</type>
<kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel>
   </os>
   <clock offset='utc'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
   <on_crash>restart</on_crash>
   <devices>
     <disk type='network' device='disk'>
       <driver name='qemu' type='raw' cache='writethrough'/>
       <source protocol='rbd' name='guests/mytwotel-a'>
       <auth username='libvirt'>
         <secret type='ceph' uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/>
       </auth>
       </source>
       <backingStore/>
       <target dev='xvda' bus='xen'/>
     </disk>
     <controller type='xenbus' index='0'/>
     <interface type='bridge'>
       <mac address='00:16:3e:a3:ba:9f'/>
       <source bridge='br0'/>
       <target dev='vif7.0'/>
     </interface>
     <console type='pty' tty='/dev/pts/0'>
       <source path='/dev/pts/0'/>
       <target type='xen' port='0'/>
     </console>
     <input type='mouse' bus='xen'/>
     <input type='keyboard' bus='xen'/>
     <memballoon model='xen'/>
   </devices>
</domain>

The virtual guest starts, but then sits in the Grub 2 boot prompt 
because the disk is not available. The qemu log shows:

qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
create drive: Could not open 
'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
No such file or directory
qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
create drive: Could not open 
'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
No such file or directory
qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
create drive: Could not open 
'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
No such file or directory
qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
create drive: Could not open 
'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
No such file or directory
...

I tried to strace libvirtd. The results are as follows:

On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:

1682  openat(AT_FDCWD, 
"rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 
O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1682  rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0
1682  mmap(NULL, 1052672, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000
1682  mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...>
1682  <... mprotect resumed>)           = 0
1682  rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM 
IO], 8) = 0
1682  rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0
1682  mmap(NULL, 1052672, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
1682  <... mmap resumed>)               = 0x7f538adfc000
1682  mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...>
1682  <... mprotect resumed>)           = 0
1682  rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...>
1682  <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0
1682  write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished ...>
...

On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we see 
this:

16267 openat(AT_FDCWD, 
"rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 
O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
16267 
stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 
0x7fff83e2e2b0) = -1 ENOENT (No such file or directory)
16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0
16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, 
st_size=27448, ...}) = 0
16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", 
O_RDONLY|O_CLOEXEC) = 60
16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 
&\0\0\0\0\0\0"..., 832) = 832
16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0
16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000
16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0
16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000
16267 close(60)                         = 0
...

Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That library 
*does* exist on openSUSE 15.2 but it doesn't seem to be used.

I've tried to update libvirt to a newer version using the Open Build 
Service repos, but then ran into so many conflicting versions that I 
gave up.

At this point I'm stuck. Does anyone have an idea I can try?

Many thanks,

Marcel