RBD volume not made available to Xen virtual guest on openSUSE 15.2 (with libvirt 6.0.0)

Tue Oct 27 23:35:57 UTC 2020

Thanks Jim,

Looking at the logs for the working and non-working setups, the command 
line for QEMU is identical and the qmp commands almost the same: they 
both do "query-chardev" and "query-vnc" and the working setup does an 
additional "qmp_capabilities" which is likely not relevant.

So I guess it must be in QEMU - I'll head over to the bug tracker.

Thanks for your help!

Marcel

On 28/10/20 9:26 am, Jim Fehlig wrote:
> On 10/22/20 6:48 PM, Marcel Juffermans wrote:
>> Hi there,
>>
>> Since upgrading to openSUSE 15.2 (which includes libvirt 6.0.0) the 
>> virtual guests don't get their RBD disks made available to them. On 
>> openSUSE 15.1 (which includes libvirt 5.1.0) that worked fine. The 
>> XML is as follows:
>>
>> <domain type='xen' id='7'>
>>    <name>mytwotel-a</name>
>>    <uuid>a56daa5d-c095-49d5-ae1b-00b38353614e</uuid>
>>    <description>mytwotel-a</description>
>>    <memory unit='KiB'>1048576</memory>
>>    <currentMemory unit='KiB'>1048576</currentMemory>
>>    <vcpu placement='static'>1</vcpu>
>>    <cputune>
>>      <vcpupin vcpu='0' cpuset='8-23'/>
>>    </cputune>
>>    <os>
>>      <type arch='x86_64' machine='xenpv'>linux</type>
>> <kernel>/usr/lib/grub2/x86_64-xen/grub.xen</kernel>
>>    </os>
>>    <clock offset='utc'/>
>>    <on_poweroff>destroy</on_poweroff>
>>    <on_reboot>restart</on_reboot>
>>    <on_crash>restart</on_crash>
>>    <devices>
>>      <disk type='network' device='disk'>
>>        <driver name='qemu' type='raw' cache='writethrough'/>
>>        <source protocol='rbd' name='guests/mytwotel-a'>
>>        <auth username='libvirt'>
>>          <secret type='ceph' 
>> uuid='3f88b59a-d85b-4b47-946d-a4c4cce3fec0'/>
>>        </auth>
>>        </source>
>>        <backingStore/>
>>        <target dev='xvda' bus='xen'/>
>>      </disk>
>>      <controller type='xenbus' index='0'/>
>>      <interface type='bridge'>
>>        <mac address='00:16:3e:a3:ba:9f'/>
>>        <source bridge='br0'/>
>>        <target dev='vif7.0'/>
>>      </interface>
>>      <console type='pty' tty='/dev/pts/0'>
>>        <source path='/dev/pts/0'/>
>>        <target type='xen' port='0'/>
>>      </console>
>>      <input type='mouse' bus='xen'/>
>>      <input type='keyboard' bus='xen'/>
>>      <memballoon model='xen'/>
>>    </devices>
>> </domain>
>>
>> The virtual guest starts, but then sits in the Grub 2 boot prompt 
>> because the disk is not available. The qemu log shows:
>>
>> qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
>> create drive: Could not open 
>> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
>> No such file or directory
>> qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
>> create drive: Could not open 
>> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
>> No such file or directory
>> qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
>> create drive: Could not open 
>> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
>> No such file or directory
>> qemu-system-i386: failed to create 'qdisk' device '51712': failed to 
>> create drive: Could not open 
>> 'rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\;none': 
>> No such file or directory
>> ...
>>
>> I tried to strace libvirtd. The results are as follows:
>>
>> On openSUSE 15.2 with libvirt 6.0.0 (not working), we see this:
>>
>> 1682  openat(AT_FDCWD, 
>> "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 
>> O_RDWR|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>> 1682  rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0
>> 1682  mmap(NULL, 1052672, PROT_READ|PROT_WRITE, 
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f538aefd000
>> 1682  mprotect(0x7f538aefd000, 4096, PROT_NONE <unfinished ...>
>> 1682  <... mprotect resumed>)           = 0
>> 1682  rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], [BUS USR1 ALRM 
>> IO], 8) = 0
>> 1682  rt_sigprocmask(SIG_BLOCK, NULL, [BUS USR1 ALRM IO], 8) = 0
>> 1682  mmap(NULL, 1052672, PROT_READ|PROT_WRITE, 
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
>> 1682  <... mmap resumed>)               = 0x7f538adfc000
>> 1682  mprotect(0x7f538adfc000, 4096, PROT_NONE <unfinished ...>
>> 1682  <... mprotect resumed>)           = 0
>> 1682  rt_sigprocmask(SIG_SETMASK, [BUS USR1 ALRM IO], <unfinished ...>
>> 1682  <... rt_sigprocmask resumed>[BUS USR1 ALRM IO], 8) = 0
>> 1682  write(2, "qemu-system-i386: failed to crea"..., 232 <unfinished 
>> ...>
>> ...
>>
>> On the other hand, on openSUSE 15.1 with libvirt 5.1.0 (working), we 
>> see this:
>>
>> 16267 openat(AT_FDCWD, 
>> "rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 
>> O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>> 16267 
>> stat("rbd:guests/mytwotel-a:id=libvirt:key=AQCAUpBbrcaiFxAA1sztXPbkdW1L54i99oUpyA==:auth_supported=cephx\\;none", 
>> 0x7fff83e2e2b0) = -1 ENOENT (No such file or directory)
>> 16267 access("/usr/lib64/qemu/block-rbd.so", F_OK) = 0
>> 16267 stat("/usr/lib64/qemu/block-rbd.so", {st_mode=S_IFREG|0644, 
>> st_size=27448, ...}) = 0
>> 16267 openat(AT_FDCWD, "/usr/lib64/qemu/block-rbd.so", 
>> O_RDONLY|O_CLOEXEC) = 60
>> 16267 read(60, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 
>> &\0\0\0\0\0\0"..., 832) = 832
>> 16267 fstat(60, {st_mode=S_IFREG|0644, st_size=27448, ...}) = 0
>> 16267 mmap(NULL, 2122672, PROT_READ|PROT_EXEC, 
>> MAP_PRIVATE|MAP_DENYWRITE, 60, 0) = 0x7f8e6030f000
>> 16267 mprotect(0x7f8e60315000, 2093056, PROT_NONE) = 0
>> 16267 mmap(0x7f8e60514000, 8192, PROT_READ|PROT_WRITE, 
>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 60, 0x5000) = 0x7f8e60514000
>> 16267 close(60)                         = 0
>> ...
>>
>> Note that the latter opens "/usr/lib64/qemu/block-rbd.so". That 
>> library *does* exist on openSUSE 15.2 but it doesn't seem to be used.
>
> It smells like a bug in the Leap 15.2 qemu package. Or perhaps in 
> xen's libxl library, which controls the qemu processes used by VMs. 
> I'd suggest entering a bug at bugzilla.opensuse.org.
>
>> I've tried to update libvirt to a newer version using the Open Build 
>> Service repos, but then ran into so many conflicting versions that I 
>> gave up.
>>
>> At this point I'm stuck. Does anyone have an idea I can try?
>
> We might be able to eliminate libxl by comparing the interaction with 
> qemu between the working and non-working setups. E.g. enable debug in 
> /etc/libvirt/libvirtd.conf, restart libvirtd, start the offending VM, 
> then look in /var/log/libvirt/libxl/libxl-driver.log for the qemu 
> command line created by libxl and qmp commands sent to qemu. If libxl 
> prepares qemu similarly between the two, then the issue is likely in 
> the qemu package.
>
> Regards,
> Jim
>