[libvirt-users] using RBD with libvirt 0.9.13

[Josh Durgin]

On 08/06/2012 12:12 PM, Jeff Strunk wrote:
> I'm having some trouble creating KVM domains with RBD block devices using
> virsh. I've managed to get virsh to define the domain, but it gives an error
> when trying to start the domain:
> error: Failed to start domain test0
> error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/3
> kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error connecting

This is the relevant error. If you don't have an /etc/ceph/ceph.conf
file, or it (or the keyring file possibly) is unreadable by qemu due to
apparmor/selinux etc, you'll need to specify all the information needed
to connect in the drive xml. This includes the monitor host and the
libvirt secret uuid, i.e.:

       <disk type='network' device='disk'>
         <driver name='qemu' type='raw'/>
         <source protocol='rbd' name='rbd/test0'>
           <host name='thinkmate3' port='6789'/>
           <host name='thinkmate4' port='6789'/>
         </source>
         <auth username='admin'>
           <secret type='ceph' uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/>
         </auth>
         <target dev='vda' bus='virtio'/>
         <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x0'/>
       </disk>

Josh

> kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:rbd/test0: No such file or directory
>
>>From libvirt's QEMU log:
> 2012-08-06 18:36:57.788+0000: starting up
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name test0 -S -M pc-1.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid 3e727624-f336-d674-4e0c-234641fc9987 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/test0.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:8c:e4:92,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus!
  -!
>   device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> Domain id=5 is tainted: high-privileges
> char device redirected to /dev/pts/2
> kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: error connecting
> kvm: -drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:rbd/test0: No such file or directory
> 2012-08-06 18:36:58.353+0000: shutting down
>
> I can, however, successfully run a KVM virtual machine with an RBD block
> device with the following command:
> kvm -drive file=rbd:rbd/test1,id=drive-virtio-disk0,format=raw -cdrom /var/lib/libvirt/images/ubuntu-12.04-server-amd64.iso
>
> kvm will boot with the "-drive file=rbd:rbd/test0,if=none,id=drive-virtio-disk0,format=raw"
> options, but the disk is not detected unless "if=none" is removed.
>
> Has anyone else figured this out?
>
> Thank you,
> Jeff
>
> Here is my rbd pool definition:
> ######### BEGIN
> <pool type="rbd">
> <name>rbd</name>
> <source>
> <name>rbd</name>
> <host name='thinkmate3' port='6789'/>
> <host name='thinkmate4' port='6789'/>
> <auth username='admin' type='ceph'>
> <secret uuid='f0836fcb-9c05-5ccf-653a-eb1d0308e247'/>
> </auth>
> </source>
> </pool>
> ######### END
>
> My rbd volumes:
> # virsh vol-list rbd
> Name                 Path
> -----------------------------------------
> test0                rbd/test0
> test1                rbd/test1
> test2                rbd/test2
>
> Here is my domain definition:
> ######### BEGIN
> <domain type='kvm'>
>    <name>test0</name>
>    <uuid>3e727624-f336-d674-4e0c-234641fc9987</uuid>
>    <memory unit='KiB'>1048576</memory>
>    <currentMemory unit='KiB'>1048576</currentMemory>
>    <vcpu placement='static'>1</vcpu>
>    <os>
>      <type arch='x86_64' machine='pc-1.0'>hvm</type>
>      <boot dev='hd'/>
>    </os>
>    <features>
>      <acpi/>
>      <apic/>
>      <pae/>
>    </features>
>    <clock offset='utc'/>
>    <on_poweroff>destroy</on_poweroff>
>    <on_reboot>restart</on_reboot>
>    <on_crash>restart</on_crash>
>    <devices>
>      <emulator>/usr/bin/kvm</emulator>
>      <disk type='network' device='disk'>
>        <driver name='qemu' type='raw'/>
>        <source protocol='rbd' name='rbd/test0'/>
>        <target dev='vda' bus='virtio'/>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
>      </disk>
>      <disk type='block' device='cdrom'>
>        <driver name='qemu' type='raw'/>
>        <target dev='hdc' bus='ide'/>
>        <readonly/>
>        <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>      </disk>
>      <controller type='ide' index='0'>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
>      </controller>
>      <controller type='usb' index='0'>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
>      </controller>
>      <interface type='network'>
>        <mac address='52:54:00:8c:e4:92'/>
>        <source network='default'/>
>        <model type='virtio'/>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>      </interface>
>      <serial type='pty'>
>        <target port='0'/>
>      </serial>
>      <console type='pty'>
>        <target type='serial' port='0'/>
>      </console>
>      <input type='mouse' bus='ps2'/>
>      <graphics type='vnc' port='-1' autoport='yes'/>
>      <video>
>        <model type='cirrus' vram='9216' heads='1'/>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>      </video>
>      <memballoon model='virtio'>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
>      </memballoon>
>    </devices>
> </domain>
> ######## END