[libvirt] how to enable ceph-rbd in libvirt.

Fri Oct 12 07:06:45 UTC 2012

Sorry, I forgot libvirt/qemu eats stderr. The extra logging wasn't 
captured.

Could you try:

name='cloud/testrbd:debug_ms=1:debug_rbd=20:debug_monc=10:log_file=/tmp/qemu-rbd.log'

If /tmp/qemu-rbd.log isn't full of debug messages, make sure selinux or 
apparmor aren't preventing
qemu from writing to it.

Josh

On 2012-10-11 22:31, yue wrote:
> hi, i add debug options related to rbd in xpSP3.xml. but error seems
> same with privious.
> ----libvirtd.log------------------
> erfaces
> 2012-10-12 13:25:28.267+0000: 9023: error : virNodeNumOfDevices:13984
> : this function is not supported by the connection driver:
> virNodeNumOfDevices
> 2012-10-12 13:25:29.264+0000: 9025: error :
> virConnectNumOfInterfaces:11014 : this function is not supported by
> the connection driver: virConnectNumOfInterfaces
> 2012-10-12 13:25:29.266+0000: 9023: error :
> virConnectNumOfDefinedInterfaces:11098 : this function is not
> supported by the connection driver: virConnectNumOfDefinedInterfaces
> 2012-10-12 13:25:29.267+0000: 9022: error : virNodeNumOfDevices:13984
> : this function is not supported by the connection driver:
> virNodeNumOfDevices
> 2012-10-12 13:25:30.264+0000: 9025: error :
> virConnectNumOfInterfaces:11014 : this f unction is not supported by
> the connection driver: virConnectNumOfInterfaces
> 2012-10-12 13:25:30.267+0000: 9023: error :
> virConnectNumOfDefinedInterfaces:11098 : this function is not
> supported by the connection driver: virConnectNumOfDefinedInterfaces
> 2012-10-12 13:25:30.269+0000: 9022: error : virNodeNumOfDevices:13984
> : this function is not supported by the connection driver:
> virNodeNumOfDevices
> 2012-10-12 13:25:30.523+0000: 9021: warning : qemuDomainObjTaint:1364
> : Domain id=1 name='xpSP3' uuid=b45bd66a-6700-4905-c5f4-4c799413d7b7
> is tainted: high-privileges
> 2012-10-12 13:25:30.943+0000: 9021: error : qemuMonitorOpenUnix:293 :
> failed to connect to monitor socket: No such process
> 2012-10-12 13:25:30.943+0000: 9021: info : qemuConnectMonitor:1247 :
> Failed to connect monitor for xpSP3
> 2012-10-12 13:25:30.943+0000: 9021: error :
> qemuProcessWaitForMonitor:1634 : internal error process exited while
> connecting to monitor: char device redirected to /dev/pts/4
> qemu-kvm: -drive
> 
> file=rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough,aio=native:
> could not open disk image
> 
> rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789:
> No such file or directory
>
> ---------xpSP3.log---is attached.---
> 2012-10-12 13:11:33.133+0000: starting up
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root
> LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name xpSP3 -S
> -M rhel6.2.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1
> -uuid b45bd66a-6700-4905-c5f4-4c799413d7b7 -nodefconfig -nodefaults
> -chardev socket,id=charmonitor,path=/var/lib/libvirt/
> qemu/xpSP3.monitor,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=readline -rtc
> base=localtime,driftfix=slew -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> 
> file=/var/lib/libvirt/images/xpSP3.img,if=none,id=drive-ide0-0-0,format=raw,cache=none
> -device
> 
> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -drive
> 
> file=rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough,aio=native
> -device
> 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0
> -netdev tap,fd=19,id=hostnet0 -device
> 
> e1000,netdev=hostnet0,id=net0,mac=52:54:00:2a:8c:48,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -device 
> usb-tablet,id=input0
> -vnc 0.0.0.0:0 -vga std -device 
> intel-hda,id=sound0,bus=pci.0,addr=0x4
> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> Domain id=1 is tainted: high-privileges
> 2012-10-12 13:11:33.144+0000: 7815: debug : virFileClose:72 : Closed 
> fd 18
> 2012-10-12 13:11:33.145+0000: 7815: debug : virFileClose:72 : Closed 
> fd 24
> 2012-10-12 13:11:33.145+0000: 7815: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.146+0000: 7818: debug : virCommandHook:2058 : Run
> hook 0x7faa10a62030 0x7faa160d4370
> 2012-10-12 13:11:33.146+0000: 7818: debug : qemuProcessHook:2589 :
> Obtaining domain lock
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virDomainLockProcessStart:164 : plugin=0x7faa080020a0
> dom=0x7faa0804af40 paused=1 fd=0x7faa160d3998
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virDomainLockManagerNew:123 : plugin=0x7faa080020a0 
> dom=0x7faa0804af40
> withResources=1
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virLockManagerPluginGetDriver:280 : plugin=0x7faa080020a0
> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockManagerNew:304 :
> driver=0x3712e9cb20 type=0 nparams=4 params=0x7faa160d3830 flags=0
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virLockManagerLogParams:104 : key=uuid type=uuid
> value=b45bd66a-6700-4905-c5f4-4c799413d7b7
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virLockManagerLogParams:100 : key=name type=string value=xpSP3
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virLockManagerLogParams:88 : key=id type=uint value=1
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virLockManagerLogParams:88 : key=pid type=uint value=7818
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virDomainLockManagerNew:135 : Adding leases
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virDomainLockManagerNew:140 : Adding disks
> 2012-10-12 13:11:33.146+0000: 7818: debug :
> virDomainLockManagerAddDisk:86 : Add disk
> /var/lib/libvirt/images/xpSP3.img
> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockMan
> agerAddResource:333 : lock=0x7faa08060820 type=0
> name=/var/lib/libvirt/images/xpSP3.img nparams=0 params=(nil) flags=0
> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockManagerAcquire:350
> : lock=0x7faa08060820 state='(null)' flags=3 fd=0x7faa160d3998
> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockManagerFree:387 :
> lock=0x7faa08060820
> 2012-10-12 13:11:33.146+0000: 7818: debug : qemuProcessHook:2614 :
> Moving process to cgroup
> 2012-10-12 13:11:33.146+0000: 7818: debug : virCgroupNew:617 : New
> group /libvirt/qemu/xpSP3
> 2012-10-12 13:11:33.151+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/mapping 0:cpu at /cgroup/cpu in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/mapping 2:cpuset at /cgroup/cpuset in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/ma pping 3:memory at /cgroup/memory in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/mapping 4:devices at /cgroup/devices in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/mapping 5:freezer at /cgroup/freezer in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
> Detected mount/mapping 6:blkio at /cgroup/blkio in
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:535 :
> Make group /libvirt/qemu/xpSP3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/cpu/libvirt/qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/cpuacct/libvirt/qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/cpuset/libvirt/qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/memory/libvirt/ qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/devices/libvirt/qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/freezer/libvirt/qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
> Make controller /cgroup/blkio/libvirt/qemu/xpSP3/
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/cpu/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/cpuacct/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/cpuset/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/memory/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/devices/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/freezer/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
> : Set value '/cgroup/blkio/libvirt/qemu/xpSP3/tasks' to '7818'
> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed 
> fd 3
> 2012-10-12 13:11:33.156+0000: 7818: debug :
> qemuProcessInitCpuAffinity:1848 : Setting CPU affinity
> 2 012-10-12 13:11:33.156+0000: 7818: debug :
> qemuProcessInitCpuAffinity:1883 : Set CPU affinity with specified
> cpuset
> 2012-10-12 13:11:33.156+0000: 7818: debug : qemuProcessHook:2626 :
> Setting up security labelling
> 2012-10-12 13:11:33.157+0000: 7818: debug : virGetUserIDByName:2539 :
> User record for user '0' does not exist
> 2012-10-12 13:11:33.157+0000: 7818: debug : virGetGroupIDByName:2621
> : Group record for group '0' does not exist
> 2012-10-12 13:11:33.157+0000: 7818: debug :
> virSecurityDACSetProcessLabel:861 : Dropping privileges of DEF to 0:0
> 2012-10-12 13:11:33.157+0000: 7818: debug : qemuProcessHook:2633 :
> Hook complete ret=0
> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2060 : 
> Done hook 0
> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2074 :
> Notifying parent for handshake start on 21
> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2098 :
> Waiting on parent for handshake complete on 22
> 2012-10-12 13:11:33.157+0 000: 7818: debug : virFileClose:72 : Closed 
> fd 21
> 2012-10-12 13:11:33.157+0000: 7818: debug : virFileClose:72 : Closed 
> fd 22
> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2119 : 
> Hook is done 0
> char device redirected to /dev/pts/4
> qemu-kvm: -drive
> 
> file=rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough,aio=native:
> could not open disk image
> 
> rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789:
> No such file or directory
> 2012-10-12 13:11:33.557+0000: shutting down
>
> ----
>
> At 2012-10-12 00:04:19,"Josh Durgi
>  n" <josh.durgin at inktank.com> wrote:
>>On 10/11/2012 12:00 AM, yue wrote:
>>> how should i do? i configurate guest-os.xml ,add rbd disk manually. 
>>> then virsh start xpSP3
>>
>>That's the right way to do it.
>>
>>> -------rbd disk------------
>>> <disk type='network' device='disk'>
>>> <driver name='qemu' type='raw' cache='writethrough' io='native'/>
>>> <auth username='cloud'>
>>> <secret type='ceph' uuid='7a91dc24-b072-43c4-98fb-4b2415322b0f'/>
>>> </auth>
>>> <source protocol='rbd' name='cloud/testrbd'>
>>> <host name='192.168.10.4' port='6789'/>
>>> </source>
>>> <target dev='vdb' bus='virtio'/>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' 
>>> function='0x0'/>
>>> </disk>
>>> ---error log---
>>>
>>> [root at KVM-ip11 git]# virsh start xpSP3
>>> error: Failed to start domain xpSP3
>>> error: internal error process exited while connecting to monitor: 
>>> char device redirected to /dev/pts/4
>>> qemu-kvm: -drive 
>>> file=rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk1,format=raw,cache=writethrough,aio=native: 
>>> could not open disk image 
>>> rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789: 
>>> No such file or directory
>>> ---xpSP3.log-----
>>> 2012-10-11 14:57:58.118+0000: starting up
>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root 
>>> LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name xpSP3 -S 
>>> -M rhel6.2.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 
>>> -uuid b45bd66a-6700-4905-c5f4-4c799413d7b7 -nodefconfig -nodefaults 
>>> -chardev 
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/xpSP3.monitor,server,nowait 
>>> -mon chardev=charmonitor,id=monitor,mode=readline -rtc 
>>> base=localtime,driftfix=slew -device 
>>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
>>> file=/var/lib/libvirt/images/xpSP3.img,if=none,id=drive-ide0-0-0,format=raw,cache=none 
>>> -device 
>>> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
>>> -drive 
>>> file=rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,
>  if=none,id=drive-virtio-disk1,format=raw -device
> 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
> -netdev tap,fd=19,id=hostnet0 -device e1000,netdev!
>>> =hostnet0,id=net0,mac=52:54:00:2a:8c:48,bus=pci.0,addr=0x3 -chardev 
>>> pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 
>>> -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga std -device 
>>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device 
>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device 
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>>> Domain id=3 is tainted: high-privileges
>>> char device redirected to /dev/pts/4
>>> qemu-kvm: -drive 
>>> file=rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk1,format=raw: 
>>> could not open disk image 
>>> rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789: 
>>> No such file or directory
>>> 2012-10-11 14:57:58.548+0000: shutting down
>>
>>It's unclear where the actual problem is - could you try again with
>>name='cloud/testrbd:debug_ms=1:debug_rbd=20:debug_monc=10:log_to_stderr=true'
>>and post the instance log (i.e. xpSP3.log)?
>>
>>Josh
>>
>>>
>>>
>>>
>>>
>>>
>>> At 2012-10-11 14:48:43,"Josh Durgin" <josh.durgin at inktank.com> 
>>> wrote:
>>>> On 10/10/2012 10:52 PM, yue wrote:
>>>>> hi, now i rebuild libvirtd, version 0.10.2. and define ceph 
>>>>> secret
>>>>> ,define ceph storage-pool. all fine up to now.
>>>>> i operate guest-os through virt-manager-GUI . i add a
>>>>> rbd-file(qemu-img create before) from ceph-rbd-pool, click 
>>>>> 'finish',it
>>>>> pop error "NoneType object has no attribute startswith", someone 
>>>>> say it
>>>>> is python error.
>>>>
>>>> python-virtinst (and thus virt-manager) don't support the rbd 
>>>> storage
>>>> pool. They don't know how to configure rbd disks yet either.
>>>>
>>>> Josh
>>
>>