[libvirt] how to enable ceph-rbd in libvirt.
yue
libvirt at 163.com
Sat Oct 13 08:20:31 UTC 2012
i try your proposal. but there are no any output in qemu-rbd.log. selinux has been disabled(then reboot my OS).
i do not know apparmor. my OS has no rpm of apparmor.
my os is centps-6.2. 2.6.32-220.el6.x86_64.
1.would you give me the options of libvirt configure. i rebuild it.
2.if there a other way to confirm whether libvirt support rbd.other way to find root cause.
thanks.
----rbd disk----
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writethrough' io='native'/>
<auth username='admin'>
<secret type='ceph' uuid='7a91dc24-b072-43c4-98fb-4b2415322b0f'/>
</auth>
<source protocol='rbd' name='cloud/testrbd:debug_ms=1:debug_rbd=20:debug_monc=20:debug_auth=20:log_file=/tmp/qemu-rbd.log'>
<host name='192.168.10.4' port='6789'/>
</source>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
--------------------
At 2012-10-12 15:06:45,"Josh Durgin" <josh.durgin at inktank.com> wrote:
>Sorry, I forgot libvirt/qemu eats stderr. The extra logging wasn't
>captured.
>
>Could you try:
>
>name='cloud/testrbd:debug_ms=1:debug_rbd=20:debug_monc=10:log_file=/tmp/qemu-rbd.log'
>
>If /tmp/qemu-rbd.log isn't full of debug messages, make sure selinux or
>apparmor aren't preventing
>qemu from writing to it.
>
>Josh
>
>On 2012-10-11 22:31, yue wrote:
>> hi, i add debug options related to rbd in xpSP3.xml. but error seems
>> same with privious.
>> ----libvirtd.log------------------
>> erfaces
>> 2012-10-12 13:25:28.267+0000: 9023: error : virNodeNumOfDevices:13984
>> : this function is not supported by the connection driver:
>> virNodeNumOfDevices
>> 2012-10-12 13:25:29.264+0000: 9025: error :
>> virConnectNumOfInterfaces:11014 : this function is not supported by
>> the connection driver: virConnectNumOfInterfaces
>> 2012-10-12 13:25:29.266+0000: 9023: error :
>> virConnectNumOfDefinedInterfaces:11098 : this function is not
>> supported by the connection driver: virConnectNumOfDefinedInterfaces
>> 2012-10-12 13:25:29.267+0000: 9022: error : virNodeNumOfDevices:13984
>> : this function is not supported by the connection driver:
>> virNodeNumOfDevices
>> 2012-10-12 13:25:30.264+0000: 9025: error :
>> virConnectNumOfInterfaces:11014 : this f unction is not supported by
>> the connection driver: virConnectNumOfInterfaces
>> 2012-10-12 13:25:30.267+0000: 9023: error :
>> virConnectNumOfDefinedInterfaces:11098 : this function is not
>> supported by the connection driver: virConnectNumOfDefinedInterfaces
>> 2012-10-12 13:25:30.269+0000: 9022: error : virNodeNumOfDevices:13984
>> : this function is not supported by the connection driver:
>> virNodeNumOfDevices
>> 2012-10-12 13:25:30.523+0000: 9021: warning : qemuDomainObjTaint:1364
>> : Domain id=1 name='xpSP3' uuid=b45bd66a-6700-4905-c5f4-4c799413d7b7
>> is tainted: high-privileges
>> 2012-10-12 13:25:30.943+0000: 9021: error : qemuMonitorOpenUnix:293 :
>> failed to connect to monitor socket: No such process
>> 2012-10-12 13:25:30.943+0000: 9021: info : qemuConnectMonitor:1247 :
>> Failed to connect monitor for xpSP3
>> 2012-10-12 13:25:30.943+0000: 9021: error :
>> qemuProcessWaitForMonitor:1634 : internal error process exited while
>> connecting to monitor: char device redirected to /dev/pts/4
>> qemu-kvm: -drive
>>
>> file=rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough,aio=native:
>> could not open disk image
>>
>> rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789:
>> No such file or directory
>>
>> ---------xpSP3.log---is attached.---
>> 2012-10-12 13:11:33.133+0000: starting up
>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root
>> LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name xpSP3 -S
>> -M rhel6.2.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1
>> -uuid b45bd66a-6700-4905-c5f4-4c799413d7b7 -nodefconfig -nodefaults
>> -chardev socket,id=charmonitor,path=/var/lib/libvirt/
>> qemu/xpSP3.monitor,server,nowait -mon
>> chardev=charmonitor,id=monitor,mode=readline -rtc
>> base=localtime,driftfix=slew -device
>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>>
>> file=/var/lib/libvirt/images/xpSP3.img,if=none,id=drive-ide0-0-0,format=raw,cache=none
>> -device
>>
>> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>> -drive
>>
>> file=rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough,aio=native
>> -device
>>
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0
>> -netdev tap,fd=19,id=hostnet0 -device
>>
>> e1000,netdev=hostnet0,id=net0,mac=52:54:00:2a:8c:48,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -device
>> usb-tablet,id=input0
>> -vnc 0.0.0.0:0 -vga std -device
>> intel-hda,id=sound0,bus=pci.0,addr=0x4
>> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> Domain id=1 is tainted: high-privileges
>> 2012-10-12 13:11:33.144+0000: 7815: debug : virFileClose:72 : Closed
>> fd 18
>> 2012-10-12 13:11:33.145+0000: 7815: debug : virFileClose:72 : Closed
>> fd 24
>> 2012-10-12 13:11:33.145+0000: 7815: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.146+0000: 7818: debug : virCommandHook:2058 : Run
>> hook 0x7faa10a62030 0x7faa160d4370
>> 2012-10-12 13:11:33.146+0000: 7818: debug : qemuProcessHook:2589 :
>> Obtaining domain lock
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virDomainLockProcessStart:164 : plugin=0x7faa080020a0
>> dom=0x7faa0804af40 paused=1 fd=0x7faa160d3998
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virDomainLockManagerNew:123 : plugin=0x7faa080020a0
>> dom=0x7faa0804af40
>> withResources=1
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virLockManagerPluginGetDriver:280 : plugin=0x7faa080020a0
>> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockManagerNew:304 :
>> driver=0x3712e9cb20 type=0 nparams=4 params=0x7faa160d3830 flags=0
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virLockManagerLogParams:104 : key=uuid type=uuid
>> value=b45bd66a-6700-4905-c5f4-4c799413d7b7
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virLockManagerLogParams:100 : key=name type=string value=xpSP3
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virLockManagerLogParams:88 : key=id type=uint value=1
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virLockManagerLogParams:88 : key=pid type=uint value=7818
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virDomainLockManagerNew:135 : Adding leases
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virDomainLockManagerNew:140 : Adding disks
>> 2012-10-12 13:11:33.146+0000: 7818: debug :
>> virDomainLockManagerAddDisk:86 : Add disk
>> /var/lib/libvirt/images/xpSP3.img
>> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockMan
>> agerAddResource:333 : lock=0x7faa08060820 type=0
>> name=/var/lib/libvirt/images/xpSP3.img nparams=0 params=(nil) flags=0
>> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockManagerAcquire:350
>> : lock=0x7faa08060820 state='(null)' flags=3 fd=0x7faa160d3998
>> 2012-10-12 13:11:33.146+0000: 7818: debug : virLockManagerFree:387 :
>> lock=0x7faa08060820
>> 2012-10-12 13:11:33.146+0000: 7818: debug : qemuProcessHook:2614 :
>> Moving process to cgroup
>> 2012-10-12 13:11:33.146+0000: 7818: debug : virCgroupNew:617 : New
>> group /libvirt/qemu/xpSP3
>> 2012-10-12 13:11:33.151+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/mapping 0:cpu at /cgroup/cpu in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/mapping 2:cpuset at /cgroup/cpuset in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/ma pping 3:memory at /cgroup/memory in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/mapping 4:devices at /cgroup/devices in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/mapping 5:freezer at /cgroup/freezer in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupDetect:273 :
>> Detected mount/mapping 6:blkio at /cgroup/blkio in
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:535 :
>> Make group /libvirt/qemu/xpSP3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/cpu/libvirt/qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/cpuacct/libvirt/qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/cpuset/libvirt/qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/memory/libvirt/ qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/devices/libvirt/qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/freezer/libvirt/qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupMakeGroup:560 :
>> Make controller /cgroup/blkio/libvirt/qemu/xpSP3/
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/cpu/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/cpuacct/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/cpuset/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/memory/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/devices/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/freezer/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virCgroupSetValueStr:331
>> : Set value '/cgroup/blkio/libvirt/qemu/xpSP3/tasks' to '7818'
>> 2012-10-12 13:11:33.156+0000: 7818: debug : virFileClose:72 : Closed
>> fd 3
>> 2012-10-12 13:11:33.156+0000: 7818: debug :
>> qemuProcessInitCpuAffinity:1848 : Setting CPU affinity
>> 2 012-10-12 13:11:33.156+0000: 7818: debug :
>> qemuProcessInitCpuAffinity:1883 : Set CPU affinity with specified
>> cpuset
>> 2012-10-12 13:11:33.156+0000: 7818: debug : qemuProcessHook:2626 :
>> Setting up security labelling
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virGetUserIDByName:2539 :
>> User record for user '0' does not exist
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virGetGroupIDByName:2621
>> : Group record for group '0' does not exist
>> 2012-10-12 13:11:33.157+0000: 7818: debug :
>> virSecurityDACSetProcessLabel:861 : Dropping privileges of DEF to 0:0
>> 2012-10-12 13:11:33.157+0000: 7818: debug : qemuProcessHook:2633 :
>> Hook complete ret=0
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2060 :
>> Done hook 0
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2074 :
>> Notifying parent for handshake start on 21
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2098 :
>> Waiting on parent for handshake complete on 22
>> 2012-10-12 13:11:33.157+0 000: 7818: debug : virFileClose:72 : Closed
>> fd 21
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virFileClose:72 : Closed
>> fd 22
>> 2012-10-12 13:11:33.157+0000: 7818: debug : virCommandHook:2119 :
>> Hook is done 0
>> char device redirected to /dev/pts/4
>> qemu-kvm: -drive
>>
>> file=rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writethrough,aio=native:
>> could not open disk image
>>
>> rbd:cloud/testrbd:debug_rbd=20:debug_monc=20:debug_auth=20:log_to_stderr=2:id=admin:key=AQCp1GFQMMOrCxAAEb4jxgUF9BuSBkQIigFi/Q==:auth_supported=cephx;none:mon_host=192.168.10.4:6789:
>> No such file or directory
>> 2012-10-12 13:11:33.557+0000: shutting down
>>
>> ----
>>
>> At 2012-10-12 00:04:19,"Josh Durgi
>> n" <josh.durgin at inktank.com> wrote:
>>>On 10/11/2012 12:00 AM, yue wrote:
>>>> how should i do? i configurate guest-os.xml ,add rbd disk manually.
>>>> then virsh start xpSP3
>>>
>>>That's the right way to do it.
>>>
>>>> -------rbd disk------------
>>>> <disk type='network' device='disk'>
>>>> <driver name='qemu' type='raw' cache='writethrough' io='native'/>
>>>> <auth username='cloud'>
>>>> <secret type='ceph' uuid='7a91dc24-b072-43c4-98fb-4b2415322b0f'/>
>>>> </auth>
>>>> <source protocol='rbd' name='cloud/testrbd'>
>>>> <host name='192.168.10.4' port='6789'/>
>>>> </source>
>>>> <target dev='vdb' bus='virtio'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>>> function='0x0'/>
>>>> </disk>
>>>> ---error log---
>>>>
>>>> [root at KVM-ip11 git]# virsh start xpSP3
>>>> error: Failed to start domain xpSP3
>>>> error: internal error process exited while connecting to monitor:
>>>> char device redirected to /dev/pts/4
>>>> qemu-kvm: -drive
>>>> file=rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk1,format=raw,cache=writethrough,aio=native:
>>>> could not open disk image
>>>> rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789:
>>>> No such file or directory
>>>> ---xpSP3.log-----
>>>> 2012-10-11 14:57:58.118+0000: starting up
>>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root
>>>> LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name xpSP3 -S
>>>> -M rhel6.2.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1
>>>> -uuid b45bd66a-6700-4905-c5f4-4c799413d7b7 -nodefconfig -nodefaults
>>>> -chardev
>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/xpSP3.monitor,server,nowait
>>>> -mon chardev=charmonitor,id=monitor,mode=readline -rtc
>>>> base=localtime,driftfix=slew -device
>>>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>>>> file=/var/lib/libvirt/images/xpSP3.img,if=none,id=drive-ide0-0-0,format=raw,cache=none
>>>> -device
>>>> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>>>> -drive
>>>> file=rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,
>> if=none,id=drive-virtio-disk1,format=raw -device
>>
>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
>> -netdev tap,fd=19,id=hostnet0 -device e1000,netdev!
>>>> =hostnet0,id=net0,mac=52:54:00:2a:8c:48,bus=pci.0,addr=0x3 -chardev
>>>> pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
>>>> -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga std -device
>>>> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>>>> Domain id=3 is tainted: high-privileges
>>>> char device redirected to /dev/pts/4
>>>> qemu-kvm: -drive
>>>> file=rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789,if=none,id=drive-virtio-disk1,format=raw:
>>>> could not open disk image
>>>> rbd:cloud/testrbd:id=cloud:key=AQCGbGRQ+M+NGBAATtylZNiSxqCTQ4uaApd+9w==:auth_supported=cephx;none:mon_host=192.168.10.4:6789:
>>>> No such file or directory
>>>> 2012-10-11 14:57:58.548+0000: shutting down
>>>
>>>It's unclear where the actual problem is - could you try again with
>>>name='cloud/testrbd:debug_ms=1:debug_rbd=20:debug_monc=10:log_to_stderr=true'
>>>and post the instance log (i.e. xpSP3.log)?
>>>
>>>Josh
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> At 2012-10-11 14:48:43,"Josh Durgin" <josh.durgin at inktank.com>
>>>> wrote:
>>>>> On 10/10/2012 10:52 PM, yue wrote:
>>>>>> hi, now i rebuild libvirtd, version 0.10.2. and define ceph
>>>>>> secret
>>>>>> ,define ceph storage-pool. all fine up to now.
>>>>>> i operate guest-os through virt-manager-GUI . i add a
>>>>>> rbd-file(qemu-img create before) from ceph-rbd-pool, click
>>>>>> 'finish',it
>>>>>> pop error "NoneType object has no attribute startswith", someone
>>>>>> say it
>>>>>> is python error.
>>>>>
>>>>> python-virtinst (and thus virt-manager) don't support the rbd
>>>>> storage
>>>>> pool. They don't know how to configure rbd disks yet either.
>>>>>
>>>>> Josh
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20121013/b03814e6/attachment-0001.htm>
More information about the libvir-list
mailing list