[libvirt] Remote Xen and libvirt >= 0.5.0 Issues

[Hany Fahim]

Hi,
I'm currently running a few servers with CentOS 5.3 (i386) with the stock
Xen version:

# rpm -q kernel-xen xen libvirt
kernel-xen-2.6.18-92.1.22.el5
kernel-xen-2.6.18-128.1.6.el5
xen-3.0.3-80.el5_3.2
libvirt-0.6.3-1

libvirt was compiled from the src RPM from libvirt.org to run on i386
architecture without issue. I'm having a problem when trying to connect to
remote Xen installations using versions of libvirt starting from 0.5.0 and
above. When issuing virsh commands to query the local instance, it works
without issue, but any remote sources gives 'error: failed to connect to the
hypervisor'. Here's what LIBVIRT_DEBUG reveals:

# LIBVIRT_DEBUG=6 virsh -d 5 -c xen://node3/ list
command: "list "
13:53:17.132: debug : virInitialize:290 : register drivers
13:53:17.132: debug : virRegisterDriver:667 : registering Test as driver 0
13:53:17.132: debug : virRegisterNetworkDriver:567 : registering Test as
network driver 0
13:53:17.132: debug : virRegisterStorageDriver:598 : registering Test as
storage driver 0
13:53:17.132: debug : virRegisterDeviceMonitor:629 : registering Test as
device driver 0
13:53:17.132: debug : xenHypervisorInit:1922 : Using new hypervisor call:
30001

13:53:17.132: debug : xenHypervisorInit:1991 : Using hypervisor call v2, sys
ver3 dom ver5

13:53:17.132: debug : virRegisterDriver:667 : registering Xen as driver 1
13:53:17.132: debug : virRegisterDriver:667 : registering OPENVZ as driver 2
13:53:17.133: debug : vboxRegister:68 : VBoxCGlueInit failed: 0.0.0,
errorval=-1
13:53:17.133: debug : virRegisterDriver:667 : registering VBOX as driver 3
13:53:17.133: debug : virRegisterDriver:667 : registering remote as driver 4
13:53:17.133: debug : virRegisterNetworkDriver:567 : registering remote as
network driver 1
13:53:17.133: debug : virRegisterStorageDriver:598 : registering remote as
storage driver 1
13:53:17.133: debug : virRegisterDeviceMonitor:629 : registering remote as
device driver 1
13:53:17.133: debug : virConnectOpenAuth:1100 : name=xen://node3/,
auth=0x67d0080, flags=0
13:53:17.133: debug : do_open:920 : name "xen://node3/" to URI components:
  scheme xen
  opaque (null)
  authority (null)
  server node3
  user (null)
  port 0
  path /

13:53:17.133: debug : do_open:930 : trying driver 0 (Test) ...
13:53:17.133: debug : do_open:936 : driver 0 Test returned DECLINED
13:53:17.133: debug : do_open:930 : trying driver 1 (Xen) ...
13:53:17.133: debug : do_open:936 : driver 1 Xen returned DECLINED
13:53:17.133: debug : do_open:930 : trying driver 2 (OPENVZ) ...
13:53:17.133: debug : do_open:936 : driver 2 OPENVZ returned DECLINED
13:53:17.133: debug : do_open:930 : trying driver 3 (VBOX) ...
13:53:17.133: debug : do_open:936 : driver 3 VBOX returned DECLINED
13:53:17.133: debug : do_open:930 : trying driver 4 (remote) ...
13:53:17.133: debug : do_open:936 : driver 4 remote returned DECLINED
13:53:17.133: debug : virUnrefConnect:210 : unref connection 0x93275d0 1
13:53:17.134: debug : virReleaseConnect:171 : release connection 0x93275d0
error: failed to connect to the hypervisor

But when connecting to localhost, it works just fine:

# LIBVIRT_DEBUG=6 virsh -d 5 -c xen:/// list
command: "list "
13:53:23.558: debug : virInitialize:290 : register drivers
13:53:23.558: debug : virRegisterDriver:667 : registering Test as driver 0
13:53:23.558: debug : virRegisterNetworkDriver:567 : registering Test as
network driver 0
13:53:23.558: debug : virRegisterStorageDriver:598 : registering Test as
storage driver 0
13:53:23.558: debug : virRegisterDeviceMonitor:629 : registering Test as
device driver 0
13:53:23.559: debug : xenHypervisorInit:1922 : Using new hypervisor call:
30001

13:53:23.559: debug : xenHypervisorInit:1991 : Using hypervisor call v2, sys
ver3 dom ver5

13:53:23.559: debug : virRegisterDriver:667 : registering Xen as driver 1
13:53:23.559: debug : virRegisterDriver:667 : registering OPENVZ as driver 2
13:53:23.559: debug : vboxRegister:68 : VBoxCGlueInit failed: 0.0.0,
errorval=-1
13:53:23.559: debug : virRegisterDriver:667 : registering VBOX as driver 3
13:53:23.559: debug : virRegisterDriver:667 : registering remote as driver 4
13:53:23.559: debug : virRegisterNetworkDriver:567 : registering remote as
network driver 1
13:53:23.560: debug : virRegisterStorageDriver:598 : registering remote as
storage driver 1
13:53:23.560: debug : virRegisterDeviceMonitor:629 : registering remote as
device driver 1
13:53:23.560: debug : virConnectOpenAuth:1100 : name=xen:///,
auth=0x67d0080, flags=0
13:53:23.560: debug : do_open:920 : name "xen:///" to URI components:
  scheme xen
  opaque (null)
  authority (null)
  server (null)
  user (null)
  port 0
  path /

13:53:23.560: debug : do_open:930 : trying driver 0 (Test) ...
13:53:23.560: debug : do_open:936 : driver 0 Test returned DECLINED
13:53:23.560: debug : do_open:930 : trying driver 1 (Xen) ...
13:53:23.560: debug : xenUnifiedOpen:295 : Trying hypervisor sub-driver
13:53:23.560: debug : xenUnifiedOpen:298 : Activated hypervisor sub-driver
13:53:23.560: debug : xenUnifiedOpen:306 : Trying XenD sub-driver
13:53:23.563: debug : xenUnifiedOpen:309 : Activated XenD sub-driver
13:53:23.563: debug : xenUnifiedOpen:315 : Trying XM sub-driver
13:53:23.563: debug : xenUnifiedOpen:318 : Activated XM sub-driver
13:53:23.563: debug : xenUnifiedOpen:322 : Trying XS sub-driver
13:53:23.564: debug : xenStoreOpen:346 : Failed to add event handle,
disabling events

13:53:23.564: debug : xenUnifiedOpen:325 : Activated XS sub-driver
13:53:23.570: debug : xenUnifiedOpen:361 : Trying Xen inotify sub-driver
13:53:23.571: debug : xenInotifyOpen:439 : Adding a watch on /etc/xen
13:53:23.571: debug : xenInotifyOpen:451 : Building initial config cache
13:53:23.571: debug : xenXMConfigCacheAddFile:387 : Adding file
/etc/xen/centos
13:53:23.571: debug : xenXMConfigCacheAddFile:436 : Failed to read
/etc/xen/centos
13:53:23.571: debug : xenXMConfigCacheAddFile:387 : Adding file
/etc/xen/test3
13:53:23.571: debug : xenXMConfigCacheAddFile:465 : Added config test3
/etc/xen/test3
13:53:23.571: debug : xenXMConfigCacheAddFile:387 : Adding file
/etc/xen/hany
13:53:23.571: debug : xenXMConfigCacheAddFile:465 : Added config hany
/etc/xen/hany
13:53:23.572: debug : xenXMConfigCacheAddFile:387 : Adding file
/etc/xen/hany2
13:53:23.572: debug : xenXMConfigCacheAddFile:465 : Added config hany4
/etc/xen/hany2
13:53:23.572: debug : xenInotifyOpen:458 : Registering with event loop
13:53:23.572: debug : xenInotifyOpen:462 : Failed to add inotify handle,
disabling events
13:53:23.572: debug : virConnectRef:1165 : conn=0x996b5c8 refs=3
13:53:23.572: debug : xenUnifiedOpen:364 : Activated Xen inotify sub-driver
13:53:23.572: debug : do_open:936 : driver 1 Xen returned SUCCESS
13:53:23.572: debug : do_open:956 : network driver 0 Test returned DECLINED
13:53:23.572: debug : doRemoteOpen:511 : proceeding with name = xen:///
13:53:23.572: debug : call:6543 : Doing call 66 (nil)
13:53:23.573: debug : call:6613 : We have the buck 66 0xb736c008 0xb736c008
13:53:23.573: debug : processCallRecvLen:6201 : Got length, now need 36
total (32 more)
13:53:23.573: debug : processCalls:6469 : Giving up the buck 66 0xb736c008
(nil)
13:53:23.573: debug : call:6644 : All done with our call 66 (nil) 0xb736c008
13:53:23.573: debug : call:6543 : Doing call 1 (nil)
13:53:23.573: debug : call:6613 : We have the buck 1 0x997f3b0 0x997f3b0
13:53:23.584: debug : processCallRecvLen:6201 : Got length, now need 28
total (24 more)
13:53:23.585: debug : processCalls:6469 : Giving up the buck 1 0x997f3b0
(nil)
13:53:23.585: debug : call:6644 : All done with our call 1 (nil) 0x997f3b0
13:53:23.585: debug : doRemoteOpen:822 : Adding Handler for remote events
13:53:23.585: debug : doRemoteOpen:829 : virEventAddHandle failed: No
addHandleImpl defined. continuing without events.
13:53:23.585: debug : do_open:956 : network driver 1 remote returned SUCCESS
13:53:23.585: debug : do_open:978 : storage driver 0 Test returned DECLINED
13:53:23.585: debug : do_open:978 : storage driver 1 remote returned SUCCESS
13:53:23.585: debug : do_open:999 : node driver 0 Test returned DECLINED
13:53:23.585: debug : do_open:999 : node driver 1 remote returned SUCCESS
13:53:23.585: debug : virConnectNumOfDomains:1440 : conn=0x996b5c8
13:53:23.585: debug : virConnectListDomains:1401 : conn=0x996b5c8,
ids=0x997a6d0, maxids=1
 Id Name                 State
----------------------------------
13:53:23.586: debug : virDomainLookupByID:1576 : conn=0x996b5c8, id=0
13:53:23.586: debug : virGetDomain:271 : New hash entry 0x997b300
13:53:23.586: debug : virDomainGetInfo:2569 : domain=0x997b300,
info=0xbfc34c64
13:53:23.586: debug : virDomainGetName:2268 : domain=0x997b300
13:53:23.586: debug : virDomainGetID:2361 : domain=0x997b300
  0 Domain-0             running
13:53:23.586: debug : virDomainFree:1806 : domain=0x997b300
13:53:23.586: debug : virUnrefDomain:345 : unref domain 0x997b300 Domain-0 1
13:53:23.586: debug : virReleaseDomain:303 : release domain 0x997b300
Domain-0
13:53:23.587: debug : virReleaseDomain:315 : unref connection 0x996b5c8 5

13:53:23.587: debug : virConnectClose:1118 : conn=0x996b5c8
13:53:23.587: debug : call:6543 : Doing call 2 (nil)
13:53:23.587: debug : call:6613 : We have the buck 2 0x997f3b0 0x997f3b0
13:53:23.588: debug : processCallRecvLen:6201 : Got length, now need 28
total (24 more)
13:53:23.588: debug : processCalls:6469 : Giving up the buck 2 0x997f3b0
(nil)
13:53:23.588: debug : call:6644 : All done with our call 2 (nil) 0x997f3b0
13:53:23.588: debug : virUnrefConnect:210 : unref connection 0x996b5c8 4
13:53:23.588: debug : virUnrefConnect:210 : unref connection 0x996b5c8 3
13:53:23.589: debug : virUnrefConnect:210 : unref connection 0x996b5c8 2
13:53:23.589: debug : virUnrefConnect:210 : unref connection 0x996b5c8 1
13:53:23.589: debug : virReleaseConnect:171 : release connection 0x996b5c8

Anybody else run into this? Any help would be greatly appreciated.

Thanks,

Hany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20090607/c71d7c73/attachment-0001.htm>