xen failure - xenstrore with 2.6.15-1.2009.4.2_FC5hypervisor #1 SMP

[Naoki]

I can't get xend to start, have not been able to for a couple of weeks,
if not more actually. Using 2.6.15-1.2009.4.2_FC5hypervisor #1 SMP

Log says this :

[2006-03-16 16:15:47 xend] INFO (SrvDaemon:285) Xend Daemon started
[2006-03-16 16:15:47 xend] INFO (SrvDaemon:289) Xend changeset:
unavailable .
[2006-03-16 16:15:47 xend] ERROR (SrvDaemon:299) Exception starting xend
((111, 'Connection refused'))
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py",
line 293, in run
    servers = SrvServer.create()
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvServer.py",
line 106, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvRoot.py",
line 40, in __init__
    self.get(name)
  File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 82, in
get
    val = val.getobj()
  File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 52, in
getobj
    self.obj = klassobj()
  File
"/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line
39, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line
550, in instance
    inst.init()
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line
70, in init
    xstransact.Mkdir(VMROOT)
  File
"/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line
317, in Mkdir
    complete(path, lambda t: t.mkdir(*args))
  File
"/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line
323, in complete
    t = xstransact(path)
  File
"/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", line
20, in __init__
    self.transaction = xshandle().transaction_start()
  File "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xsutil.py",
line 18, in xshandle
    xs_handle = xen.lowlevel.xs.xs()
RuntimeError: (111, 'Connection refused')

I traced the binary and I see this :
[pid  2954] socket(PF_FILE, SOCK_STREAM, 0) = 4
[pid  2954] connect(4, {sa_family=AF_FILE,
path="/var/run/xenstored/socket"}, 110) = -1 ECONNREFUSED (Connection
refused)
[pid  2954] close(4)  
[pid  2943] socket(PF_FILE, SOCK_STREAM, 0 <unfinished ...>
[pid  2954] write(2, "Failed to contact xenstore (Conn"..., 65
<unfinished ...>

The socket exists but nothing has a handle on it though :

$ ls -l /var/run/xenstored/socket
srw------- 1 root root 0 Mar 16 16:18 /var/run/xenstored/socket

It suggests xenstore is my problem but in the same trace I see it
executing ok :

[pid  2946] execve("/usr/sbin/xenstored", ["xenstored",
"--pid-file=/var/run/xenstore.pid"...], [/* 20 vars */]) = 0

So I ran it manually and I get this :

# FATAL: Failed to initialize dom0 state: No such device
full talloc report on 'null_context' (total 56 bytes in 3 blocks)
    struct domain                  contains     56 bytes in   2 blocks
(ref 0)
        /local/domain/0                contains     16 bytes in   1
blocks (ref 0)

Another trace shows the failure probably being here :

open("/proc/xen/xsd_kva", O_RDWR)       = 10
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 10, 0) = -1 ENODEV
(No such device)
close(10)    

# ls -l /proc/xen/xsd_kva
-r-------- 1 root root 0 Mar 16 16:31 /proc/xen/xsd_kva


That's about as far as my debugging skills can go, anybody seen/know
this?