[libvirt] Libvirt/LXC (lxcControllerMain:380 : EPOLLHUP from fd 7)

Tony Risinger sweetsinsemilla at gmail.com
Mon Dec 7 09:02:53 UTC 2009 

hello,

i am trying to use libvirt with LXC, and having some issues...
everything seems ok when using lxc-* tools.  host box is Arch Linux.
ultimately i'm getting this error in the domain log:

.....
05:45:31.353: debug : lxcControllerMain:380 : EPOLLHUP from fd 7

i was getting this last night in #virt:

.....
20:49:57.545: error : lxcFdForward:230 : read of fd 7 failed: Input/output error
libvir: Linux Container error : read of fd 7 failed: Input/output error

instead of the former, but i can seem to reproduce now. i might have
replaced 0.7.4 version with git version... i dont remember.  right now
the build is direct from master, and i am getting EPOLLHUP.  didnt see
anything interesting in virsh or libvirtd debug output, but i can
paste/bin the whole thing or whatever else if need be.

-----VERSIONS/SOURCES

(loadmodconfig make target + below)
kernel: Linux PHS-001 2.6.32-custom #1 SMP PREEMPT Sun Dec 6 05:29:37
CST 2009 i686 Intel(R) Xeon(TM) CPU 3.20GHz GenuineIntel GNU/Linux
  CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
  CONFIG_GROUP_SCHED=y
  CONFIG_FAIR_GROUP_SCHED=y
  CONFIG_RT_GROUP_SCHED=y
  CONFIG_CGROUP_SCHED=y
  CONFIG_CGROUPS=y
  CONFIG_CGROUP_NS=y
  CONFIG_CGROUP_FREEZER=y
  CONFIG_CGROUP_DEVICE=y
  CONFIG_CPUSETS=y
  CONFIG_PROC_PID_CPUSET=y
  CONFIG_CGROUP_CPUACCT=y
  CONFIG_RESOURCE_COUNTERS=y
  CONFIG_CGROUP_MEM_RES_CTLR=y
  CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
  CONFIG_MM_OWNER=y
  CONFIG_NAMESPACES=y
  CONFIG_UTS_NS=y
  CONFIG_IPC_NS=y
  CONFIG_USER_NS=y
  CONFIG_PID_NS=y
  CONFIG_NET_NS=y
  CONFIG_NET_CLS_CGROUP=y
  CONFIG_SECURITY_FILE_CAPABILITIES=y

(built from master)
libvirt: http://aur.archlinux.org/packages/libvirt-git/libvirt-git/PKGBUILD
  --with-capng
  --with-network
  --with-lxc
  --without-xen
  --without-xen-inotify
  --without-qemu
  --without-uml
  --without-openvz
  --without-vbox
  --without-one
  --without-esx

libcap-ng: http://aur.archlinux.org/packages/libcap-ng/libcap-ng/PKGBUILD

container: http://www.stgraber.org/2009/11/06/lxc-containers-or-extremely-fast-virtualization
  http://www.stgraber.org/download/lxc-ubuntu-8.04-i386.tar.gz

-----CONFIGURATION

# host ptmx symlinked to pts/ptmx
[root at PHS-001 vps]# ls -l /dev/ptmx
lrwxrwxrwx 1 root root 8 2009-12-08 04:40 /dev/ptmx -> pts/ptmx

# guest ptmx symlinked to pts/ptmx
[root at PHS-001 vps]# ls -l /vps/dom/ubuntu/root/dev/ptmx
lrwxrwxrwx 1 root root 8 2009-12-06 04:11
/vps/dom/ubuntu/root/dev/ptmx -> pts/ptmx

# host devpts mounted newinstance
[root at PHS-001 vps]# mount | grep devpts
none on /dev/pts type devpts (rw,newinstance)

# single cgroup mount on host
[root at PHS-001 vps]# mount | grep cgroup
none on /cgroup type cgroup (rw)

# libvirt config
[root at PHS-001 vps]# grep -v -e '^#\|^ *$' /etc/libvirt/libvirtd.conf
auth_unix_ro = "none"
auth_unix_rw = "none"

# guest config...
# i have tried the legacy <console type='pty' /> with same results
# i dont fully understand this, why do i need to specify a host/source pty?
[root at PHS-001 vps]# cat /vps/def/exec/sys/ubuntu.xml
<domain type='lxc'>
  <name>ubuntu</name>
  <memory>500000</memory>
  <os>
    <type arch='i686'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <filesystem type='mount'>
      <source dir='/vps/dom/ubuntu/root'/>
      <target dir='/'/>
    </filesystem>
    <console type='pty'>
      <source path='/dev/pts/4'/>
      <target port='0'/>
    </console>
  </devices>
</domain>

-----PROCEDURE

[root at PHS-001 vps]# LIBVIRT_DEBUG=1 libvirtd --daemon
05:42:46.121: debug : virInitialize:278 : register drivers
.....

[root at PHS-001 vps]# virsh -c lxc:/// define /vps/def/exec/sys/ubuntu.xml
Domain ubuntu defined from /vps/def/exec/sys/ubuntu.xml

[root at PHS-001 vps]# virsh -c lxc:/// define /vps/def/exec/sys/ubuntu.xml
Domain ubuntu defined from /vps/def/exec/sys/ubuntu.xml

[root at PHS-001 vps]# virsh -c lxc:/// start ubuntu
Domain ubuntu started

[root at PHS-001 vps]# virsh -c lxc:/// console ubuntu
error: Unable to get domain status
error: internal error Unable to get cgroup for ubuntu

-----LOGS

# /var/log/libvirt/lxc/ubuntu.log
05:45:31.105: debug : lxcControllerRun:540 : Setting up private /dev/pts
05:45:31.125: debug : lxcControllerRun:566 : Mouting 'devpts' on
/vps/dom/ubuntu/root/dev/pts
05:45:31.125: debug : lxcControllerRun:581 : Opening tty on private
/vps/dom/ubuntu/root/dev/pts/ptmx
05:45:31.125: debug : virCgroupNew:492 : New group /
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
0:cpu at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
1:cpuacct at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
2:cpuset at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
3:memory at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
4:devices at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
5:freezer at /cgroup in /1534
05:45:31.125: debug : virCgroupNew:492 : New group /libvirt
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
0:cpu at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
1:cpuacct at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
2:cpuset at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
3:memory at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
4:devices at /cgroup in /1534
05:45:31.125: debug : virCgroupDetect:230 : Detected mount/mapping
5:freezer at /cgroup in /1534
05:45:31.125: debug : virCgroupMakeGroup:450 : Make group /libvirt
05:45:31.125: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/
05:45:31.125: debug : virCgroupCpuSetInherit:416 : Setting up
inheritance / -> /libvirt
05:45:31.126: debug : virCgroupGetValueStr:305 : Get value
/cgroup/1534/cpuset.cpus
05:45:31.126: debug : virCgroupCpuSetInherit:429 : Inherit cpuset.cpus = 0-3
05:45:31.126: debug : virCgroupSetValueStr:275 : Set value
/cgroup/1534/libvirt/cpuset.cpus
05:45:31.126: debug : virCgroupGetValueStr:305 : Get value
/cgroup/1534/cpuset.mems
05:45:31.126: debug : virCgroupCpuSetInherit:429 : Inherit cpuset.mems = 0
05:45:31.126: debug : virCgroupSetValueStr:275 : Set value
/cgroup/1534/libvirt/cpuset.mems
05:45:31.126: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/
05:45:31.126: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/
05:45:31.126: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/
05:45:31.126: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/
05:45:31.126: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/
05:45:31.126: debug : virCgroupNew:492 : New group /libvirt/lxc
05:45:31.126: debug : virCgroupDetect:230 : Detected mount/mapping
0:cpu at /cgroup in /1534
05:45:31.126: debug : virCgroupDetect:230 : Detected mount/mapping
1:cpuacct at /cgroup in /1534
05:45:31.126: debug : virCgroupDetect:230 : Detected mount/mapping
2:cpuset at /cgroup in /1534
05:45:31.126: debug : virCgroupDetect:230 : Detected mount/mapping
3:memory at /cgroup in /1534
05:45:31.126: debug : virCgroupDetect:230 : Detected mount/mapping
4:devices at /cgroup in /1534
05:45:31.126: debug : virCgroupDetect:230 : Detected mount/mapping
5:freezer at /cgroup in /1534
05:45:31.126: debug : virCgroupMakeGroup:450 : Make group /libvirt/lxc
05:45:31.126: debug : virCgroupMakeGroup:462 : Make controller
/cgroup/1534/libvirt/lxc/
05:45:31.211: debug : lxcContainerStart:832 : clone() returned, 1536
05:45:31.353: debug : lxcControllerMain:380 : EPOLLHUP from fd 7

-----END

i've tried endless variations of the <devices> section, but i just
cant seem to get anything going with libvirt/LXC once i move the root
to a new location for a system container.  same container works fine
with lxc-* tools (although i manually make a bridge).  any help is
appreciated,

C Anthony Risinger

More information about the libvir-list mailing list