[libvirt] the <filesystem> tag (WAS: simple LXC/libvirt busybox container (Unable to get cgroup))

Thu Dec 10 23:44:30 UTC 2009

wow!  i was starting to think i would never get past this problem!

[root at PHS-001 ~]# echo $VIRSH_DEFAULT_CONNECT_URI
lxc:///

[root at PHS-001 ~]# virsh create /vps/def/exec/sys/arch-nano.xml
Domain arch-nano created from /vps/def/exec/sys/arch-nano.xml

[root at PHS-001 ~]# virsh console arch-nano
Connected to domain arch-nano
Escape character is ^]
#

[root at PHS-001 ~]# grep cgroup /proc/mounts
none /cgroup cgroup rw,relatime,devices,memory,cpuacct,cpu 0 0

everything seems to be working fine now after disabling "ns" from
cgroup mount; thank you very much Ryota and Daniel for you time and
assistance.  i must have sunk 40+ hours into this, but i learned a
tremendous amount, so nothing is lost. :-)

this is somewhat of a digression, and i can submit a new thread if
need be, but my last remaining question is about the <filesystem>
tag/root mounting.  i have not found any mention of the <filesystem>
tag in the docs or how to work with it... i want to have libvirt mount
a specified device (a btrfs subvolume) and use it for the root
filesystem of the created LXC container.  is there a way to do this?
perhaps <source dev="/dev/sdb" options="subvol....."/>, or do i need
to use the storage XML?  i didnt see a way to use a storage/device
definition as the rootfs.

im eager to get working with python bindings, just trying to see it
all work with XML first.

thanks++

On Thu, Dec 10, 2009 at 6:45 AM, Ryota Ozaki <ozaki.ryota at gmail.com> wrote:
> On Thu, Dec 10, 2009 at 9:36 PM, Daniel P. Berrange <berrange at redhat.com> wrote:
>> On Thu, Dec 10, 2009 at 09:26:39PM +0900, Ryota Ozaki wrote:
>>> On Thu, Dec 10, 2009 at 9:03 PM, Daniel P. Berrange <berrange at redhat.com> wrote:
>>> > On Thu, Dec 10, 2009 at 02:22:37AM -0600, Tony Risinger wrote:
>>> >> i'm trying to get even the simplest busybox container with libvirt+LXC
>>> >> with very limited success.  I feel l am missing something supremely
>>> >> simple for me to be hung on this for weeks.
>>> >>
>>> >> i dont see anything interesting in domain log, but getting this error
>>> >> from "LIBVIRT_DEBUG=1 libvirtd":
>>> >>
>>> >> 05:27:56.113: error : lxcDomainGetInfo:462 : internal error Unable to
>>> >> get cgroup for arch-nano
>>> >> 05:27:56.113: debug : virDomainFree:2004 : domain=0x81d8e68
>>> >> 05:27:56.113: debug : virUnrefDomain:422 : unref domain 0x81d8e68 arch-nano 1
>>> >> 05:27:56.113: debug : virReleaseDomain:376 : release domain 0x81d8e68 arch-nano
>>> >> 05:27:56.113: debug : virReleaseDomain:392 : unref connection 0x81dc0f0 2
>>> >> 05:27:56.113: debug : remoteSerializeError:141 : prog=536903814 ver=1
>>> >> proc=16 type=1 serial=4, msg=internal error Unable to get cgroup for
>>> >> arch-nano
>>> >>
>>> >> i've been using this root filesystem layout:
>>> >>
>>> >> [root at PHS-001 arch-nano]# tree
>>> >> .
>>> >> |-- bin
>>> >> |   |-- cat -> ../sbin/busybox
>>> >> |   |-- chdir -> ../sbin/busybox
>>> >> |   |-- chmod -> ../sbin/busybox
>>> >> |   |-- ls -> ../sbin/busybox
>>> >> |   |-- rm -> ../sbin/busybox
>>> >> |   |-- sh -> ../sbin/busybox
>>> >> |   `-- vi -> ../sbin/busybox
>>> >> |-- dev
>>> >> |   `-- pts
>>> >> |-- etc
>>> >> |-- proc
>>> >> |-- sbin
>>> >> |   |-- busybox
>>> >> |   `-- init -> busybox
>>> >> `-- sys
>>> >>
>>> >> all folders besides /bin and /sbin were created by libvirt.  i tried
>>> >> using the /sbin/init script previously suggested:
>>> >>
>>> >> #!/sbin/busybox
>>> >> sh
>>> >
>>> >
>>> > Sorry, my suggestion was wrong. I forgot that if you have #!/sbin/busybox
>>> > it will attempt to execute the command matching the name of the script.
>>> > So it will in fact try to run 'init', rather than 'sh'.
>>> >
>>> > Just make the libvirt XML point directly to /bin/sh  instead and it
>>> > should work. I even tested it this time :-)
>>>
>>> Hem, I still have a problem with ns subsystem enabled. Yes, I can launch
>>> a container however the cgroup hierarchy is wrong from libvirtd expecting
>>> like:
>>>
>>> /: libvirtd --daemon
>>> /5345: /usr/libexec/libvirt_lxc --name
>>>
>>> Daniel, could you confirm how about your cgroup hierarchy?
>>
>> What you do mean by 'ns' subsystem ?
>
> 'ns' is one of functions of cgroups like such as devices, memory,
> cpu, etc. and it is enabled if you mount cgroup without any options
> that Tony is doing.
>
>>
>>
>> # grep cgroup /proc/mounts
>> cgroup /dev/cgroups/cpu cgroup rw,relatime,cpuacct,cpu 0 0
>> cgroup /dev/cgroups/memory cgroup rw,relatime,memory 0 0
>> cgroup /dev/cgroups/devices cgroup rw,relatime,devices 0 0
>
> Oh, you don't enable 'ns', so yes, things go fine in your environment.
>
>
>>
>> # cat /proc/`pgrep libvirtd`/cgroup
>> 32:devices:/sysdefault
>> 16:memory:/sysdefault
>> 12:cpuacct,cpu:/sysdefault
>>
>> # cat /proc/`pgrep libvirt_lxc`/cgroup
>> 32:devices:/sysdefault/libvirt/lxc/vm1
>> 16:memory:/sysdefault/libvirt/lxc/vm1
>> 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1
>>
>>
>> And the process inside the contanier is PID 12309
>>
>> # cat /proc/12309/cgroup
>> 32:devices:/sysdefault/libvirt/lxc/vm1
>> 16:memory:/sysdefault/libvirt/lxc/vm1
>> 12:cpuacct,cpu:/sysdefault/libvirt/lxc/vm1
>>
>>
>> Which all appears to be correct to me
>>
>> This is on a Fedora 12 host 2.6.31.6-145.fc12.i686.PAE with
>>
>> CONFIG_UTS_NS=y
>> CONFIG_IPC_NS=y
>> CONFIG_USER_NS=y
>> CONFIG_PID_NS=y
>> CONFIG_NET_NS=y
>> CONFIG_CGROUP_SCHED=y
>> CONFIG_CGROUPS=y
>> # CONFIG_CGROUP_DEBUG is not set
>> CONFIG_CGROUP_NS=y
>
> This is the function I'm mentioning.
>
>  ozaki-r
>
>> CONFIG_CGROUP_FREEZER=y
>> CONFIG_CGROUP_DEVICE=y
>> CONFIG_CGROUP_CPUACCT=y
>> CONFIG_CGROUP_MEM_RES_CTLR=y
>> CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
>> CONFIG_NET_CLS_CGROUP=y
>>
>>
>> Regards,
>> Daniel
>> --
>> |: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
>> |: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
>> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
>> |: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
>>
>