[libvirt-users] unprivileged container has wrong owners inside in /proc/ and /sys

[mxs kolo]

We run unprivileged lxc containers (libvirt based) with next config:
...
  <idmap>
    <uid start='0' target='65535' count='65535'/>
    <gid start='0' target='65535' count='65535'/>
  </idmap>
...
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/var/lib/libvirt/lxc/test1'/>
      <target dir='/'/>
    </filesystem>
...
  </devices>

Before start we need mount container's LVM:
  mount /dev/data/test1  /var/lib/libvirt/lxc/test1
And shift uid/gid:
  ./uidmapshift -b /var/lib/libvirt/lxc/test1 0 65535 65535
(uidmapshift.c is simple utility, found in LXD community)

As result, our FS permissions look from hardware node as:
[root at ops-node01.infra]# ls -la /var/lib/libvirt/lxc/test1/
total 8
dr-xr-xr-x 19 65535 65535  275 Apr  9 18:55 .
drwxrwxrwx  3 root  root    26 Apr  9 19:32 ..
-rw-r--r--  1 65535 65535    0 Mar 30 18:20 .autorelabel
lrwxrwxrwx  1 65535 65535    7 Mar  2  2017 bin -> usr/bin
dr-xr-xr-x  2 65535 65535    6 Nov  5  2016 boot
drwxr-xr-x  4 65535 65535  206 Mar 30 18:03 dev
drwxr-xr-x 59 65535 65535 4096 Apr  9 18:57 etc
drwxr-xr-x  2 65535 65535    6 Nov  5  2016 home
lrwxrwxrwx  1 65535 65535    7 Mar  2  2017 lib -> usr/lib
lrwxrwxrwx  1 65535 65535    9 Mar  2  2017 lib64 -> usr/lib64
drwxr-xr-x  2 65535 65535    6 Nov  5  2016 media
drwxr-xr-x  2 65535 65535    6 Nov  5  2016 mnt
drwxr-xr-x  2 65535 65535    6 Apr  9 18:55 .oldroot
drwxr-xr-x  3 65535 65535   24 Mar  2  2017 opt
dr-xr-xr-x  2 65535 65535    6 Nov  5  2016 proc
dr-xr-x---  4 65535 65535  172 Apr  9 18:57 root
drwxr-xr-x 13 65535 65535  178 Mar 30 18:03 run
lrwxrwxrwx  1 65535 65535    8 Mar  2  2017 sbin -> usr/sbin
drwxr-xr-x  2 65535 65535   21 Mar 30 18:03 selinux
drwxr-xr-x  2 65535 65535    6 Nov  5  2016 srv
dr-xr-xr-x  2 65535 65535    6 Nov  5  2016 sys
drwxrwxrwt  7 65535 65535   93 Apr  6 21:08 tmp
drwxr-xr-x 13 65535 65535  155 Mar  2  2017 usr
drwxr-xr-x 18 65535 65535  254 Mar 30 18:20 var

Container running, but some FS objects inside get owner 65534.65534:
[root at test1 ~]# find / -uid 65534 -ls | wc -l
40084

It's mostly /sys and /proc objects:
[root at test1 ~]# ls -la /
total 8
dr-xr-xr-x  19 root  root   275 Apr  9 18:55 .
dr-xr-xr-x  19 root  root   275 Apr  9 18:55 ..
-rw-r--r--   1 root  root     0 Mar 30 18:20 .autorelabel
drwxr-xr-x   2 root  root     6 Apr  9 18:55 .oldroot
lrwxrwxrwx   1 root  root     7 Mar  2  2017 bin -> usr/bin
dr-xr-xr-x   2 root  root     6 Nov  5  2016 boot
drwxr-xr-x   6 root  root   440 Apr  9 20:40 dev
drwxr-xr-x  59 root  root  4096 Apr  9 18:57 etc
drwxr-xr-x   2 root  root     6 Nov  5  2016 home
lrwxrwxrwx   1 root  root     7 Mar  2  2017 lib -> usr/lib
lrwxrwxrwx   1 root  root     9 Mar  2  2017 lib64 -> usr/lib64
drwxr-xr-x   2 root  root     6 Nov  5  2016 media
drwxr-xr-x   2 root  root     6 Nov  5  2016 mnt
drwxr-xr-x   3 root  root    24 Mar  2  2017 opt
dr-xr-xr-x 348 65534 65534    0 Apr  9 20:40 proc
dr-xr-x---   4 root  root   172 Apr  9 18:57 root
drwxrwxrwt  15 root  root   360 Apr  9 20:40 run
lrwxrwxrwx   1 root  root     8 Mar  2  2017 sbin -> usr/sbin
drwxr-xr-x   2 root  root    21 Mar 30 18:03 selinux
drwxr-xr-x   2 root  root     6 Nov  5  2016 srv
dr-xr-xr-x  13 65534 65534    0 Apr  9 20:40 sys
drwxrwxrwt   7 root  root    93 Apr  6 21:08 tmp
drwxr-xr-x  13 root  root   155 Mar  2  2017 usr
drwxr-xr-x  18 root  root   254 Mar 30 18:20 var

[root at test1 ~]# find /sys/  -uid 65534 -ls | wc -l
36871
[root at t1 ~]# find /proc  -uid 65534 -ls | wc -l
3200
[root at test1 ~]# find /dev  -uid 65534 -ls
2121875    0 drwxrwxrwt   2 65534    65534          40 Apr 10 13:57 /dev/mqueue

It's feature ? Look like libvirt make all this objects from real root
with owner root.root
You can't call virLXCControllerSetupUserns() before make this objects, right ?
Because you need root permissions for some operations.
After you set uid range for process 65535-131070, we can't show object
with uid.gid 0 and then kernel use values from
/proc/sys/kernel/overflowuid and /proc/sys/kernel/overflowgid.

It's can probably affect some process in container ?
As solution is shift uid/gid _after_ make all objects and _before_
enter userNS ?
As workaround we can try set 0 for overflowuid and overflowgid

p.s.
 We use type='mount' instead type='block' because in unprivileged
container mount() call failed,  more detailes in
https://bugzilla.redhat.com/show_bug.cgi?id=1328946

b.r.
 Maxim Kozin