[Virtio-fs] virtiofs mounted filesystems & SELinux

[Harry G. Coin]

Hi Vivek,

I've cut the o.p.'s text as your questions and my answers are off-topic
to his interest.

On 6/4/21 9:11 AM, Vivek Goyal wrote:

> You are using virtiofs as rootfs and upgrade breaks? If virtiofs is
> being used only as volumes, then its outside the upgrade path. Will
> like to know what broke for you. Was it all SELinux related or something
> else.

Seems 100% selinux related -- even in permissive mode on the guest. 
Here's your short-and-sweet 100% reproducer:  
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1930756  There's a
similar bug report on redhat.

Even with selinux set to permissive in the guest, no selinux installed
in the host, but with xattrs on, no and no xattr mapping, lsetfilecon
returns with a permission error.  The files on the host are entirely
dedicated to the guest -- no use of those files is made on the host. 
Underlying fs on the host is btrfs.
>> You'll have to learn a fair few quite obscure selinux
>> commands to get it to comprehend virtiofs is a filesystem and not flood
>> your logs, use the 'xml' features of virtual machine manager to get the
>> bits of virtiofs you want enabled properly (like 'numa',
> I am assumig libvirt takes care of specifying "numa" stuff? Is that
> not the case?

Not outside the lab.  There is no virt-manager GUI way to coordinate
memory allocation with hugepages/numa.  Hand-editing the xml is required
to manually match the numa set-aside with the memory allocation.  If
reliability is important, unless you use systctl.d to set aside the
hugepages the system can randomly not launch the guests owing to lack of
sufficient contiguous memory at guest launch time.

>> learn what
>> 'hugepages' are and do the math on each host to set enough of those
>> aside with in sysctl.d (e.g. vm.nr_hugepages = 11776) ,
> What's the virtiofs dependency on vm.nr_hugepages? I think I am not
> aware of this aspect.

Detailed here https://libvirt.org/kbase/virtiofs.html


>
>> and also accept
>> some versions of hosts and guests will develop incompatibilities so
>> severe you'll have to restore to backups and downgrades from time to
>> time.  And you won't be able to use virtiofs as a root filesystem
>> without patching dracut config directories (search this list) or
>> compiling your own custom kernel.  But you will learn a whole lot.
> Agreed that dracut does not seem to have support to mount virtiofs
> as rootfs. That's one TODO item. 

I've done it. It's been working for months, automatically maintained and
updated through both host and guest kernel upgrades.  I haven't posted
the bit that updates the kernel / initrd links for direct kernel launch
(using initrds) but I have posted what's needed for dracut to make
rootfs virtiofs work.

All that's required for stability -- for now -- is for the host to treat
the files shared on the guest as 'unusable' while the guest is running
-- or at least 'read only'.

> I am assuming you prefer using virtiofs as rootfs for your VMs and
> that's why running into these dracut related issues.

Let me restate -- I am NOT having a dracut issue.  I solved that long
ago and it works very well.   In fact the whole virtiofs thing was
working very well (without dax as main-distro kernels still lack the pci
bar in the host) -- until a regression in the last few weeks that broke
rpm / dnf / yum (noted above, even with permissive on the guest
lsetfilecon returns a permission error blocking all dnf operations). 
The only 'maintainable' way forward I can see is to reduce the number of
core packages involved.  The disk space required for a basic workstation
or server image is 'small' by today's storage standards.  I see no
reason multiply complexities by implementing one host-guest relationship
for 'booting' a guest and another for 'the files on the guest'.  Making
it possible to just snapshot one directory tree on the host and know
everything important to the guest is there -- that's a big win.

Harry