[Libguestfs] virt-builder fails in fedora 34 container:

[Nir Soffer]

On Mon, Jun 14, 2021 at 11:57 AM Martin Kletzander <mkletzan at redhat.com> wrote:
>
> On Sun, Jun 13, 2021 at 11:15:31PM +0300, Nir Soffer wrote:
> >On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones at redhat.com> wrote:
> >>
> >> On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:
> >> > I'm running this inside a fedora 34 container in ovirt-imageio CI:
> >> >
> >> >     'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write',
> >> > '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"',
> >> > '--root-password', 'password:', '-o',
> >> > '/var/tmp/imageio-images/cirros-0.3.5'
> >> >
> >> > The command fails in:
> >> >
> >> > supermin: ext2: copying kernel modules
> >> > supermin: ext2: creating minimal initrd
> >> > '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd'
> >> > supermin: exception:
> >> > Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such
> >> > file or directory")
> >>
> >> We use this file in order to construct the minimal initrd with some
> >> virtio modules, which is sufficient to bring up the supermin root
> >> filesystem.  The file itself is used to calculate the correct set of
> >> module dependencies (so that, for example, if a virtio kmod needs
> >> another kmod then the other kmod will be pulled in).  See:
> >>
> >> https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13b1595b4f/src/format_ext2_initrd.ml
> >>
> >> I believe in Fedora the file is created via a %post script in
> >> kernel-core which runs /bin/kernel-install (from systemd-udev).  This
> >> in turn is a shell script which runs snippets from
> >> /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied
> >> from all kinds of other packages.  On my machine one of these is
> >> /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev)
> >> which eventually runs depmod -a which creates this file.
> >>
> >> So those are some of the packages which might be missing.  Does
> >> modules.dep get created anywhere inside the container?  It's possible
> >> it moved.
> >>
> >> It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1
> >> which should show (in the dnf install output I guess?) if/when the
> >> depmod command runs.
> >>
> >> > I think I'm missing some package removed from the fedora 34 container,
> >> > but I don't have
> >> > any idea which one.
> >> >
> >> > I hope someone has more insight on this.
> >>
> >> It's curious that things have changed since F33.  Could you build
> >> containers for both and diff the list of package names?  Maybe a
> >> dependency is missing or has become a soft dep.
> >>
> >> rpm -qa --qf '%{name}\n' | sort
> >
> >I ran this:
> >
> >$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 rpm -qa
> >--qf '%{name}\n' | sort > fedora-34.out
> >$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 rpm -qa
> >--qf '%{name}\n' | sort > fedora-33.out
> >$ diff -u fedora-33.out fedora-34.out > fedora.diff
> >$ egrep -- ^- fedora.diff > removed-in-fedora-34
> >$ egrep -- '^[+]' fedora.diff > added-in-fedora-34
> >
> >$ cat removed-in-fedora-34
> >--- fedora-33.out 2021-06-13 22:51:13.156150245 +0300
> >-autogen-libopts
> >-genisoimage
> >-iptables
> >-isl
> >-kbd-legacy
> >-libcbor
> >-libdb-utils
> >-libfido2
> >-libguestfs-tools-c
> >-libmodman
> >-libusal
> >-nmap-ncat
> >-pciutils
> >-pciutils-libs
> >-python3-idna
> >-rdma-core
> >-selinux-policy-minimum
> >
> >$ cat added-in-fedora-34
> >+++ fedora-34.out 2021-06-13 22:50:59.650016271 +0300
> >+guestfs-tools
> >+inih
> >+iptables-legacy
> >+iptables-legacy-libs
> >+langpacks-core-en_GB
> >+langpacks-en_GB
> >+libbsd
> >+libcanberra
> >+libcanberra-gtk3
> >+libcloudproviders
> >+libpmemobj
> >+libretls
> >+libtdb
> >+netcat
> >+pcsc-lite-libs
> >+selinux-policy-targeted
> >+sound-theme-freedesktop
> >
> >I don't see anything that can explain the difference.
> >
> >I also compared contents of the /lib/modules/xxx/ directories:
> >
> >$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 ls -1
> >/lib/modules/5.12.9-200.fc33.x86_64 >
> >fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out
> >$ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 ls -1
> >/lib/modules/5.12.9-300.fc34.x86_64 >
> >fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out
> >$ diff -u fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out
> >fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out
> >--- fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out 2021-06-13
> >23:12:04.177539132 +0300
> >+++ fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out 2021-06-13
> >23:12:18.072675988 +0300
> >@@ -3,23 +3,14 @@
> > build
> > config
> > kernel
> >-modules.alias
> >-modules.alias.bin
> > modules.block
> > modules.builtin
> > modules.builtin.alias.bin
> >-modules.builtin.bin
> > modules.builtin.modinfo
> >-modules.dep
> >-modules.dep.bin
> >-modules.devname
> > modules.drm
> > modules.modesetting
> > modules.networking
> > modules.order
> >-modules.softdep
> >-modules.symbols
> >-modules.symbols.bin
> > source
> > symvers.gz
> > updates
> >
> >Looks like some post install step is missing on the fedora 34 container.
> >
>
> Just to be clear, that is on the ovirt/imageio container, not any usual
> container out there.  No upstream fedora container has anything in /lib/modules
> because it usually is pointless to distribute something that is not going to be
> needed as the container is not coupled with any kernel because it runs on the
> host one.

I think this is up to the package installing virt-builder. We cannot know about
virt-builder dependencies and make sure they are met.

> I tested upstream fedora:33 and fedora:34 and fedora-minimal:34 (with microdnf)
> containers and all got modules.dep file from the kernel-(debug-)core package,
> although it is not owned by the package, so it is clearly installed in a
> postscript or similar.  So it has to be something like not installing the
> package correctly in the imageio container image.

Thanks, I'll test adding this package.

But shouldn't it be a dependency of the package installing virt-builder?

> I've got a tiny hunch, so I must ask.  Is someone, by any stroke of misfortune,
> using bazel for building that imageio container image?

We build the images in quay.io, I don't know how they build them :-)

Nir