[libvirt] [PATCH v2 0/6] rewrite virt-host-validate to be data driven, using Go & YAML
Daniel P. Berrangé
berrange at redhat.com
Mon Sep 30 14:48:29 UTC 2019
On Mon, Sep 30, 2019 at 10:47:39AM +0200, Martin Kletzander wrote:
> On Fri, Sep 27, 2019 at 01:52:19PM +0100, Daniel P. Berrangé wrote:
> > This is a followup to a previous PoC patch I submitted a
> > month ago:
> >
> > https://www.redhat.com/archives/libvir-list/2019-September/msg00036.html
> >
> > The commit messages in the individual patches given quite a
> > bit of detail, so I'll keep this cover letter brief.
> >
> > In my previous posting I was unhappy with the implications for
> > the RPM packaging, and was considering having this as a separate
> > source repo & RPM. On further investigation such an approach
> > would not in fact solve the RPM packaging problem, because we
> > would still not be using a pure go build toolchain, as we have
> > data files that need installing in the right place.
> >
> > This forced me to actually address the RPM packaging problems
> > that Fedora had with Go when used from a build tool like make
> > or meson.
> >
> > After alot of debugging I finally got a viable solution merged
> > into the Fedora go-rpm-macros package:
> >
> > https://pagure.io/go-rpm-macros/c/67b4fbbbfce0986ac46cd1329bf85a18ea7a43d2
> >
> > commit 67b4fbbbfce0986ac46cd1329bf85a18ea7a43d2
> > Author: Daniel P. Berrangé <berrange at redhat.com>
> > Date: Wed Sep 18 16:49:58 2019 +0100
> >
> > macros: define a %gobuildflags macro
> >
> > Using the %gobuild macro is fine for a project where the go
> > code is the only thing being built, and can be built directly
> > by invoking the Go toolchain from RPM.
> >
> > In more complex cases though, the Go code is just a small part
> > of the project and the Go toolchain is invoked by a build
> > system such as make (possibly automake), or meson. In such a
> > case we need to be able to tell this build system what flags
> > to pass to the compiler.
> >
> > The %gobuildflags macros services this purpose allowing a
> > RPM spec todo
> >
> > GOBUILDFLAGS="%gobuildflags" %configure
> >
> > or
> >
> > %make GOBUILDFLAGS="%gobuildflags"
> >
> > Ideally the %gobuild macro would in turn reference the
> > %gobuildflags macro, but that does not appear possible
> > given the semantics around quote expansion and escaping
> > across RPM and shell.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
> >
> > As a result in this series, we're now fully integrated into the
> > RPM build, on Fedora at least. I've not checked what approach
> > RHEL takes for Go, whether it requires separate RPM for each
> > 3rd party dep, or prefers bundling. Either way though, we can
> > deal with the problem now.
> >
> > The other obvious change is that this is now a patch series,
> > to make it easier to review the code in managable chunks.
> >
> > The really big difference though is that I replaced the use
> > of XML data files with YAML data files. This was done with
> > the aim of making the data more human friendly. XML is really
> > optimized for machines, not humans, so writing the data files
> > was not pretty. YAML is optimized for human readability, and
> > is actually even easier to consume in Go than the XML was,
> > so its a double win.
> >
> > Finally, we also add new checks at the end for the various
> > CPU hardware side channel mitigations, and report whether
> > SMT/HT is unsafe or not (any Intel host is basically unsafe
> > before Icelake).
> >
> > Daniel P. Berrangé (6):
> > build: introduce logic for using golang in libvirt
> > tools: introduce a data driven impl of virt-host-validate
> > tools: define YAML rules for virt-host-validate checks
> > tools: switch to build the new virt-host-validate impl
> > tools: delete the old virt-host-validate impl
> > tools: make virt-host-validate check CPU vulnerabilities
> >
> > configure.ac | 1 +
> > libvirt.spec.in | 35 +-
> > m4/virt-golang.m4 | 46 ++
> > m4/virt-host-validate.m4 | 8 +-
> > po/POTFILES | 5 -
> > tools/Makefile.am | 76 +--
> > tools/host-validate/go.mod | 10 +
> > tools/host-validate/go.sum | 9 +
> > tools/host-validate/main.go | 98 +++
> > tools/host-validate/pkg/engine.go | 481 ++++++++++++++
> > tools/host-validate/pkg/facts.go | 585 ++++++++++++++++++
> > .../pkg/facts_test.go} | 36 +-
> > tools/host-validate/rules/builtin.yaml | 20 +
> > tools/host-validate/rules/cpu.yaml | 50 ++
> > tools/host-validate/rules/freebsd-kernel.yaml | 77 +++
> > tools/host-validate/rules/linux-acpi.yaml | 39 ++
> > tools/host-validate/rules/linux-cgroups.yaml | 470 ++++++++++++++
> > .../rules/linux-cpu-hardware-flaws.yaml | 165 +++++
> > tools/host-validate/rules/linux-cpu.yaml | 134 ++++
> > tools/host-validate/rules/linux-devices.yaml | 71 +++
> > tools/host-validate/rules/linux-iommu.yaml | 113 ++++
> > .../host-validate/rules/linux-namespaces.yaml | 119 ++++
> > tools/host-validate/rules/linux-pci.yaml | 10 +
> > tools/virt-host-validate-bhyve.c | 77 ---
> > tools/virt-host-validate-common.c | 419 -------------
> > tools/virt-host-validate-common.h | 85 ---
> > tools/virt-host-validate-lxc.c | 87 ---
> > tools/virt-host-validate-lxc.h | 24 -
> > tools/virt-host-validate-qemu.c | 116 ----
> > tools/virt-host-validate-qemu.h | 24 -
> > tools/virt-host-validate.c | 152 -----
> > tools/virt-host-validate.pod | 12 +-
> > 32 files changed, 2609 insertions(+), 1045 deletions(-)
>
> So this ^^ plus:
>
> 2 languages added, 0 languages removed
>
> makes me feel like this goes against what you were trying to do in another
> series. I understand that adding new fact checks is "easier" and does not
> require recompilation, but I don't see any use for that benefit. I went through
> the cover letters for both series just to find a reason for it and I didn't.
The language consolidation is not about reducing languages as an absolute.
Where we were using multiple langauges, to tackle problems in the same
problem space, we consolidated on one language for consistency. This opens
the door to adding use of other languages in libvirt, without continually
increasing the burden of developers.
The mix of Perl, Python & Shell all for doing misc build scripts was the
prime example as there was nothing that can be done in Perl or SHell that
could not be done just as well in Python. There's no justification for
using all three languages in this context.
In this case, our existing C language for writing our production code is
not well suited for the task we're applying it too. You could make an
argument that the virt-host-validate could be written in Python, but
I think it is preferred to keep all our deployable code written in
compiled languages, leaving the scripting languages for our supporting
build system.
Counting the addition of another language solely against this tool is not
a reflection on the long term direction of libvirt code. It is highly
likely that we'll incorporate Go code for more use cases over time as it
is a good systems programming language, able to replace C where Python
could not.
The choice of YAML is a valid point to use as that is definitely adding
a new language where one already exists (XML) that is expressive enough
to cope with the problem. As you saw the original v1 did indeed use XML
for this reason.
The decision to try YAML was an experiment to see if the simplicity and
readability of YAML would outweigh the cost of having another data language.
Personally I think it is a net win to use YAML instead of XML, but it
would not be the end of the world to go back to the v1 approach using
XML if people really want a less readable language just to not have to
add another data language.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
More information about the libvir-list
mailing list