[virt-tools-list] [PATCH virtio-win-pkg-scripts v3 0/2] helpers to standardize driver directory layout

[Roman Kagan]

On Wed, Feb 17, 2016 at 06:11:31PM -0500, Cole Robinson wrote:
> On 02/08/2016 09:23 AM, Roman Kagan wrote:
> > On Sat, Feb 06, 2016 at 06:20:42PM -0500, Cole Robinson wrote:
> > Actually we started putting together our stuff, too, and encountered a
> > number of issues I'd be interested to discuss.
> > 
> 
> Can I ask what you plan to do with virtio-win RPM/ISO on your side? I'm trying
> to spec out in my head all the bits that need to change, so understanding what
> you're needs are will help my get things straight.
> 
> Are you just interested in extending the RHEL RPM, or the public one as well?
> What types of bits are you adding? drivers, agents, etc

Our top-level goal is to ship a set of Windows guest drivers and tools
in the upcoming major version of Virtuozzo.  This includes kvm paravirt
drivers, qga, and a number of in-house management tools to be run in the
guest via qga guest-exec (mainly for compatibility with previous
versions of Virtuozzo).

We're not extending the RHEL RPM nor "the public one"; however I thought
it could make sense to consolidate the efforts on that.  The reason why
I started this thread was that we needed a well-defined convention for
storing drivers per arch/os for consumption by humans or other tools
like libguestfs; but presumably we have more things in common so it
looks like a good idea to share that.

> > 1) since the drivers can only be built on a Windows machine, there's a
> >    need to define the artifact(s) produced on a Windows machine and used
> >    as source(s) for rpmbuild on a Linux machine (koji).
> > 
> >    We thought the most natural responsibility split (as applied to
> >    kvm-guest-drivers-windows) is to collect the stuff in "Install"
> >    subdirectories of every driver upon buildAll.bat execution in a
> >    single zip archive and use it as that artifact; the rest will be
> >    taken care of from under rpmbuild on a Linux machine.  One benefit is
> >    that it's going to be easy to intervene manually, e.g.  when doing
> >    signatures or whatever else.
> > 
> >    This differs from what's currently in virtio-win srpm where the
> >    drivers are received in a tarball whose contents was obviously
> >    subject to certain manipulations after building on Windows.
> > 
> 
> I'm not really familiar with the windows build process, so I'm not sure I
> follow all of that. But the idea of pushing some of this rework down into the
> kvm-guest-drivers-windows build infrastructure sounds like a good one to me,
> since as you say that's the root source.
> 
> Regarding handling everything in rpmbuild, I'm not entirely sure how I feel.
> Doing all the processing at rpmbuild time is probably okay as long as the
> logic doesn't live entirely in the .spec file, and exists as external scripts
> that can be tested independently of actually triggering 'rpmbuild'. Then say
> virtio-win-pkg-scripts-gitXXX.tar.gz should just be about SOURCE in the spec
> file. That would certainly streamline the process a bit and make it more clear
> exactly what is going on.
> 
> Though on the RHEL side that will be a bit difficult, because the shipped
> drivers are not composed of output from a single virtio-win build, but from
> _many_ virtio-win builds.
> 
> For example, the latest winxp viostor WHQL submission might have used drivers
> from internal build virtio-win-prewhql-26, and the latest win7 vioser WHQL
> submission may have used bits from virtio-win-prewhql-52.
> 
> Now a new internal build pops up, virtio-win-prewhql-78. It's used to feed
> win10 WHQL balloon submission. That build may ship have a newer winxp viostor
> driver and a newer win7 vioser driver. But we aren't going to ship them,
> because we are tied to the content that was used for the last WHQL submission.
> 
> Basically we have an internal git repo that tracks all whql submissions, their
> associated prewhql input (which is raw kvm-guest-windows-drivers build
> output), and we use the mappings to create a frankenstein symlink tree that
> matches the driver layout you see on the virtio-win iso, or from upstream
> make-driver-dir.py output. The public and RHEL build process then sync's up
> the rest of the way.
> 
> So not sure how to handle that in rpmbuild without jumping through more hoops.
> 
> That said, a fix to sync up the processes a bit more is to generate a symlink
> tree that matches the same layout as the kvm-guest-windows-drivers build
> output. Then the public and RHEL build processes would be identical, except
> for RHEL's prep step of splicing in the WHQL output.

That's approximately what I'm trying to do, too.  The idea is that
rpmbuild gets as its source a zip archive with windows drivers, with the
only requirement that every driver is in its own directory with all the
necessary files and nothing else.  The archive can be generated on a
Windows machine upon build, or in whatever else process including
manually.

Then laying them out in the defined way is a job for the script I
posted, called from %build phase of rpm spec.

> > 3) current virtio-win package includes several loosely related packages:
> >    virtio drivers, qxl, qga; we add our own, too.  They have independent
> >    release cycles and build procedures (e.g. qga builds nicely with
> >    mingw on Linux and can be turned into a normal rpm suitable for
> >    shipping in e.g. Fedora)
> > 
> 
> Certainly for the public virtio-win distribution there's no good reason to be
> dependent on internal RH builds for qemu-guest-agent. It only works like that
> because I haven't gotten fixing it.
> 
> There's a question then of where we get our qemu-guest-agent build. I don't
> know if having a mingw-qemu-guest-agent RPM in fedora would be acceptable,
> shipping a windows binary like that. It could be a separate RPM that is
> distributed in the virtio-win.repo but it'll take a bit of scripting to set up.
> 
> But while it seems feasible for the public distribution, it's less so for
> RHEL. RHEL doesn't have mingw and likely never will. The qemu-ga builds we use
> in RHEL are actually performed on a windows machine, so aren't distributed in
> an RPM. So this bit may end up as a diverging point for RHEL vs public/fedora
> virtio-win RPM building

Well there's a precedent already: Fedora ships virt-v2v with
/usr/share/virt-tools/rhsrvany.exe which is a symlink to
/usr/i686-w64-mingw32/sys-root/mingw/bin/rhsrvany.exe which belongs to
mingw32-srvany and is GPL and built with mingw.  Now RHEL ships it as a
binary, and that binary is an exact copy of the one from Fedora.

> >    Therefore we're considering splitting the rpm into several
> >    independent ones with the stuff installed onto the host filesystem as
> >    a directory structure; the iso and floppy images would be generated
> >    from that right on the host (e.g. by rpm triggers or by a tool like
> >    supermin) rather than brought in pre-generated.
> 
> Hmm now that's an interesting idea! A little scary though :) I wouldn't want
> to have to install the RPMs locally just to get an iso or vfd image to play
> with or test. And not sure how that will work for RHEL if we don't end up
> RPMifying qemu-ga (or eventually spice agent) for example
> 
> Certainly splitting the RPM at least into sub-RPMs is a no brainer, though I'd
> need to check with RHEL consumers and documentation to make sure any
> dependencies are updated. Maybe 'virtio-win' becomes a meta package that pulls
> in virtio-win-drivers (/usr/share/virtio-win/drivers) and virtio-win-media
> (iso and vfds), that would probably make things back compat. But that doesn't
> help with the separate release cycle point you mention.

We're still discussing this internally as well; the risk of improper
updating of generated contents is the main one we fear of.

Roman.