[Libguestfs] 1.39 proposal: Let's split up the libguestfs git repo and tarballs

[Daniel P. Berrangé]

On Mon, Feb 12, 2018 at 12:09:23PM +0000, Richard W.M. Jones wrote:
> On Mon, Feb 12, 2018 at 09:22:30AM +0000, Daniel P. Berrangé wrote:
> > On Fri, Feb 09, 2018 at 06:01:53PM +0000, Richard W.M. Jones wrote:
> > > My contention is that the libguestfs git repository is too large and
> > > unwieldy.  There are too many separate, unrelated projects and as a
> > > result of that the source has too many dependencies and takes too long
> > > to build and test.
> > > 
> > > The project divides (sort of) naturally into layers -- the library,
> > > the bindings, the various virt tools -- and could be split along those
> > > lines into separate projects which can then be released and evolve at
> > > their own pace.
> > > 
> > > My suggested split would be something like this:
> > > 
> > > * libguestfs: The library, daemon and appliance.  That would include
> > >   the following directories in a single project:
> > >        appliance
> > >        bash
> > >        contrib
> > >        daemon
> > >        docs
> > >        examples
> > >        gnulib
> > >        lib
> > >        logo
> > >        test-tool
> > >        tmp
> > >        utils
> > >        website
> > > 
> > > * 1 project for each language binding:
> > >        csharp
> > >        erlang
> > >        gobject
> > >        golang
> > >        haskell
> > >        java
> > >        lua
> > >        ocaml
> > >        php
> > >        perl
> > >        python
> > >        ruby
> > 
> > So, the core library above would still include the API description
> > and "make install" it into some location, such that these language
> > bindings cna auto-generate themselves I presume. I guess that means
> > you would rarely need to do releases of these language bindings, as
> > one release ought to be capable of being built against multiple
> > versions of the core library ?
> 
> Certainly the language bindings are the hardest to deal with but also
> the most important to move out in terms of reducing dependencies.  The
> "API description" continues to be the generator, turned into a git
> submodule and shared across all of them.

Is there any sense in having the generator program "make install"d by
the core library, so it can be consumed by the language bindings that
way ? You can use  pkg-config to report custom variables to inform
them where it can be found.

eg libvirt uses this for its API XML description

$ pkg-config --variable libvirt_api libvirt
/usr/share/libvirt/api/libvirt-api.xml

> But it's not a fully formed plan.  One particular difficulty is - as
> you note - that some of the bindings cannot be compiled against a
> different version of libguestfs (we discovered this when we turned the
> Python bindings into a PyPi module), so likely they'd all need to be
> released at the same time, or else modified so at least they need a
> minimum version of libguestfs.

I don't know if you solved the problem wrt python bindings, but I think
it is particularly important to be able to compile the language bindings
against arbitrary versions if at all possible.  Applications like OpenStack
like to track the latest pypi versions, but at the same time have to work
against whatever libguestfs C version the distro provides.

For libvirt, we had a cut-off point at 0.9.11 as min C library version
for the python to be able to build against, as that's where we started
installing the API XML description in -devel packages.

> > > * I'd be inclined to drop the legacy Perl tools virt-tar,
> > >   virt-list-filesystems, virt-list-partitions unless someone
> > >   especially wished to step forward to maintain them.
> > > 
> > > * common code and generator: Off to the side we'd somehow need to
> > >   package up the common code and the generator for use by all of the
> > >   above projects.  It wouldn't be a separate project for downstream
> > >   packagers, but instead the code would be included (ie. duplicated)
> > >   in tarballs and upstream available as a side git repo that you'd
> > >   need to include when building (git submodule?).  This is somewhat
> > >   unspecified.
> > 
> > I guess sub-modules are reasonable for this, unless you actually
> > modulized the generator itself, such that the language binding
> > generation code could be a loadable module. That way the core
> > generator could be in the core library (and its -devel) package,
> > and the language binding repo could have the langauge specific
> > plugin for the generator ?
> 
> We can keep the generator as a single program with only a small
> modification (it needs to check if a directory exists before putting
> files there).
> 
> How exactly this all works when compiling from tarballs is also not
> clear.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|