[libvirt] [PATCH 0/4] build: take full ownership of syntax-check from gnulib

[Daniel P. Berrangé]

On Mon, Oct 07, 2019 at 04:31:29PM +0200, Michal Privoznik wrote:
> On 10/7/19 3:43 PM, Daniel P. Berrangé wrote:
> > On Mon, Oct 07, 2019 at 03:03:51PM +0200, Michal Privoznik wrote:
> > > On 10/3/19 12:51 PM, Daniel P. Berrangé wrote:
> > > > Our syntax-check rule relies in various files imported during the gnulib
> > > > bootstrap process. As we switch to meson & try to eliminate gnulib, we
> > > > need to take ownership of syntax-check.
> > > > 
> > > > This patch series follows the approach taken for libosinfo & other
> > > > projects wrt syntax-check rules, but goes a step further and eliminates
> > > > alot of baggage we don't care about & merges all logic into one file.
> > > > 
> > > > Daniel P. Berrangé (4):
> > > >     build: import gnulib's syntax-check make rules
> > > >     build: remove all logic unrelated to syntax-check
> > > >     build: delete all syntax check rules we're skipping
> > > >     build: merge all syntax-check logic into one file
> > > > 
> > > >    Makefile.am                         |    7 +-
> > > >    bootstrap.conf                      |    4 -
> > > >    cfg.mk => build-aux/syntax-check.mk | 1141 +++++++++++++++++++++++++--
> > > >    build-aux/useless-if-before-free    |  226 ++++++
> > > >    build-aux/vc-list-files             |  113 +++
> > > >    5 files changed, 1411 insertions(+), 80 deletions(-)
> > > >    rename cfg.mk => build-aux/syntax-check.mk (52%)
> > > >    create mode 100755 build-aux/useless-if-before-free
> > > >    create mode 100755 build-aux/vc-list-files
> > > > 
> > > 
> > > Problem I'm seeing with these patches is that when I "make -j10 all
> > > syntax-check check" (which is how I usually compile libvirt) then make does
> > > not order targets properly and:
> > 
> > That's odd. There's no dependancy between 'syntax-check' and 'all' that
> > I know of even in current git master. We can run a syntax-check without
> > forcing an 'all' build. 'check' has a dep on 'all' obviously so those
> > two are ordered.
> > 
> > IOW, I would have expect make to parallelize 'syntax-check' with execution
> > of 'all' and 'check' already.
> > 
> > I wonder what black magic is preventing this....
> > 
> > > 1) I can see syntax-check output interleaved with 'all' output (or even
> > > 'check'), but which is worse:
> > > 
> > > 2) there must be a dependency missing somewhere, because after 'git clean
> > > -fxd; ./autogen.sh --system' I see the following error:
> > > 
> > > 
> > > make[4]: Entering directory
> > > '/home/zippy/work/libvirt/libvirt.git/gnulib/lib'
> > >    GEN      sys/ioctl.h
> > >    GEN      arpa/inet.h
> > >    GEN      sys/select.h
> > >    GEN      pthread.h
> > >    GEN      sys/socket.h
> > >    GEN      sched.h
> > >    GEN      stdio.h
> > >    GEN      sys/stat.h
> > >    GEN      string.h
> > >    GEN      stdlib.h
> > >    GEN      sys/time.h
> > >    GEN      sys/socket.h
> > >    GEN      sys/stat.h
> > >    GEN      sys/types.h
> > > mv: cannot stat 'string.h-t': No such file or directory
> > >    GEN      sys/uio.h
> > > make[4]: *** [Makefile:4311: string.h] Error 1
> > > make[4]: *** Waiting for unfinished jobs....
> > 
> > What distro do you get this on ?  I can't reproduce that problem yet
> 
> This is gentoo, but I'm able to reproduce on rawhide too. The important part
> is to enable parallel build (make -j10 or so).

I think I might see the cause.  The GNUmakefile has some black magic
in it to block parallelisation

ALL_RECURSIVE_TARGETS += $(AM_RECURSIVE_TARGETS)

ifneq ($(word 2, $(MAKECMDGOALS)), )
ifneq ($(filter $(ALL_RECURSIVE_TARGETS), $(MAKECMDGOALS)), )
.NOTPARALLEL:
endif
endif

This means that the three targets you list are built strictly in
order.

My change looses this rule and I thin that probably casues both
the failures you report. I'll repost a new series with a fix
for this

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|