[PATCH] add build dependency on lxc_protocol.h to remote_daemon

Wed Mar 16 10:35:02 UTC 2022

On Wed, Mar 16, 2022 at 09:51:48AM +0000, Daniel P. Berrangé wrote:
> On Wed, Mar 16, 2022 at 09:39:52AM +0100, Michal Prívozník wrote:
> > On 3/14/22 15:58, Slater, Joseph wrote:
> > > I was really only wanting to make sure that both remote_driver and remote_daemon have access to lxc_protocol.h.  I am not an expert in meson builds but I think the only way to do that is have a loop outside of remote_daemon_generated and remote_driver_generated.  You could start with an empty remote_daemon_generated and append remote_xxx_generated after the foreach loop instead of initializing the way I did.
> > > 
> > > So, your 1) is just a side-effect, not something I was explicitly seeking.
> > > 
> > 
> > Hey, I'm no meson expert either, but one thing that I've noticed is that
> > with this patch I'm seeing more tasks that meson wants to run:
> > 
> > Without the patch: 1410
> > With the patch: 1447
> 
> While likely correct in this case, be wary of believing what Meson
> prints as the total number of tasks it intends to run upfront. I've
> seen cases where it claimed it needed to run 100's of tasks, then
> after running 3 tasks, it changed it mind and decided it only
> needed 15 tasks :-)

I've seen that many times. My guess is that it computes the dependency
tree and once meson starts compiling things it will notice that some of
the bits are already compiled and up to date so there is no need to do
it.

> > My understanding was that we need only to order tasks not add new ones.
> > Now, looking at generated build.ninja I can see why:
> > 
> > libvirt.git $ grep "^build.*lxc_protocol.c.o:" build/build.ninja | cut
> > -d':' -f1
> > build
> > src/remote/libvirt_remote_driver.a.p/meson-generated_.._lxc_protocol.c.o
> > build src/libvirtd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtproxyd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtinterfaced.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtnetworkd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtnodedevd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtnwfilterd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtsecretd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtstoraged.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtxend.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtlxcd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtchd.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtqemud.p/meson-generated_.._remote_lxc_protocol.c.o
> > build src/virtvboxd.p/meson-generated_.._remote_lxc_protocol.c.o
> > 
> > Meson decided it's faster to compile remote_lxc_protocol.c multiple
> > times, for different daemons, each time with the same set of arguments
> > (not shown in my paste above), than to compile it just once. I'm sure
> > virqemud is going to be happy with understanding LXC driver protocol,
> > but I am not happy about that. Le sigh.
> 
> I wouldn't say it has decided it is faster to compile many times.
> 
> In the past meson wasn't compiling it at all for the daemons.
> 
> AFAICT, currently the daemons only get access to a the
> remote_protocol.c.o implementations indirectly through
> linking to libvirt.so
> 
> With this patch, we are explicitly telling meson to link
> the daemons with remote_protocol.c.o.
> 
> Meson strives for maximum correctness, so if an object file
> is used from many output targets, it will compile the source
> file separately for each target, to take into account differences
> in compiler flags used for each output target.
> 
> IOW, meson is doing exactly what we told it to do here.
> 
> The result is that the daemons actually get 2 copies of the
> the remote_protocol.c.o - one copy is directly linked into
> the binary, and one copy is indirectly via libvirt.so
> 
> > Pavel, since you introduced meson, can you please shed more light into
> > here? How should we set up ORDERING and not DEPENDENCY? In Makefile it's
> > as easy as:
> 
> ORDERING / DEPENDENCY are the same thing. This patch is not actually
> setting up a dependancy - it is explicitly telling meson to compile the
> files many times.
> 
> I suspect what we need is this:
> 
> diff --git a/src/remote/meson.build b/src/remote/meson.build
> index b2aafe6320..769fc2e97f 100644
> --- a/src/remote/meson.build
> +++ b/src/remote/meson.build
> @@ -3,7 +3,8 @@ remote_driver_sources = [
>    'remote_sockets.c',
>  ]
>  
> -remote_driver_generated = []
> +remote_generated_sources = []
> +remote_generated_headers = []
>  
>  foreach name : [ 'remote', 'qemu', 'lxc' ]
>    client_bodies_h = '@0 at _client_bodies.h'.format(name)
> @@ -11,7 +12,7 @@ foreach name : [ 'remote', 'qemu', 'lxc' ]
>    protocol_h = '@0 at _protocol.h'.format(name)
>    protocol_x = '@0 at _protocol.x'.format(name)
>  
> -  remote_driver_generated += custom_target(
> +  remote_generated_headers += custom_target(
>      client_bodies_h,
>      input: protocol_x,
>      output: client_bodies_h,
> @@ -21,7 +22,7 @@ foreach name : [ 'remote', 'qemu', 'lxc' ]
>      capture: true,
>    )
>  
> -  remote_driver_generated += custom_target(
> +  remote_generated_sources += custom_target(

this should be `remote_generated_headers += custom_target(`

>      protocol_h,
>      input: protocol_x,
>      output: protocol_h,
> @@ -30,7 +31,7 @@ foreach name : [ 'remote', 'qemu', 'lxc' ]
>      ],
>    )
>  
> -  remote_driver_generated += custom_target(
> +  remote_generated_sources += custom_target(
>      protocol_c,
>      input: protocol_x,
>      output: protocol_c,
> @@ -49,7 +50,9 @@ remote_daemon_sources = files(
>    'remote_daemon_stream.c',
>  )
>  
> -remote_daemon_generated = []
> +remote_driver_generated = remote_generated_headers + remote_generated_sources
> +# Daemon gets the impls indirectly via libvirt.so
> +remote_daemon_generated = remote_generated_headers
>  
>  virt_ssh_helper_sources = files(
>    'remote_sockets.c',
> 
> (not tested)

otherwise looks good and should fix the issue.

Pavel

> With regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20220316/20669386/attachment-0001.sig>