[PATCH 3/4] qemuxml2argvtest: Increase timeout on macOS

[Roman Bolshakov]

On Fri, Nov 13, 2020 at 04:58:36PM +0100, Michal Privoznik wrote:
> On 11/8/20 10:24 PM, Roman Bolshakov wrote:
> > The test takes 100+ seconds if all test suite is run with meson and 45+
> > seconds if it's run directly. According to the output of sample tool,
> > most of the time (~72 seconds) is spent in poll():
> > 
> > Sort by top of stack, same collapsed (when >= 5):
> >          poll  (in libsystem_kernel.dylib)                         72396
> >          tiny_free_no_lock  (in libsystem_malloc.dylib)             1726
> >          tiny_malloc_should_clear  (in libsystem_malloc.dylib)      1671
> >          free_tiny  (in libsystem_malloc.dylib)                     1287
> >          tiny_free_list_add_ptr  (in libsystem_malloc.dylib)        1258
> >          tiny_malloc_from_free_list  (in libsystem_malloc.dylib)    1256
> > 
> > Closes https://gitlab.com/libvirt/libvirt/-/issues/58
> > Signed-off-by: Roman Bolshakov <r.bolshakov at yadro.com>
> > ---
> >   tests/meson.build | 8 +++++++-
> >   1 file changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tests/meson.build b/tests/meson.build
> > index 8decdfa20c..3b77a211bb 100644
> > --- a/tests/meson.build
> > +++ b/tests/meson.build
> > @@ -571,7 +571,13 @@ foreach data : tests
> >       ],
> >       export_dynamic: true,
> >     )
> > -  test(data['name'], test_bin, env: tests_env)
> > +  if data['name'] == 'qemuxml2argvtest' and host_machine.system() == 'darwin'
> > +    timeout = 180
> > +  else
> > +    # default meson timeout
> > +    timeout = 30
> > +  endif
> > +  test(data['name'], test_bin, env: tests_env, timeout: timeout)
> >   endforeach
> > 
> 
> I think the last time I wanted to increase the timeout I was told that it is
> machine specific and since I know my machine the best I should use: meson
> test --timeout-multiplier=X
> 

But qumexml2argvtest has 1000+ test cases. And prior to the series the
test never fully passed. In some sense the series raises a baseline (but
with a higher timeout than on Linux).

> Do you need this for the next cirrus patch? If so I think we should just add
> --timeout-multiplier= there.
> 

No, I needed it for local environment. And I profiled it a bit to find
where the time is spent [1]. I can also build a flamegraph if it helps.

FWIW, it takes less time in Cirrus CI [2] than on MBP Pro 2012 but
timeout margin is quite narrow for default timeout:

90/127 qemuxml2argvtest                        OK      29.4992618560791 s

So, I've measured test execution time on a few apple laptops at hand.

On 16" MBP 2019 I get:
 90/285 libvirt / qemuxml2argvtest      OK             22.11s

On 15" MBP 2012:
 80/117 qemuxml2argvtest                OK             48.31s

On 13" MBA 2015 running on battery:
 80/117 qemuxml2argvtest                OK             50.20s

On 13" MBA 2015 running on A/C power:
 80/117 qemuxml2argvtest                OK             48.71s

Without the explicit timeout everyone has to remember that at least "-t 2"
has to be always passed to "meson test" on darwin. In my opinion,
project defaults should be reasonable without tweaking, to allow nice
"meson test -C build" to run all tests and be sure that they run without
failures if no issues are actually introduced, i.e. there should be no
false positives. This is especially important for "meson dist".

What do you think about other timeout values for qemuxml2argvtest on
darwin: 60s, 90s, 120s, 150s? Is there a chance that one of them would
be ok?

1. https://www.dropbox.com/s/0rxid8me0976m69/qemuxml2argvtest_2020-11-08_220422_4J2C.sample.txt.xz?dl=0
2. https://gitlab.com/roolebo/libvirt/-/jobs/837071634

Thanks,
Roman