[PATCH] Increase timeout for tests and syntax-check

[Pavel Hrdina]

On Fri, Jan 29, 2021 at 02:07:31PM +0100, Michal Privoznik wrote:
> On 1/29/21 1:45 PM, Pavel Hrdina wrote:
> > On Fri, Jan 29, 2021 at 09:30:24AM -0300, Daniel Henrique Barboza wrote:
> > > 
> > > 
> > > On 1/27/21 2:59 PM, Michal Privoznik wrote:
> > > > Since we've switched to meson our tests run with a timeout (meson
> > > > uses 30 seconds as the default). However, not every machine that
> > > > builds libvirt is fast enough to run every test under 30 seconds
> > > > (each test binary has its own timeout, but still). For instance
> > > > when building a package for distro on a farm that's under load.
> > > > Or on a generally slow ARM hardware. While each developer can
> > > > tune their command line for building by adding
> > > > --timeout-multiplier=10, this is hard to do for aforementioned
> > > > build farms.
> > > > 
> > > > It's time to admit that not everybody has the latest, top shelf
> > > > CPU and increase the timeout.
> > > > 
> > > > Signed-off-by: Michal Privoznik <mprivozn at redhat.com>
> > > > ---
> > > 
> > > This sure will help these build farms environments, but what about the cases
> > > where an actual timeout means that there is something wrong with the code?
> > > E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was
> > > seeing tests timing out in Power hosts when the 30 sec timeout was being
> > > enforced.
> > > 
> > > A 120 second default timeout for the majority of the test cases is a long time.
> > > virschematest in this laptop I use takes 2.5 sec to complete. If I do something
> > > wrong in the code and now the test is now 4 times slower (10 sec) I will not be
> > > able to detect it (I'll need to start keeping track or something). You'll have to
> > > run the test suit on your RasPi 2B to see that something went wrong because the
> > > timeout is better tuned to your RasPI than this laptop, but then the code is already
> > > upstream.
> > > 
> > > And the tests will get more complex and will naturally take longer to complete.
> > > Eventually this timeout might no be enough. Increase the timeout again?
> > > 
> > > Meson 0.57 has support for disabling test timeout entirely with --timeout-multiplier=0,
> > > disabling test timeout entirely. Can't we bump the meson version to 0.57 and
> > > then tell the distros to use timeout-multiplier=0? That will solve the problems
> > > for them, I keep the short timeouts for development, and we won't need to keep
> > > bumping the test timeout once every 2 years or something because the s390x
> > > TCG enviroment in Fedora COPR is timing out again.
> > 
> > Agreed. The timeout is useful and we can always find a machine where the
> > current timeout will not be good enough. I don't think it is a good idea
> > to increase the timeout like this for only few specific use-cases.
> 
> So we agree that timeouts are evil :-) Back in the autotools days we did not

No we don't :) timeouts are useful in test suites to reveal issues that
something takes a long time when it shouldn't.

Pavel

> use any timeout at all and I don't remember that causing any trouble. But
> maybe it was so painful that I blocked it in my mind :-)
> 
> Michal
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20210129/bc26f6d0/attachment-0001.sig>