[PATCH] Increase timeout for tests and syntax-check

[Michal Privoznik]

On 1/29/21 1:45 PM, Pavel Hrdina wrote:
> On Fri, Jan 29, 2021 at 09:30:24AM -0300, Daniel Henrique Barboza wrote:
>>
>>
>> On 1/27/21 2:59 PM, Michal Privoznik wrote:
>>> Since we've switched to meson our tests run with a timeout (meson
>>> uses 30 seconds as the default). However, not every machine that
>>> builds libvirt is fast enough to run every test under 30 seconds
>>> (each test binary has its own timeout, but still). For instance
>>> when building a package for distro on a farm that's under load.
>>> Or on a generally slow ARM hardware. While each developer can
>>> tune their command line for building by adding
>>> --timeout-multiplier=10, this is hard to do for aforementioned
>>> build farms.
>>>
>>> It's time to admit that not everybody has the latest, top shelf
>>> CPU and increase the timeout.
>>>
>>> Signed-off-by: Michal Privoznik <mprivozn at redhat.com>
>>> ---
>>
>> This sure will help these build farms environments, but what about the cases
>> where an actual timeout means that there is something wrong with the code?
>> E.g. commits 46d88d8dba56 and 2ba0b7497ce7 were only possible because I was
>> seeing tests timing out in Power hosts when the 30 sec timeout was being
>> enforced.
>>
>> A 120 second default timeout for the majority of the test cases is a long time.
>> virschematest in this laptop I use takes 2.5 sec to complete. If I do something
>> wrong in the code and now the test is now 4 times slower (10 sec) I will not be
>> able to detect it (I'll need to start keeping track or something). You'll have to
>> run the test suit on your RasPi 2B to see that something went wrong because the
>> timeout is better tuned to your RasPI than this laptop, but then the code is already
>> upstream.
>>
>> And the tests will get more complex and will naturally take longer to complete.
>> Eventually this timeout might no be enough. Increase the timeout again?
>>
>> Meson 0.57 has support for disabling test timeout entirely with --timeout-multiplier=0,
>> disabling test timeout entirely. Can't we bump the meson version to 0.57 and
>> then tell the distros to use timeout-multiplier=0? That will solve the problems
>> for them, I keep the short timeouts for development, and we won't need to keep
>> bumping the test timeout once every 2 years or something because the s390x
>> TCG enviroment in Fedora COPR is timing out again.
> 
> Agreed. The timeout is useful and we can always find a machine where the
> current timeout will not be good enough. I don't think it is a good idea
> to increase the timeout like this for only few specific use-cases.

So we agree that timeouts are evil :-) Back in the autotools days we did 
not use any timeout at all and I don't remember that causing any 
trouble. But maybe it was so painful that I blocked it in my mind :-)

Michal