[Avocado-devel] no sigterm_handler stack trace on test timeout?

[Cleber Rosa]

On 11/13/18 1:54 PM, Brian J. Murrell wrote:
> On Tue, 2018-11-13 at 12:49 -0500, Cleber Rosa wrote:
>>
>> AFAICT, every test gets this handler installed.
> 
> ...
> 
>> I don't see a code path in which test processes don't have the
>> sigterm
>> handler installed, but there are different code paths regarding the
>> ending of the test process.  On the first example I can see:
>>
>> "ERROR Test reported status but did not finish"
> 
> Right.  After the SIGTERM handler was invoked which it should be
> because the test timed out.
> 
>>
>> While on the second I can see:
>>
>> "ERROR Test died without reporting the status"
> 
> Which means what exactly?  That the python test died out from under
> avocado by some means other than exiting due to an exception?
> 
> These two messages don't seem to square:
> 
> 2018-11-13 01:18:15,555 runner           L0253 ERROR| ERROR Test died
> without reporting the status. -> TestAbortedError: 3-
> /var/lib/jenkins/workspace/daos-stack-org_daos_PR-13-
> centos7/src/tests/ftest/pool/PoolSvc.py:PoolSvc.test_poolsvc;mode_RW-
> createset-createsize-svc3-test_machine1-server-aa67.
> 2018-11-13
> 01:18:15,555 runner           L0255 WARNI| Killing hanged test process
> 29558
> 
> Since the first one is saying the test died but then the second one
> says it's hanged and killing it.  Died and hanged are very different
> and reporting both is confusing at best.
> 

I agree the behavior is weird.  Would you happen to have an easy
reproducer to help us pin down the weirdness?

>> On this second example, the process is supposed to have died already,
>> so
>> it makes no sense to send it a SIGTERM.
> 
> So why did the timeout expire and it report:
> 
> 2018-11-13 01:18:16,663 runner           L0061 ERROR| Runner error occurred: Timeout reached
> 
>> With regards to tearDown(), it's not something that gets run on test
>> the
>>  process signal handler (as a cleanup stage), but it's part of the
>> general test execution flow.
> 
> I'm afraid I'm not parsing the above.
> 

I was basically stating that the test process is responsible for running
setUp()+test()+tearDown().  The runner creates one process with an entry
point that follows those phases.  If an exception happens in, say,
test(), it will be handled accordingly and tearDown() will be executed.

See
https://github.com/avocado-framework/avocado/blob/master/avocado/core/test.py#L810

But, if there's an action impacting the test process, the runner will
not (currently) fire a new one and run just a part of those phases, such
as just the tearDown().  It may be possible to implement such a
behavior, but there may also be some corner cases with tests that may
fail to run __init__() more than once the same way.

>> Avocado attempts to make sure tearDown()
>> is executed in spite of error conditions on the test() methods, but
>> not
>> to the point of intercepting SIGTERM and running tearDown() code at
>> that
>> time.
> 
> So tearDown() is not suppose to run if a test times out?  I think my
> first example from the previous post proves that's not true and that
> when a test times out tearDown() is run.
> 

That's because the SIGTERM is caught, and turned into an exception at
the sigterm_handler, which in turn is dealt properly (as explained
before).  But, for instance sending a SIGKILL to the test process will
not be caught, will not become an exception, so tearDown() won't be
executed (AFAICT).

>> Is this what you're trying to achieve here?
> 
> I am trying to make sure that no matter what the test does, if setUp()
> even got started (and perhaps it times out before completing)
> tearDown() is run (which needs 
> https://github.com/avocado-framework/avocado/issues/2908 to ensure it
> has time to run it in the event of a timeout).
> 
> TL;DR, every setUp() (even if it times out) requires a tearDown() no
> matter what happens in between.
> 

Yes, I see your point.  Like I said before, there may be a few
challenges to solve, but I'd like very much to achieve what you propose
here.

> Cheers,
> b.
> 
> 

Thanks!

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]