[Avocado-devel] [Autotest] Feasibility study - issues clarification

[Cleber Rosa]

----- Original Message -----
> From: "Ademar Reis" <areis at redhat.com>
> To: "Cleber Rosa" <crosa at redhat.com>
> Cc: "Lukasz Majewski" <l.majewski at samsung.com>, "avocado-devel" <avocado-devel at redhat.com>,
> autotest-kernel at redhat.com
> Sent: Thursday, February 11, 2016 3:08:16 PM
> Subject: Re: [Avocado-devel] [Autotest] Feasibility study - issues clarification
> 
> On Thu, Feb 11, 2016 at 08:33:39AM -0500, Cleber Rosa wrote:
> > 
> > 
> > ----- Original Message -----
> > > From: "Lukasz Majewski" <l.majewski at samsung.com>
> > > To: autotest-kernel at redhat.com
> > > Sent: Thursday, February 11, 2016 6:27:22 AM
> > > Subject: [Autotest]  Feasibility study - issues clarification
> > > 
> > > Dear all,
> > > 
> > > I'd be grateful for clarifying a few issues regarding Autotest.
> > > 
> > > I have following setup:
> > > 1. Custom HW interface to connect Target to Host
> > > 2. Target board with Linux
> > > 3. Host PC - debian/ubuntu.
> > > 
> > > I would like to unify the test setup and it seems that the Autotest
> > > test framework has all the features that I would need:
> > > 
> > > - Extensible Host class (other interfaces can be used for communication
> > >   - i.e. USB)
> > > - SSH support for sending client tests from Host to Target
> > > - Control of tests execution on Target from Host and gathering results
> > > - Standardized tests results format
> > > - Autotest host's and client's test results are aggregated and
> > >   displayed as HTML
> > > - Possibility to easily reuse other tests (like LTP, linaro's PM-QA)
> > > - Scheduling, HTML visualization (if needed)
> > > 
> > > On the beginning I would like to use test harness (server+client) to
> > > run tests and gather results in a structured way.
> > > 
> > > However, I have got a few questions (please correct me if I'm wrong):
> > > 
> > > - On several presentations it was mentioned that Avocado project is a
> > >   successor of Autotest. However it seems that Avocado is missing the
> > >   client + server approach from Autotest.
> > 
> > Right. It's something that is being worked on at this very moment:
> > 
> > https://trello.com/c/AnoH6vhP/530-experiment-multiple-machine-support-for-tests
> > 
> > > 
> > > - What is the future of Autotest? Will it be gradually replaced by
> > >   Avocado?
> > 
> > Autotest has been mostly in maintenance mode for the last 20 months or
> > so. Most of the energy of the Autotest maintainers has been shifted
> > towards Avocado. So, while no Open Source project can be killed (nor
> > should), yes, Autotest users should start looking into Avocado.
> > 
> > > 
> > > - It seems that there are only two statuses returned from a simple
> > >   test (like sleeptest), namely "PASS" and "FAIL". How can I indicate
> > >   that the test has ended because the environment was not ready to run
> > >   the test (something similar to LTP's "BROK" code, or exit codes
> > >   complying with POSIX 1003.1)?
> > 
> > I reckon this is a question on Autotest test result status, so I'll try
> > to answer in that context. First, the framework itself gives you
> > intentionally
> > limited test result status. If you want to save additional information
> > about
> > your test, including say the mapping to POSIX 1003.1 codes, you can try to
> > use
> > the test's "keyval" store for that. The "keyval" is both saved to a local
> > file
> > and to the server's database (when that is used).
> 
> You're probably referring to the whiteboard:
> http://avocado-framework.readthedocs.org/en/latest/WritingTests.html#saving-test-generated-custom-data

Nope, I was describing how that could be done in Autotest. I cross-posted the
message to the Avocado list, since it also related to it.

Then, yes, in Avocado, the whiteboard could be used for that.

> 
> Thanks.
>    - Ademar
> 
> > 
> > Avocado INSTRUMENTED tests, though, have a better separation of test setup
> > and
> > execution, and a test can be SKIPPED during the setup phase. A few
> > pointers:
> > 
> >  *
> >  https://github.com/avocado-framework/avocado/blob/master/examples/tests/skiponsetup.py
> >  *
> >  http://avocado-framework.readthedocs.org/en/latest/api/core/avocado.core.html#avocado.core.test.Test.skip
> > 
> > > 
> > > - Is there any road map for Autotest development? I'm wondering if
> > >   avocado's features (like per test SHA1 generation) would be ported to
> > >   Autotest?
> > 
> > Not really. Avocado's roadmap though, is accessible here:
> > 
> > https://trello.com/b/WbqPNl2S/avocado
> > 
> 
> --
> Ademar Reis
> Red Hat
> 
> ^[:wq!
>