[Avocado-devel] [Autotest] Feasibility study - issues clarification

[Cleber Rosa]

----- Original Message -----
> From: "Lukasz Majewski" <l.majewski at samsung.com>
> To: autotest-kernel at redhat.com
> Sent: Thursday, February 11, 2016 6:27:22 AM
> Subject: [Autotest]  Feasibility study - issues clarification
> 
> Dear all,
> 
> I'd be grateful for clarifying a few issues regarding Autotest.
> 
> I have following setup:
> 1. Custom HW interface to connect Target to Host
> 2. Target board with Linux
> 3. Host PC - debian/ubuntu.
> 
> I would like to unify the test setup and it seems that the Autotest
> test framework has all the features that I would need:
> 
> - Extensible Host class (other interfaces can be used for communication
>   - i.e. USB)
> - SSH support for sending client tests from Host to Target
> - Control of tests execution on Target from Host and gathering results
> - Standardized tests results format
> - Autotest host's and client's test results are aggregated and
>   displayed as HTML
> - Possibility to easily reuse other tests (like LTP, linaro's PM-QA)
> - Scheduling, HTML visualization (if needed)
> 
> On the beginning I would like to use test harness (server+client) to
> run tests and gather results in a structured way.
> 
> However, I have got a few questions (please correct me if I'm wrong):
> 
> - On several presentations it was mentioned that Avocado project is a
>   successor of Autotest. However it seems that Avocado is missing the
>   client + server approach from Autotest.

Right. It's something that is being worked on at this very moment:

https://trello.com/c/AnoH6vhP/530-experiment-multiple-machine-support-for-tests

> 
> - What is the future of Autotest? Will it be gradually replaced by
>   Avocado?

Autotest has been mostly in maintenance mode for the last 20 months or
so. Most of the energy of the Autotest maintainers has been shifted
towards Avocado. So, while no Open Source project can be killed (nor
should), yes, Autotest users should start looking into Avocado.

> 
> - It seems that there are only two statuses returned from a simple
>   test (like sleeptest), namely "PASS" and "FAIL". How can I indicate
>   that the test has ended because the environment was not ready to run
>   the test (something similar to LTP's "BROK" code, or exit codes
>   complying with POSIX 1003.1)?

I reckon this is a question on Autotest test result status, so I'll try
to answer in that context. First, the framework itself gives you intentionally
limited test result status. If you want to save additional information about
your test, including say the mapping to POSIX 1003.1 codes, you can try to use
the test's "keyval" store for that. The "keyval" is both saved to a local file
and to the server's database (when that is used).

Avocado INSTRUMENTED tests, though, have a better separation of test setup and
execution, and a test can be SKIPPED during the setup phase. A few pointers:

 * https://github.com/avocado-framework/avocado/blob/master/examples/tests/skiponsetup.py
 * http://avocado-framework.readthedocs.org/en/latest/api/core/avocado.core.html#avocado.core.test.Test.skip

> 
> - Is there any road map for Autotest development? I'm wondering if
>   avocado's features (like per test SHA1 generation) would be ported to
>   Autotest?

Not really. Avocado's roadmap though, is accessible here:

https://trello.com/b/WbqPNl2S/avocado

> 
> 
> Thanks in advance for support.
> 
> 
> --
> Best regards,
> 
> Lukasz Majewski
> 
> Samsung R&D Institute Poland (SRPOL) | Linux Platform Group
> 
> _______________________________________________
> Autotest-kernel mailing list
> Autotest-kernel at redhat.com
> https://www.redhat.com/mailman/listinfo/autotest-kernel
>