[Avocado-devel] Avocado 101 questions

[Fajun Chen]

Hi Lukas,

Sorry for the late reply. I was pulled into other work. Please see my
comments below.

On Wed, Jan 30, 2019 at 11:23 AM Lukáš Doktor <ldoktor at redhat.com> wrote:

> >>>
> >>>    1. How to define the test sequence in yaml/json? An example file
> with
> >>>    test references and test parameters would be very helpful.
> >>
> >> Simplest is to enumerate the tests on cmdline. We do support filters so
> >> people can specify tests_methods, classes, files or directories, where
> the
> >> order is preserved (alphabetical for files/dirs and according to
> location
> >> for test_methods/classes).
> >> https://avocado-framework.readthedocs.io/en/latest/Loaders.html
> >>
> >> For finer-granularity we have `yaml_loader`
> >>
> https://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
> >> that allows specifying tests in a yaml file. Basic usage is:
> >>
> >> ```
> >> !mux
> >> 1:
> >>     test_reference: passtest.py
> >> 2:
> >>     test_reference: failtest.py
> >> ```
> >>
> >> but the format allows to define loaders, loader parameters, changing
> some
> >> avocado arguments and even directly modifying test params. I'm using it
> to
> >> run various external_runner tests together with avocado-vt tests using
> >> different params in a single job.
> >>
> >>
> > What's the syntax to set test params for a test (not test file) in the
> yaml
> > loader file. For instance, how to pass in sleep_cycles and sleep_length
> > into the test in sleeptenmin.py:
> >       test_reference: SleepTenMin.test
> >                  # how to set test params for the test?
> >
>
> The yaml_loader iterates through the resulting params and looks for
> "test_reference" and other key-words (see
> https://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html
> for the complete list). In the end it adds all the resulting tests (as
> multiple ones can match) to the job attaching the current params slice to
> it. What it means is that each test gets the "test_reference" and all other
> variables of the current slice defined in the yaml file):
>
> # loader.yaml
> !mux
> test1:
>     test_reference: passtest.py
>     foo: bar
> test2:
>     test_reference: passtest.py
>     foo: baz
>
> $ avocado --show all run loader.yaml --dry-run
> ...
> avocado.test: INIT 1-passtest.py:PassTest.test;-e630
> ...
> avocado.test: Test params:
> avocado.test: /run/test1:test_reference ==> passtest.py
> avocado.test: /run/test1:foo ==> bar
> ...
> avocado.test: INIT 2-passtest.py:PassTest.test;-e630
> ...
> avocado.test: Test params:
> avocado.test: /run/test2:test_reference ==> passtest.py
> avocado.test: /run/test2:foo ==> baz
>
> (note, the `--dry-run` is one of the neat features where you can see all
> globally-available params)



>

This works as suggested. Thanks for the tips

>

> Does the yaml loader support test discovery by tags?
> >
>
> It is possible to discover directories in file-loader, but unfortunately
> it is not possible to override filter-by-tag per variant. What I mean is
> you can only specify one `filter-by-tag` globally via `--filter-by-tags` on
> the command line and it will be applied to all yaml_loader references.
>
> This is because filter-by-tags feature had been implemented as global
> filter on the resulting suite, but your request seems valid and we should
> probably re-evaluate and make it part of the loaders. It would require some
> slightly deeper changes to the loaders as each of them would have to do the
> filtering during the discovery (which shouldn't be that hard as all plugins
> "should" inherit from the base TestLoader so implementation there should
> serve as the basis for all of them).
>
> Anyway let's discuss it on release/planning meeting
> https://trello.com/c/poL0jWIi/1487-rfc-move-filter-by-tag-to-loaders
> (feel free to join, but don't feel obligated, we'll discuss it anyway).
> >
>

Test planning and test execution are done by different teams in our test
process. Would be nice to expand the yaml loader to include more
functionality so it can be a one-stop shop for test planning.  For
instance, in addition to having filter-by-tag capability, could it support
other command-line options such as failfast in the yaml file? The test
sequencing capability provided by yaml loader seems to be very limited. For
instance, how to define the loop count of a test? how to define the
dependency or the conditional execution of tests in a sequence?

>
> >>>    2. Could the tests in a test job be executed sequentially and in
> >>>    predefined order?
> >>
> >> See (1), there is a defined order.
> >>
> >>>    3. Could we skip subsequent tests when a test fails?
> >>
> >> Unfortunately not available as a feature yet. There is a workaround to
> use
> >> `teststmpdir` to create a file and in `setUp` phase of following test to
> >> check for it's presence and we do have an RFC to implement Job API, that
> >> should allow dynamic job definition
> >> https://trello.com/c/hRmShGIp/671-rfc-job-api but there is no ETA yet.
> >>
> >>>    4. How to abort test job upon critical test failure?
> >>
> >> The same situation. We don't have a framework support, one can use the
> >> same workaround to skip all remaining tests. Anyway this feature should
> be
> >> fairly simple to develop and I see a value in it. We could simply add
> >> CRITICAL test status and in such case abort the execution.
> >>
> >> Note if you mean to interrupt testing after any failed test, you can use
> >>
> https://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#interrupting-the-job-on-first-failed-test-failfast
> >
> >
> > This fail fast feature can probably meet our needs assuming it supports
> the
> > tests from either command-line or yaml loader. Thanks.
> >
>
> The source of the tests does not matter, that's a different level. The
> fail-fast feature is very simple and basically just interrupts the job
> whenever there is a failure. It won't look whether it's first, last or the
> only test in the queue. (it works the same way as python-unittest's
> --fail-fast)
>

This matches what I thought. Thanks for the confirmation.


> >>
> >>
> >>>    5. How to add custom code to handle exceptions from tests?
> >>
> >> Directly to the tests. Not sure what exactly you have in mind, but you
> can
> >> have your own utils library and use it in the tests.
> >>
> >> Note if you mean test failure vs. test error, we do have some decorators
> >>
> https://avocado-framework.readthedocs.io/en/latest/WritingTests.html#turning-errors-into-failures
> >> to declare certain exceptions as test failure, rather than anonymous
> test
> >> errors.
> >>
> >
> > We would like to catch the exceptions from tests and take some actions
> such
> > as collecting error logs. Should this custom code raise the exception
> again
> > so Avocado can handle it?   Would be nice to hide this from tests.
> Thinking
> > about creating a subclass of avocado.test and add  custom exception
> handler
> > there.
> >
>
> This is definitely encouraged way, especially when you have many custom
> things to do in your testing. We even have a documentation on how to do
> that:
> https://avocado-framework.readthedocs.io/en/latest/SubclassingAvocado.html
>
> and our sub-project Avocado-vt might serve as an example of a very complex
> test (including custom loader, which might not be necessary in your case):
> https://github.com/avocado-framework/avocado-vt
>
> another example of simpler sub-class is the fedora-modularity:
> https://github.com/fedora-modularity/meta-test-family/blob/master/moduleframework/avocado_testers/avocado_test.py
> but disclaimer: "We have nothing to do with that project"
>
>
Thanks for the references. Will spend time for a proof-of-concept
implementation

>
> >>
> >>>    6. rsync is used to upload test logs to remote server, which is
> >> started
> >>>    after test completion. How to upload test logs incrementally for
> long
> >>>    running test before it completes
> >>
> >> Avocado supports streamline results like tap or journal. We don't have a
> >> plugin to upload individual test results (or even to stream the files as
> >> the test runs), but we could give you pointers to develop such plugin.
> Note
> >> if you only need to log files after test execution, the plugin would be
> as
> >> simple as:
> >>
> >>     def test_end(self, test):
> >>         self.publish(test.job.logdir)
> >>
> >> where publish would simply rsync the logdir to your server. Logging
> files
> >> as the test goes would be probably a bit harder and we'd have to think
> >> about it. One way is to add logger, but that way you only get logs and
> not
> >> the produced files. Other would be to have plugin that would create a
> >> thread and keep syncing the logdir. Again, should not be that hard, we
> >> could give you pointers but not sure how significant is it currently
> for us.
> >>
> >
> > Thanks for the ideas. We could start with rsync at test end and add
> feature
> > for periodical sync later for long running tests.
> >
>
> Sure, btw if the job-log is enough for you, you can use `avocado --show
> test` to show the test log in console (or `--show all`).
>
> And speaking of loggers, you can define a certain logger in your tests and
> use it to mark important steps. We used to use something like that
> previously in Avocado-vt and with that log enabled in output (`avocado
> --show app,context`) would show something like:
>
> JOB ID     : d5eb807b581736ffa923c4305287af82a014718d
> JOB LOG    :
> /home/medic/avocado/job-results/job-2019-01-30T19.02-d5eb807/job.log
>  (1/1) io-github-autotest-qemu.boot: |
> Starting VM/
> SSH to VM-
> Shutting down VM\
> PASS (19.26 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
> CANCEL 0
> JOB TIME   : 20.17 s
> JOB HTML   :
> /home/medic/avocado/job-results/job-2019-01-30T19.02-d5eb807/results.html
>
> Documentation about this feature is here:
> https://avocado-framework.readthedocs.io/en/latest/WritingTests.html#advanced-logging-capabilities


This is very similar to the Python logger configuration which we used for
internal testing. Good to know that Avocado has something similar or
better.


>
>
> >>
> >>>    7. How to run multiple test jobs in parallel?
> >>
> >> Unfortunately current runner allows only a single job per execution.
> >> Anyway running multiple avocado instances at the same time is not a
> >> problem. This is another goal of the Job API RFC mentioned in answer (3)
> >>
> >>>    8. The test logs from low level lib modules are captured in job.log
> >>>    instead of in test specific directories.Just wonder if there're
> hooks
> >> in
> >>>    place to customize the logger so more test data are reported in test
> >>>    specific logs.
> >>
> >> I'm not sure I understand this properly. There is the `$results/job.log`
> >> file that should contain some job logs + all logs from tests. Then there
> >> are the per-test `$results/test-results/*/debug.log` logs that contain
> the
> >> part of `job.log` since the test started until the test ends. It should
> not
> >> contain more, nor less (with one exception and that is interrupted test,
> >> where extra messages might get injected out-of-place). If that's not the
> >> case than we have a bug and would like to ask you for a reproducer.
> >>
> >
> > Thanks for the clarification. I'll double check to see if the logs are
> > organized per design. Will let you know.
> >
>

Use test_env.py in Avocado examples/tests directory as an example. The logs
from log_std_io() function is in both job.log and per-test debug.log.
However, if the function is moved to a separate module like this
>cat log_io.py

import logging
import os

log = logging.getLogger(__name__)

def log_std_io(name, std_io):
    log.debug('%s:', name.upper())
    log.debug(' sys.%s: %s', name, std_io)
    log.debug(' sys.%s is a tty: %s', name, std_io.isatty())
    if hasattr(std_io, 'fileno'):
        log.debug(' fd: %s', std_io.fileno())
        log.debug(' fd is tty: %s', os.isatty(std_io.fileno()))
    else:
        log.debug(' fd: not available')
        log.debug(' fd is a tty: can not determine, most possibly *not* a
tty')

and test_env.py calls this as external function, the logs from this
function is in job.log, but NOT in per-test debug.log. Is this per design?

>
> >>
> >> Also note that the test is a python script, so "evil" test might
> re-define
> >> it's loggers. Anyway even "evil" test should not affect the main Avocado
> >> loggers, nor the other executed tests. We do have a strict boundary
> between
> >> runner and each tests (they are actually a different processes) as in
> other
> >> frameworks we had problems with tests affecting other tests.
> >>
> >
> > I understood the rationale behind running each test in a separate process
> > now. This does pose problem for manual/interactive testing where we need
> > user input during a test. How to support this in Avocado?
> >
>
> Well, we try to prevent that as much as we can. The STDIN is always closed
> in Avocado tests as we believe tests should be fully automated and we have
> aexpect https://github.com/avocado-framework/aexpect to interact with
> programs under testing.
>
> Still for debugging purposes it is possible to interact with the program
> via sockets. I'm using `pydev` to debug/single-step test execution and we
> also have support for gdb that interrupts the test and provides shell
> script to connect to the process when it reaches breakpoint. Similarly you
> can write your plugins to allow manual testing, but I'd rather you
> reconsidered that option and used random timeouts and aexpect....
>
>
This is one of the main road blockers on our list. We can't eliminate human
interaction in our test process. Could we pipe test subprocess standin and
standout through test runner? Would need out-of-band communication path if
not.


> >>
> >> All in all, Avocado is fairly established project now, we still have the
> >> JobAPI coming, persistent test (allowing reboot during test execution)
> and
> >> multi-stream (multi-host) testing in progress. Current LTS version is
> >> primarily python2-compatible, but for a longish period the master
> switched
> >> to python3 by default with backward compatibility and we are about to
> >> release it as another LTS (LTS releases are maintained for about 1.5
> years,
> >> there is a overlap with previous LTS to allow smooth transition and we
> do
> >> have guidelines on what changed between versions and how to adjust).
> While
> >> evaluating definitely checkout `avocado.utils.*` where we have several
> >> useful utils to speedup writing tests and also the Aexpect sub-project
> >> https://github.com/avocado-framework/aexpect that is inspired by Expect
> >> language and inherits from pexpect project, but is improved (it was
> >> actually forked because of slow pexpect development) and could be useful
> >> even when you decide not to use Avocado itself.
> >>
> >
> > Thanks again for the info. Will explore these utils. I'm impressed with
> the
> > modular architecture of Avocado. Kudos to the development team! I have
> some
> > items on my wish list:
>
> Thank you, there are still some burdens to be refactored but overall I do
> like Avocado.
>
> > - Web front end
>
> There is the https://github.com/avocado-framework/avocado-server but it
> was just an experiment and we don't really had usage for that. But feel
> free to take that over.
>

I have looked at this a while ago. Could use this as a starting point if we
go this route. Have you guys considered porting autotest web UI over?


>
> > - Host management - Add tags to test hosts
>
> The same story, RH heavily depends on Beaker and Jenkins and these do that
> well. No point in re-inventing the wheel. I meant to write a guide on how
> to integrate Avocado with Jenkins but never had time to put things
> together:
> https://trello.com/c/9zTcfa0v/904-add-avocado-based-solution-for-jenkins-solutions-for-python
>
> > - Test scheduling -  Allow test planning based on test tags and host tags
> > These may sound familiar to your guys from Autotest project. Would be
> great
> > to know your thoughts for Avocado.
> >
>
> Well, we only have one tagging mechanism, but it allows list support, etc.
> It should be possible to configure your scheduler (Jenkins in my example)
> to insert host tags to filter-by-tags cmdline along with your custom
> filter-by-tags, which might result in what you're looking for. What I mean
> is:
>
> 1. Configure your hosts in Jenkins to export HOST_TAGS=... (for example
> "arch:x86_64 os:RHEL.8")
> 2. Configure the job to add parameter JOB_TAGS
> 3. In the Jenkins job use "avocado run ... --filter-by-tags $HOST_TAGS
> $JOB_TAGS ... -- $TESTS_REPOSITORY/"
>
> What do you think? (note the categorized tags documentation is here:
> https://avocado-framework.readthedocs.io/en/latest/WritingTests.html#using-further-categorization-with-keys-and-values
> )
>
>
Thanks for the ideas. Jenkins integration is on my todo list. Will work on
it and provide updates later.


>
> PS: Release/planning meeting is next Monday, feel free to join to discus
> your questions/prioritization live (or let's continue on ML, whatever suits
> you)
>
>  Would like to join if my schedule allows. When is the meeting and how to
join?

Thanks,
Fajun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20190203/2737966f/attachment.htm>