[Avocado-devel] RFC: Avocado Job API
Cleber Rosa
crosa at redhat.com
Wed Apr 13 05:00:44 UTC 2016
On 04/12/2016 12:50 PM, Lukáš Doktor wrote:
> Dne 12.4.2016 v 17:04 Ademar Reis napsal(a):
>> On Tue, Apr 12, 2016 at 11:22:40AM +0200, Lukáš Doktor wrote:
>>> Dne 12.4.2016 v 02:31 Ademar Reis napsal(a):
>>>> On Mon, Apr 11, 2016 at 09:09:58AM -0300, Cleber Rosa wrote:
>>>>> Note: the same content on this message is available at:
>>>>>
>>>>> https://github.com/clebergnu/avocado/blob/rfc_job_api/docs/rfcs/job-api.rst
>>>>>
>>>>>
>>>>> Some users may find it easier to read with a prettier formatting.
>>>>>
>>>>> Problem statement
>>>>> =================
>>>>>
>>>>> An Avocado job is created by running the command line ``avocado``
>>>>> application with the ``run`` command, such as::
>>>>>
>>>>> $ avocado run passtest.py
>>>>>
>>>>> But most of Avocado's power is activated by additional command line
>>>>> arguments, such as::
>>>>>
>>>>> $ avocado run passtest.py --vm-domain=vm1
>>>>> $ avocado run passtest.py --remote-hostname=machine1
>>>>>
>>>>> Even though Avocado supports many features, such as running tests
>>>>> locally, on a Virtual Machine and on a remote host, only one those can
>>>>> be used on a given job.
>>>>>
>>>>> The observed limitations are:
>>>>>
>>>>> * Job creation is limited by the expressiveness of command line
>>>>> arguments, this causes mutual exclusion of some features
>>>>> * Mapping features to a subset of tests or conditions is not possible
>>>>> * Once created, and while running, a job can not have its status
>>>>> queried and can not be manipulated
>>>>>
>>>>> Even though Avocado is a young project, its current feature set
>>>>> already exceeds its flexibility. Unfortunately, advanced users are
>>>>> not always free to mix and match those features at will.
>>>>>
>>>>> Reviewing and Evaluating Avocado
>>>>> ================================
>>>>>
>>>>> In light of the given problem, let's take a look at what Avocado is,
>>>>> both by definition and based on its real world, day to day, usage.
>>>>>
>>>>> Avocado By Definition
>>>>> ---------------------
>>>>>
>>>>> Avocado is, by definition, "a set of tools and libraries to help with
>>>>> automated testing". Here, some points can be made about the two
>>>>> components that Avocado are made of:
>>>>>
>>>>> 1. Libraries are commonly flexible enough and expose the right
>>>>> features in a consistent way. Libraries that provide good APIs
>>>>> allow users to solve their own problems, not always anticipated by
>>>>> the library authors.
>>>>>
>>>>> 2. The majority of the Avocado library code fall in two categories:
>>>>> utility and test APIs. Avocado's core libraries are so far, not
>>>>> intended to be consumed by third party code and its use is not
>>>>> supported in any way.
>>>>>
>>>>> 3. Tools (as in command line applications), are commonly a lot less
>>>>> flexible than libraries. Even the ones driven by command line
>>>>> arguments, configuration files and environment variables fall
>>>>> short in flexibility when compared to libraries. That is true
>>>>> even
>>>>> when respecting the basic UNIX principles and features that
>>>>> help to
>>>>> reuse and combine different tools in a single shell session.
>>>>>
>>>>> How Avocado is used
>>>>> -------------------
>>>>>
>>>>> The vast majority of the observed Avocado use cases, present and
>>>>> future, includes running tests. Given the Avocado architecture and
>>>>> its core concepts, this means running a job.
>>>>>
>>>>> Avocado, with regards to its real world usage, is pretty much a job
>>>>> (and test) runner, and there's no escaping that. It's probable that,
>>>>> for every one hundredth ``avocado run`` commands, a different
>>>>> ``avocado <subcommand>`` is executed.
>>>>>
>>>>> Proposed solution & RFC goal
>>>>> ----------------------------
>>>>>
>>>>> By now, the title of this document may seem a little less
>>>>> misleading. Still, let's attempt to make it even more clear.
>>>>>
>>>>> Since Avocado is mostly a job runner that needs to be more flexible,
>>>>> the most natural approach is to turn more of it into a library. This
>>>>> would lead to the creation of a new set of user consumable APIs,
>>>>> albeit for a different set of users. Those APIs should allow the
>>>>> creation of custom job executions, in ways that the Avocado authors
>>>>> have not yet anticipated.
>>>>>
>>>>> Having settled on this solution to the stated problem, the primary
>>>>> goal of this RFC is to propose how such a "Job API" can be
>>>>> implemented.
>>>>
>>>> So in theory, given a comprehensive enough API it should be
>>>> possible to rewrite the entire "Avocado Test Runner" using the
>>>> Job API.
>>>>
>>>> Actually, in the future we could have multiple Test Runners (for
>>>> example in contrib/) with different feature sets or approaches at
>>>> creating and managing jobs.
>>>>
>>>> (in practice we will approach the problem incrementally, so this
>>>> should be a very long term goal)
>>>>
>>> Exactly, for example run on several machines, or run in parallel.
>>>
>>>>>
>>>>> Analysis of a Job Environment
>>>>> =============================
>>>>>
>>>>> To properly implement a Job API, it's necessary to review what
>>>>> influences the creation and execution of a job. Currently, a Job
>>>>> execution based on the current command line, is driven by, at least,
>>>>> the following factors:
>>>>>
>>>>> * Configuration state
>>>>> * Command line parameters
>>>>> * Active plugins
>>>>>
>>>>> The following subsections examines how these would behave in an API
>>>>> based approach to Job execution.
>>>>>
>>>>> Configuration state
>>>>> -------------------
>>>>>
>>>>> Even though Avocado has a well defined `settings`_ module, it only
>>>>> provides support for `getting the value`_ of configuration keys. It
>>>>> lacks the ability to set configuration values at run time.
>>>>>
>>>>> If the configuration state allowed modifications at run time (in a
>>>>> well defined and supported way), users could then create many types of
>>>>> custom jobs with that "tool" alone.
>>>>>
>>>>> Command line parameters
>>>>> -----------------------
>>>>>
>>>>> The need for a strong and predictable correlation between application
>>>>> builtin defaults, configuration keys and command line parameters is
>>>>> also a MUST for the implementation of the Job API.
>>>>>
>>>>> Users writing a custom job will very often need to set a given
>>>>> behavior that may influence different parts of the Job execution.
>>>>>
>>>>> Not only that, many use cases may be implemented simply by changing
>>>>> those defaults in the midst of the job execution.
>>>>>
>>>>> If users know how to map command line parameters into their
>>>>> programmable counterparts, advanced custom jobs will be created much
>>>>> more naturally.
>>>>
>>>> So if I understand it correctly, the configuration state
>>>> (configuration keys/values) and command line parameters are two
>>>> high-level approaches to the same thing, which we could probably
>>>> define as the job environment, job configuration, or job state.
>>> Currently the mapping is:
>>>
>>> config -> args -> job.args
>>>
>>> but we discussed that this is wrong, as some functions use
>>> `job.args`, some
>>> rely on `config`. So we need to rework this and either create yet
>>> another
>>> abstract entity which contains all, or update the `config` values and
>>> use
>>> that one everywhere, or accept only related params for each level (job
>>> accepts, test accepts, ...)
>>>
>>> From this RFC I understand (Cleber) wants to use `config` inside
>>> plugins/job/test, so the existing argument parser would have to be
>>> modified
>>> to map arguments not to processed-arguments, but rather to update the
>>> config
>>> values. Do I understand it correctly, Cleber?
>>
>> That's also my understanding. From a high level point of view,
>> they're all mechanisms to configure the environment. That's why I
>> would call it something like "job environment" or "job state"
>> or "job configuration".
>>
>> For the sake of simplicity, I'll use "job environment".
>>
>>>
>>>>
>>>> So, for example, there could be multiple ways to configure a job
>>>> environment:
>>>>
>>>> 1. Via configuration file (at least some parameters, as of today)
>>>> 2. Via command line options available in the Job API (at least
>>>> some parameters, as of today)
>>>> 3. Via fine-grained APIs (future)
>>>>
>>>> In the case of (2), we could have an opaque configuration API
>>>> available as part of the Job API which would allow avocado
>>>> command line options to be processed at runtime.
>>>>
>>>> For example, by having methods such as:
>>>>
>>>> job.process_args(argv) --> process "argv" and configure the
>>>> job environment according to the options provided by user.
>>>>
>>>> job.args_help() and job.args_usage() --> show command line
>>>> help and usage messages
>>> Could be (after we finish with everything else). I had a different
>>> idea in
>>> my head, please see the other email. (sorry I did not wanted to
>>> provide my
>>> feedback before reading other responses)
>>>
>>>>
>>>> In this case all the Job API "knows" is that
>>>> job.process_args(sys.argv) was called. Internally it could
>>>> implement things such as --multiplexer, --profilers, --wrappers,
>>>> --gdb, etc.
>>>>
>>>> Using job.process_args() would change the *job configuration* at
>>>> runtime.
>>> That's what the env.config.set() does, right? It prepares the env
>>> (parsed
>>> args) and it's instantiated during `run_test`.
>>
>> My idea is to provide it as a black-box (opaque implementation)
>> in the Job API. It can start small (say, just the basics such as
>> --wrappers would get implemented).
>>
>>>
>>>>
>>>>>
>>>>> Plugins
>>>>> -------
>>>>>
>>>>> Avocado currently relies exclusively on setuptools `entry points`_ to
>>>>> define the active plugins. It may be beneficial to add a secondary
>>>>> activation and deactivation mechanism, one that is locally
>>>>> configurable. This is a rather common pattern, and well supported by
>>>>> the underlying stevedore library.
>>>>>
>>>>> Given that all plugable components of Avocado are updated to adhere to
>>>>> the "new plugin" standard, some use cases could be implemented simply
>>>>> by enabling/disabling plugins (think of "driver" style plugins). This
>>>>> can be exclusively or in addition to setting the plugin's own
>>>>> configuration.
>>>>>
>>>>> Also, depending on the type of plugin, it may be useful to activate,
>>>>> deactivate and configure those plugins per job. Thus, as part of the
>>>>> Job state, APIs would allow for querying/setting plugins.
>>>>
>>>> The plugin interfaces in Avocado are not very mature yet and I
>>>> anticipate many discussions about what plugins are and which kind
>>>> of interfaces should be supported during the creation of the Job
>>>> API.
>>>>
>>>> My impression is that there are two different levels of APIs in
>>>> the context of plugins:
>>>>
>>>> 1. The abstract APIs available *to* plugins
>>>> 2. The specific APIs provided *by* plugins
>>> +1
>>>
>>>>
>>>> Avocado should properly expose (1) via the Job API when
>>>> necessary, while (2) should be available in a generic plugin
>>>> configuration API.
>>>>
>>>> Let me explain with an example using the multiplexer API (this is
>>>> hypothetical, because in practice that's not how the current
>>>> Avocado implementation works):
>>> I think multiplexer is not the best example here as it's not a
>>> plugin. It
>>> needs to be extracted and the API should be (finally) defined.
>>
>> The current status is not ideal. The multiplexer is part of
>> Avocado Core, but:
>>
>> - It has a plugin to implement "$ avocado multiplexer" (see
>> "$ avocado plugins"
>>
>> - Its design is supposed to be modular, abstract, optional and
>> easy to replace with a different implementation. That's
>> probably just in theory right now, but that's how I would like
>> to see it grow in the future: an implementation that delivers
>> what is defined in a more abstract "Variant API". And it
>> should be detached from the params handling, which is a
>> different thing (The Variants API would be just one of the
>> many APIs or components that make use of the params interface)
>>
>> So when I say it's a plugin, I mean in the sense of the latter
>> case.
>>
>>>
>>> The other plugins define the interface either by abstract class.
>>> Still we
>>> have 3 ways of invoking the plugins:
>>>
>>> 1. Stevedore: registers all plugins and skips the execution based on
>>> job.environment variable (subcommands, arguments, pre/post job plugin)
>>> 2. Proxy: We have a proxy and we manually add plugins to be used there
>>> (TestResultProxy, TestLoaders, ...)
>>> 3. We set one plugin to handle the execution - TestRunner
>>>
>>> The (1) can be tweaked to actually invoke only plugins enabled by
>>> job.environment variable by default (currently it's always executed).
>>> The
>>> (2) is currently hardcoded inside job (loaders use config) and allows
>>> multiple instances of the same plugin and I like that.
>>> And the (3) is handled by arguments.
>>>
>>> As I mentioned earlier, we should identify types of plugin and the
>>> test-related should probably be either passed to the test by the
>>> user, or be
>>> instantiated by the test based on the env variables.
>>
>> Thanks for the explanation about the different types of plugins
>> currently implemented. I still think the approach I described
>> would work:
>>
>> "Plugins are enabled, disabled and configured via job
>> environment"
>>
>>>
>>> Anyway back to the multiplexer:
>>>
>>>>
>>>> - The multiplexer is a plugin which turns a yaml file into a
>>>> set of variants (a variant is basically a dictionary). It has
>>>> a complex filtering mechanisms to control how the yaml file
>>>> is processed and generates variants with combinations from a
>>>> tree structure. That's what we're all familiar with in
>>>> Avocado (`run --multiplexer` and `avocado multiplexer`)
>>> It's iterative
>>
>> I see. When you say it's iterative, do you mean "a generator once
>> initialized", or do you mean "have its behavior changed while
>> being used?" Does it *have* to be that way?
>>
> I mean that in the end it produces generator of variants (one can re-run
> it as many times as he wants to).
>
Right... but I failed to understand if/how this impacts the discussion
at hand. Sorry if I'm missing something obvious.
> PS: That's actually the MuxTree object allowing that, and the MuxTree is
> another piece which should be part of the abstract params
> implementation. But that is not the scope of this RFC.
>
>>>
>>>>
>>>> - The multiplexer plugin should use a "Variant API", which is
>>>> way more abstract and generic: it simply provides a set of
>>>> variants (dictionaries) identified by a "Variant ID" (please
>>>> check the "Test ID" RFC for details on how it relates to
>>>> everything else in Avocado).
>>> Not dictionaries (that'd be way to strict). It allows `AvocadoParams`
>>> object. `AvocadoParams` is "a better dictionary" and is not related to
>>> multiplexer and is the API (currently inside `avocado.core`).
>>
>> I'm just trying to keep it simple at a higher level.
>>
> Sure
>
>>>
>>>>
>>>> There could be multiple plugins that use this "Variant API"
>>>> to deliver functionality similar to what the multiplexer
>>>> does. Which one is being used and how it's configured will
>>>> depend on the "job configuration" (or state, whatever you
>>>> prefer to call it).
>>> There are still missing pieces, but some are already there.
>>>
>>>>
>>>> So Avocado should provide the "Variant API" in the "Job API", but
>>>> the more complex and plugin-specific operations (like filtering
>>>> in this case), should be made available via a generic plugin API.
>>>>
>>>> At runtime, the "Variant API" will behave in ways dictated by how
>>>> the job environment is *configured*.
>>>>
>>>> Some pseudo-code examples are provided further down, when you
>>>> mention this specific multiplex use-case.
>>>>
>>>>>
>>>>> Use cases
>>>>> =========
>>>>>
>>>>> To aid in the design of an API that solves unforeseen needs, let's
>>>>> think about a couple of use cases. Most of these use cases are based
>>>>> on feedback already received and/or features already requested.
>>>>>
>>>>> Ordered and conditional test execution
>>>>> --------------------------------------
>>>>>
>>>>> A user wants to create a custom job that only runs a benchmark test on
>>>>> a VM if the VM installation test succeeds.
>>>>>
>>>>> Possible use case fulfillment
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> Pseudo code::
>>>>>
>>>>> #!/usr/bin/env python
>>>>> from avocado import Job
>>>>> from avocado.resolver import resolve
>>>>>
>>>>> job = Job()
>>>>>
>>>>> vm_install =
>>>>> resolve('io-github-autotest-qemu.unattended_install.cdrom.http_ks.default_install.aio_native')
>>>>>
>>>>> vm_disk_benchmark =
>>>>> resolve('io-github-autotest-qemu.autotest.bonnie')
>>>>>
>>>>> if job.run_test(vm_install).result == 'PASS':
>>>>> job.run_test(vm_disk_benchmark)
>>>>>
>>>>> API Requirements
>>>>> ~~~~~~~~~~~~~~~~
>>>>>
>>>>> 1. Job creation API
>>>>> 2. Test resolution API
>>>>> 3. Single test execution API
>>>>>
>>>>> Run profilers on a single test
>>>>> ------------------------------
>>>>>
>>>>> A user wants to create a custom job that only runs profilers for the
>>>>> very first test. Running the same profilers for all other tests may
>>>>> be useless to the user, or maybe consume too much I/O resources that
>>>>> would influence the remaining tests.
>>>>>
>>>>> Possible use case fulfillment
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> Avocado, has a configuration key that controls profilers::
>>>>>
>>>>> [sysinfo.collect]
>>>>> ...
>>>>> profiler = False
>>>>> ...
>>>>>
>>>>> By exposing the configuration state, the ``profiler`` key of the
>>>>> ``sysinfo.collect`` section could be enabled for one test, and
>>>>> disabled for all others. Pseudo code::
>>>>>
>>>>> #!/usr/bin/env python
>>>>> from avocado import Job
>>>>> from avocado.resolver import resolve
>>>>>
>>>>> job = Job()
>>>>> env = job.environment # property
>>>>>
>>>>> env.config.set('sysinfo.collect', 'profiler', True)
>>>>> job.run_test(resolve('build'))
>>>>>
>>>>> env.config.set('sysinfo.collect', 'profiler', False)
>>>>> job.run_test(resolve('benchmark'))
>>>>> job.run_test(resolve('stress'))
>>>>> ...
>>>>> job.run_test(resolve('netperf'))
>>>>>
>>>>> API Requirements
>>>>> ~~~~~~~~~~~~~~~~
>>>>>
>>>>> 1. Job creation API
>>>>> 2. Test resolution API
>>>>> 3. Configuration API
>>>>> 4. Single test execution API
>>>>>
>>>>> Multi-host test execution
>>>>> -------------------------
>>>>>
>>>>> Use case description
>>>>> ~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> User needs to run the same test on different platforms. User has
>>>>> hosts with the different platforms already setup and remotely
>>>>> accessible.
>>>>>
>>>>> Possible use case fulfillment
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> Avocado currently runs all tests in a job with a single runner. The
>>>>> `default runner`_ implementation is a local test runner. Other tests
>>>>> runners include the `remote runner`_ and the `vm runner`_.
>>>>>
>>>>> Pseudo code such as the following could implement the (serial, for
>>>>> simplicity) test execution in multiple different hosts::
>>>>>
>>>>> from avocado import Job
>>>>> from avocado.plugin_manager import require
>>>>> from avocado.resolver import resolve
>>>>>
>>>>> job = Job()
>>>>> print('JOB ID: %s' % job.unique_id)
>>>>> print('JOB LOG: %s' % job.log)
>>>>>
>>>>> runner_plugin = 'avocado.plugins.runner:RemoteTestRunner'
>>>>> require(runner_plugin)
>>>>>
>>>>> env = job.environment # property
>>>>> env.config.set('plugin.runner', 'default', runner_plugin)
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'username',
>>>>> 'root')
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'password',
>>>>> '123456')
>>>>>
>>>>> test = resolve('hardware_validation.py:RHEL.test')
>>>>>
>>>>> host_list = ['rhel6.x86_64.internal',
>>>>> ...
>>>>> 'rhel7.ppc64.internal']
>>>>>
>>>>> for host in host_list:
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'host', host)
>>>>> job.run_test(test)
>>>>>
>>>>> print('JOB STATUS: %s' % job.status)
>>>>>
>>>>> It's actually quite simple to move from a custom Job execution to a
>>>>> custom Job runner, example::
>>>>>
>>>>> #!/usr/bin/env python
>>>>> import sys
>>>>> from avocado import Job
>>>>> from avocado.plugin_manager import require
>>>>> from avocado.resolver import resolve
>>>>>
>>>>> test = resolve(sys.argv[1])
>>>>> host_list = sys.argv[2:]
>>>>>
>>>>> runner_plugin = 'avocado.plugins.runner:RemoteTestRunner'
>>>>> require(runner_plugin)
>>>>>
>>>>> job = Job()
>>>>> print('JOB ID: %s' % job.unique_id)
>>>>> print('JOB LOG: %s' % job.log)
>>>>> env = job.environment # property
>>>>> env.config.set('plugin.runner', 'default', runner_plugin)
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'username',
>>>>> 'root')
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'password',
>>>>> '123456')
>>>>>
>>>>> for host in host_list:
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'host', host)
>>>>> job.run_test(test)
>>>>>
>>>>> print('JOB STATUS: %s' % job.status)
>>>>>
>>>>> Which could be run as::
>>>>>
>>>>> $ multi hardware_validation.py:RHEL.test
>>>>> rhel{6,7}.{x86_64,ppc64}.internal
>>>>> JOB ID: 54cacfb42f3fa9566b6307ad540fbe594f4a5fa2
>>>>> JOB LOG:
>>>>> /home/<user>/avocado/job-results/job-2016-04-07T16.46-54cacfb/job.log
>>>>> JOB STATUS: AVOCADO_ALL_OK
>>>>>
>>>>> API Requirements
>>>>> ~~~~~~~~~~~~~~~~
>>>>>
>>>>> 1. Job creation API
>>>>> 2. Test resolution API
>>>>> 3. Configuration API
>>>>> 4. Plugin Management API
>>>>> 5. Single test execution API
>>>>>
>>>>> Current shortcomings
>>>>> ~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> 1. The current Avocado runner implementations do not follow the "new
>>>>> style" plugin standard.
>>>>>
>>>>> 2. There's no concept of job environment
>>>>>
>>>>> 3. Lack uniform definition of plugin implementation for "driver" style
>>>>> plugins.
>>>>>
>>>>> 4. Lack of automatic ownership of configuration namespace by plugin
>>>>> name.
>>>>>
>>>>>
>>>>> Other use cases
>>>>> ===============
>>>>>
>>>>> The following is a list of other valid use cases which can be
>>>>> discussed at a later time:
>>>>>
>>>>> * Use the multiplexer only for some tests.
>>>>
>>>> The example I promised while discussing how to handle the core
>>>> and plugin APIs with the example of the multiplexer:
>>>>
>>>> Using only the Variant API (a core concept):
>>>>
>>>> from avocado import Job
>>>> from avocado import resolver
>>>>
>>>> job = Job()
>>>> job.process_args(sys.argv)
>>>> print('JOB ID: %s' % job.unique_id)
>>>> print('JOB LOG: %s' % job.log)
>>>>
>>>> test = resolver("passtest.py")
>>>>
>>>> for v in env.variants():
>>>> ...
>>>> print "%s;%s" % (test.name, v.id)
>>>> test.variants.append(v)
>>>>
>>>> job.run_test(test)
>>>>
>>>> job.run_test(resolver("foo.py"))
>>>> job.run_test(resolver("bar.py"))
>>>>
>>>> print('JOB STATUS: %s' % job.status)
>>>> exit()
>>>>
>>>>
>>>> Now using the Multiplexer API (from the plugin):
>>>>
>>>> from avocado import Job
>>>> from avocado import resolver
>>>> from avocado.plugin_manager import require
>>>>
>>>> job = Job()
>>>> job.process_args(sys.argv)
>>>>
>>>> print('JOB ID: %s' % job.unique_id)
>>>> print('JOB LOG: %s' % job.log)
>>>> env = job.environment # property
>>>>
>>>> require('avocado.plugins.runner:Multiplexer')
>>>>
>>>> env.config.set("plugin.runner:Multiplexer", "reset")
>>>> env.config.set("plugin.runner:Multiplexer", "yaml-file",
>>>> "pass.yaml")
>>>> env.config.set("plugin.runner:Multiplexer", "filter-out", "hw/")
>>>> env.config.set("plugin.runner:Multiplexer", "filter-only", "linux/")
>>>>
>>>> test = resolver("passtest.py")
>>>>
>>>> for v in env.variants():
>>>> ...
>>>> print "%s;%s" % (test.name, v.id)
>>>> test.variants.append(v)
>>>>
>>>> job.run(test)
>>>>
>>>> env.config.set("plugin.runner.multiplexer", "reset")
>>>> env.config.set("plugin.runner.multiplexer", "yaml-file", "foo.yaml")
>>>>
>>>> test = resolver("foo.py")
>>>>
>>>> for v in env.variants():
>>>> ...
>>>> print "%s;%s" % (test.name, v.id)
>>>> test.variants.append(v)
>>>>
>>>> job.run(test)
>>>>
>>>> job.run_test(resolver("bar.py"))
>>> Btw this raised another concern in my head. Until now I thought
>>> `job.run_test` runs one job, but it's true that the resolver can return
>>> several tests (eg. resolve("virtio_console")). So shouldn't the API be:
>>>
>>> tests = resolver(...)
>>>
>>> and then:
>>>
>>> for test in tests:
>>> job.run_test(test)
>>
>> I think we've been avoiding discussing this kind of low level
>> details on purpose. So yes, job.run() is not a trivial API and
>> there are different ways of designing it.
>>
> OK, but the important piece here is, does it run only a single test,
> what it reports and what it requires.
>
It will run either a single or many tests (whatever we understand that
is the right choice). It will report whatever we believe should be
reported.
What I'm saying may be useless, but I'm trying to make a point: we can
all will go into all these details. At this point, we just have to look
at the general design and look design flaws... we'll eventually have to
choose our nuts and bolts.
I see there's already a lot of convergence. Do you guys see any of those
major or even minor design flaws?
Thanks yet again!
- Cleber.
>>>
>>> alternatively it'd have to be `job.run_tests(tests)`, which would return
>>> list of results including multiplexed test results. I could live with
>>> `run_tests` but I don't like to trigger all variants of the test.
>>> That would
>>> make:
>>>
>>> passtest.1 failtest.1 passtest2 failtest2
>>>
>>> harder to define. IMO params belongs to test, not all combinations.
>>> (see the
>>> modified example below)
>>>
>>
>> Yep, agree on that. I think we haven't discussed or focused at
>> this level of specificity of the API yet.
>>
> I know, it's one of the future items. It's just worth mentioning it here
> as it adds requirements.
>
>>>
>>>
>>>>
>>>> print('JOB STATUS: %s' % job.status)
>>>> exit()
>>> Please take a look at my plugins description in the other email. I think
>>> `multiplexer` (and other variants generating) plugins are a bit
>>> different
>>> and should be handled separately and even allow multiple instances of
>>> it and
>>> maybe it should be like that with other plugins too:
>>>
>>> job = Job()
>>> job.add(avocado.plugins.sysinfo.SysInfo())
>>> # Uses the defaults
>>> job.add(avocaod.plugins.json.JsonResults(filename="myfile"))
>>> # Uses defaults, but overrides the "filename"
>>> resolver.cleanup() # remove previously defined resolvers
>>> resolver.add(my.plugin())
>>> test = resolver("passtest")
>>> test_gdb = resolver("passtest")
>>> test_gdb.add(avocado.plugins.gdb.Gdb())
>>> mux1 = avocado.plugins.multiplexer.Multiplexer()
>>> # Get's the value from "--multiplex", or config value []
>>> mux2 = avocado.plugins.multiplexer.Multiplexer("MyFile.yaml")
>>> for variant in mux1:
>>> test.params = variant
>>> test_gdb.params = variant
>>> job.run_test(test)
>>> job.run_test(test_gdb)
>>> for variant in mux2:
>>> test.params = variant
>>> job.run_test(test)
>>>
>>> Eventually we could simplify things and allow some assignments
>>> directly on
>>> call, for example `job.run_test(test, params=None, ...)`.
>>>
>>
>> Your example appears to be aligned with what's being proposed
>> here. I think we haven't got to the point of discussing actual
>> API details yet (design of classes, methods, parameters,
>> inheritance vs composition, etc).
>>
> Yes, I like the RFC so I focused on the depth and possible issues.
>
>>>
>>> Note: By default avocado would load all plugins enabled in config.
>>>
>>
>> Like I said in the beginning, I think config is part of the "job
>> environment". Naturally, loading the avocado *config file* should
>> probably be the first step of setting up this job environment.
> Second if we allow parser, but yes...
>
>>
>> Thanks.
>> - Ademar
>>
>>>
>>>>
>>>>>
>>>>> * Use the gdb or wrapper feature only for some tests.
>>>>>
>>>>> * Run Avocado tests and external-runner tests in the same job.
>>>>
>>>> The same applies to these two use-cases: Runtime job
>>>> configuration will make gdb or wrappers be used when
>>>> job.run(test) is used.
>>>>
>>>> I think the above makes sense, although the specifics of
>>>> how the actual API will look like still needs to be properly
>>>> defined. Please let me know if we're headed in the same
>>>> direction.
>>>>
>>>> Thanks.
>>>> - Ademar
>>>>
>>>>>
>>>>> * Run tests in parallel.
>>>>>
>>>>> * Take actions based on test results (for example, run or skip other
>>>>> tests)
>>>>>
>>>>> * Post-process the logs or test results before the job is done
>>>>>
>>>>> Development Milestones
>>>>> ======================
>>>>>
>>>>> Since it's clear that Avocado demands many changes to be able to
>>>>> completely fulfill all mentioned use cases, it seems like a good idea
>>>>> to define milestones. Those milestones are not intended to set the
>>>>> pace of development, but to allow for the maximum number of real world
>>>>> use cases fulfillment as soon as possible.
>>>>>
>>>>> Milestone 1
>>>>> -----------
>>>>>
>>>>> Includes the delivery of the following APIs:
>>>>>
>>>>> * Job creation API
>>>>> * Test resolution API
>>>>> * Single test execution API
>>>>>
>>>>> Milestone 2
>>>>> -----------
>>>>>
>>>>> Adds to the previous milestone:
>>>>>
>>>>> * Configuration API
>>>>>
>>>>> Milestone 3
>>>>> -----------
>>>>>
>>>>> Adds to the previous milestone:
>>>>>
>>>>> * Plugin management API
>>>>>
>>>>> Milestone 4
>>>>> -----------
>>>>>
>>>>> Introduces proper interfaces where previously Configuration and Plugin
>>>>> management APIs were being used. For instance, where the following
>>>>> pseudo code was being used to set the current test runner::
>>>>>
>>>>> env = job.environment
>>>>> env.config.set('plugin.runner', 'default',
>>>>> 'avocado.plugins.runner:RemoteTestRunner')
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'username',
>>>>> 'root')
>>>>> env.config.set('plugin.runner.RemoteTestRunner', 'password',
>>>>> '123456')
>>>>>
>>>>> APIs would be introduced that would allow for the following pseudo
>>>>> code::
>>>>>
>>>>> job.load_runner_by_name('RemoteTestRunner')
>>>>> if job.runner.accepts_credentials():
>>>>> job.runner.set_credentials(username='root', password='123456')
>>>>>
>>>>> .. _settings:
>>>>> https://github.com/avocado-framework/avocado/blob/0.34.0/avocado/core/settings.py
>>>>>
>>>>> .. _getting the value:
>>>>> https://github.com/avocado-framework/avocado/blob/0.34.0/avocado/core/settings.py#L221
>>>>>
>>>>> .. _default runner:
>>>>> https://github.com/avocado-framework/avocado/blob/0.34.0/avocado/core/runner.py#L193
>>>>>
>>>>> .. _remote runner:
>>>>> https://github.com/avocado-framework/avocado/blob/0.34.0/avocado/core/remote/runner.py#L37
>>>>>
>>>>> .. _vm runner:
>>>>> https://github.com/avocado-framework/avocado/blob/0.34.0/avocado/core/remote/runner.py#L263
>>>>>
>>>>> .. _entry points:
>>>>> https://pythonhosted.org/setuptools/pkg_resources.html#entry-points
>>>>>
>>>>> --
>>>>> Cleber Rosa
>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>>
>>>>> _______________________________________________
>>>>> Avocado-devel mailing list
>>>>> Avocado-devel at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/avocado-devel
>>>>
>>>
>>
>
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
More information about the Avocado-devel
mailing list