[Avocado-devel] Multi Stream Test Support
Cleber Rosa
crosa at redhat.com
Fri Apr 7 16:53:21 UTC 2017
On 04/06/2017 05:01 AM, Lukáš Doktor wrote:
> Dne 5.4.2017 v 18:13 Cleber Rosa napsal(a):
>>
>>
>> On 04/05/2017 03:29 AM, Lukáš Doktor wrote:
>>> Dne 3.4.2017 v 15:48 Cleber Rosa napsal(a):
>>>> Note: this document can be view in rendered format at:
>>>>
>>>> https://github.com/clebergnu/avocado/blob/RFC_multi_stream_v1/docs/source/rfcs/multi_stream.rst
>>>>
>>>>
>>>>
>>>> ===========================
>>>> Multi Stream Test Support
>>>> ===========================
>>>>
>>>> Introduction
>>>> ============
>>>>
>>>> Avocado currently does not provide test writers with standard tools
>>>> or guidelines for developing tests that spawn multiple machines.
>>>>
>>>> Since these days the concept of a "machine" is blurring really
>>>> quickly, this proposal for Avocado's version of "multi machine" test
>>>> support is more abstract (that's an early and quick explanation of
>>>> what a "stream" means). One of the major goal is to be more flexible
>>>> and stand the "test" (pun intended) of time.
>>>>
>>>> This is a counter proposal to a previous RFC posted and discussed on
>>>> Avocado development mailing list. Many of the concepts detailed here
>>>> were introduced there:
>>>>
>>>> *
>>>> https://www.redhat.com/archives/avocado-devel/2016-March/msg00025.html
>>>> *
>>>> https://www.redhat.com/archives/avocado-devel/2016-March/msg00035.html
>>>> *
>>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00042.html
>>>> *
>>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00072.html
>>>>
>>>> Background
>>>> ==========
>>>>
>>>> The prior art that influences Avocado the most is Autotest. The
>>>> reason is that many of the Avocado developers worked on Autotest
>>>> before, and both share various common goals. Let's use Autotest,
>>>> which provided support for multiple machine test support as a basis
>>>> for comparison.
>>>>
>>>> Back in the Autotest days, a test that would spawn multiple machines
>>>> was a very particular type of test. To write such a test, one would
>>>> write a **different** type of "control file" (a server one). Then, by
>>>> running a "server control file" with an **also different** command
>>>> line application (``autoserv``, A.K.A. ``autotest-remote``), the
>>>> server control file would have access to some special variables, such
>>>> as the ``machines`` one. By using an **also different** type of job
>>>> implementation, the control file could run a given **Python function**
>>>> on these various ``machines``.
>>>>
>>>> An actual sample server control file (``server/samples/reboot.srv``)
>>>> for Autotest looks like this::
>>>>
>>>> 1 def run(machine):
>>>> 2 host = hosts.create_host(machine)
>>>> 3 host.reboot()
>>>> 4
>>>> 5 job.parallel_simple(run, machines)
>>>>
>>>> Line #5 makes use of the different (server) job implementation to run
>>>> function ``run`` (defined in line #1) in parallel on machines given by
>>>> the special variable ``machines`` (made available by the also special
>>>> ``autoserv`` tool).
>>>>
>>>> This quick background check shows two important facts:
>>>>
>>>> 1) The functionality is not scoped to tests. It's not easy to
>>>> understand
>>>> where a test begins or ends by looking at such a control file.
>>>>
>>>> 2) Users (and most importantly test writers) have to learn about
>>>> different tools and APIs when writing "multi machine" code;
>>>>
>>>> 3) The machines are defined outside the test itself (in the form of
>>>> arguments to the ``autoserv`` command line application);
>>>>
>>>> Please keep these Autotest characteristics in mind: Avocado's multi
>>>> stream test support goals will be presented shortly, and will detail
>>>> how they contrast with those.
>>>>
>>>> Avocado's Multi Stream Test Support Goals
>>>> =========================================
>>>>
>>>> This is a hopefully complete summary of our goals:
>>>>
>>>> 1) To not require a different type of test, that is, allow users
>>>> to *write* a plain `avocado.Test` while still having access to
>>>> multi stream goodies;
>>>>
>>>> 2) To allow for clear separation between the test itself and its
>>>> execution environment (focus here on the execution streams
>>>> environment);
>>>>
>>>> 3) To allow increased flexibility by abstracting the "machines"
>>>> concept into "excution streams";
>>>>
>>>> 4) To allow for even increased flexibility by allowing test writers to
>>>> use not only Python functions, but other representations of code to
>>>> be executed on those separate streams;
>>>>
>>>> Comparison with prior art
>>>> -------------------------
>>>>
>>>> When compared to the Autotest version of multiple machine support for
>>>> tests, Avocado's version is similar in that it keeps the separation of
>>>> machine and test definition. That means that tests written in
>>>> accordance to the official guidelines, will not contain reference to
>>>> the machines ("execution streams") on which they will have portions of
>>>> themselves executed on.
>>>>
>>>> But, a major difference from the Autotest version is that this
>>>> proposal attempts to provide the **same basic tools and test APIs** to
>>>> the test writers needing the multiple stream support. Of course,
>>>> additional tools and APIs will be available, but they will not
>>>> incompatible with traditional Avocado INSTRUMENTED tests.
>>>>
>>>> Core concepts
>>>> =============
>>>>
>>>> Because the first goal of this RFC is to set the general scope and
>>>> approach to Multi Stream test support, it's important to properly
>>>> describe each of the core concepts (usually abstractions) that will be
>>>> used in later parts of this document.
>>>>
>>>> Execution Stream
>>>> ----------------
>>>>
>>>> An *Execution Stream* is defined as a disposable execution environment,
>>>> different and ideally isolated from the main test execution
>>>> environment.
>>>>
>>>> A simplistic but still valid implementation of an execution
>>>> environment could be based on an Operating System level process.
>>>> Another valid implementation would be based on a lightweight
>>>> container. Yet another valid example could be based on a remote
>>>> execution interface (such as a secure shell connection).
>>>>
>>>> These examples makes it clear that level of isolation is determined
>>>> solely by the implementation.
>>>>
>>>> .. note:: Even though the idea is very similar, the term *thread* was
>>>> intentionally avoided here, so that readers are not led to
>>>> think
>>>> that the architecture is based on an OS level thread.
>>>>
>>>> An execution stream is the *"where"* to execute a "Block Of Code"
>>>> (which is the *"what"*).
>>>>
>>>> Block of Code
>>>> -------------
>>>>
>>>> A *Block of Code* is defined as computer executable code that can run
>>>> from start to finish under a given environment and is able to report
>>>> its outcome.
>>>>
>>>> For instance, a command such as ``grep -q vmx /proc/cpuinfo; echo $?``
>>>> is valid computer executable code that can run under various shell
>>>> implementations. A Python function or module, a shell command, or
>>>> even an Avocado INSTRUMENTED test could qualify as a block of code,
>>>> given that an environment knows how to run them.
>>>>
>>>> Again, this is the *what* to be run on a "Execution Streams" (which,
>>>> in turn, is *"where"* it can be run).
>>>>
>>>> Basic interface
>>>> ===============
>>>>
>>>> Without initial implementation attempts, it's unreasonable to document
>>>> interfaces at this point and do not expect them to change. Still, the
>>>> already existing understanding of use cases suggests an early view of
>>>> the interfaces that would be made available.
>>>>
>>>> Execution Stream Interface
>>>> --------------------------
>>>>
>>>> One individual execution stream, within the context of a test, should
>>>> allow its users (test writers) to control it with a clean interface.
>>>> Actions that an execution stream implementation should provide:
>>>>
>>>> * ``run``: Starts the execution of the given block of code (async,
>>>> non-blocking).
>>>> * ``wait``: Block until the execution of the block of code has
>>>> finished. ``run`` can be given a ``wait`` parameter that will
>>>> automatically block until the execution of code has finished.
>>>> * ``terminate``: Terminate the execution stream, interrupting the
>>>> execution of the block of code and freeing all resources
>>>> associated with this disposable environment
>>>>
>>>> The following properties should be provided to let users monitor the
>>>> progress and outcome of the execution:
>>>>
>>>> * ``active``: Signals with True or False wether the block of code
>>>> given on ``run`` has finished executing. This will always return
>>>> False if ``wait`` is used, but can return either True or False when
>>>> running in async mode.
>>>> * ``success``: A simplistic but precise view of the outcome of the
>>>> execution.
>>>> * ``output``: A dictionary of various outputs that may have been
>>>> created by ``run``, keyed by a descriptive name.
>>>>
>>>> The following properties could be provided to transport block of code
>>>> payloads to the execution environment:
>>>>
>>>> * ``send``: Sends the given content to the execution stream
>>>> environment.
>>> This is ambitious. Arbitrary code chunks have dependencies, be it
>>> binary, script, module, whatever... But sure, each implementation (Bash,
>>> PythonModule, PythonCode) can support various means to allow deps.
>>>
>>
>> Writing many different implementations is indeed ambitious. The major
>> goal here, though, is to define a sane interface that works for the
>> basic and most urgent use cases while still not locking out doors to
>> future needs. Basically, making sure we don't loose the investment here
>> and rewriting large portions of Avocado just because we now find that
>> running, say, Ansible playbooks, in different streams is now a very
>> important thing for our tests.
>>
>>> Anyway for the python script I'd recommend looking at
>>> `avocado-vt/virttest/remote_commander` which is a generic interface to
>>> execute tasks remotely to get the sight of complexity.
>>>
>>
>> Yep, I'm aware of it, but thanks for the pointer since it does indeed
>> make sense to keep it in mind.
>>
>>>>
>>>> Block of Code Interface for test writers
>>>> ----------------------------------------
>>>>
>>>> When a test writer intends to execute a block code, he must choose from
>>>> one of the available implementations. Since the test writer must know
>>>> what type of code it's executing, the user inteface with the
>>>> implementation
>>>> can be much more flexible.
>>>>
>>>> For instance, suppose a Block Of Code implementation called
>>>> ``PythonModule`` exists. This implementation would possibly run
>>>> something like
>>>> ``python -m <modulename>`` and collect its outcome.
>>>>
>>>> A user of such an implementation could write a test such as::
>>>>
>>>> from avocado import Test
>>>> from avocado.streams.code import PythonModule
>>>>
>>>> class ModuleTest(Test):
>>>> def test(self):
>>>> self.streams[1].run(PythonModule("mymodule",
>>>> path=["/opt/myproject"]))
>>>>
>>>> The ``path`` interface in this example is made available and supported
>>>> by the ``PythonModule`` implementation alone and will not be used the
>>>> execution stream implementations. As a general rule, the "payload"
>>>> should be the first argument to all block of code implementations.
>>>> Other arguments can follow.
>>>>
>>>> Another possibility related to parameters is to have the Avocado's own
>>>> test parameters ``self.params`` passed through to the block of code
>>>> implementations, either all of them, or a subset based on path. This
>>>> could allow for example, a parameter signaling a "debug" condition to
>>>> be passed on to the execution of the block of code. Example::
>>>>
>>>> from avocado import Test
>>>> from avocado.streams.code import PythonModule
>>>>
>>>> class ModuleTest(Test):
>>>> def test(self):
>>>> self.streams[1].run(PythonModule("mymodule",
>>>> path=["/opt/myproject"],
>>>> params=self.params))
>>>>
>>>> Block of Code Interface for Execution Stream usage
>>>> --------------------------------------------------
>>>>
>>>> Another type of public interface, in the sense that it's well known
>>>> and documented, is the interface that Execution Stream implementations
>>>> will use to interact with Block of Code implementations. This is not
>>>> intended to be used by test writers, though.
>>>>
>>>> Again, it's too early to define a frozen implementation, but this is
>>>> how it could look like:
>>>>
>>>> * ``send_self``: uses the Execution Stream's ``send`` interface to
>>>> properly
>>>> populate the payload or other necessary assets for its execution.
>>>> * ``run``: Starts the execution of the payload, and waits for the
>>>> outcome
>>>> in a synchronous way. The asynchronous support is handled at the
>>>> Execution
>>>> Stream side.
>>>> * ``success``: Reports the positive or negative outcome in a
>>>> simplistic but precise way.
>>>> * ``output``: A dictionary of various outputs that may be generated by
>>>> the
>>>> execution of the code. The Execution Stream implementation may
>>>> merge this
>>>> content with its own ``output`` dictionary, given an unified view of
>>>> the
>>>> output produced there.
>>>>
>>>> Advanced topics and internals
>>>> =============================
>>>>
>>>> Execution Streams
>>>> -----------------
>>>>
>>>> An execution stream was defined as a "disposable execution
>>>> environment". A "disposable execution environment", currently in the
>>>> form of a fresh and separate process, is exactly what the Avocado
>>>> test runner gives to a test in execution.
>>>>
>>>> While there may be similarities between the Avocado Test Process
>>>> (created by the test runner) and execution streams, please note that
>>>> the execution streams are created *by* one's test code. The following
>>>> diagram may help to make the roles clearer::
>>>>
>>>> +-----------------------------------+
>>>> | Avocado Test Process | <= created by the test runner
>>>> | +-------------------------------+ |
>>>> | | main execution stream | | <= executes your `test*()`
>>>> method
>>>> | +-------------------------------+ |
>>>> | | execution stream #1 | | <= initialized on demand by
>>>> one's
>>>> | | ... | | test code. utilities to
>>>> do so
>>>> | | execution stream #n | | are provided by the
>>>> framework
>>>> | +-------------------------------+ |
>>>> +-----------------------------------+
>>>>
>>>> Even though the proposed mechanism is to let the framework create the
>>>> execution lazily (on demand), the use of the execution stream is the
>>>> definitive trigger for its creation. With that in mind, it's accurate
>>>> to say that the execution streams are created by one's test code
>>>> (running on the "main execution stream").
>>>>
>>>> Synchronous, asynchronous and synchronized execution
>>>> ----------------------------------------------------
>>>>
>>>> As can be seen in the interface proposal for ``run``, the default
>>>> behavior is to have asynchronous executions, as most observed use
>>>> cases seem to fit this execution mode.
>>>>
>>>> Still, it may be useful to also have synchronous execution. For that,
>>>> it'd be a matter of setting the ``wait`` option to ``run``.
>>>>
>>>> Another valid execution mode is synchronized execution. This has been
>>>> thoroughly documented by the previous RFCs, under sections named
>>>> "Synchronization". In theory, both synchronous and asynchronous
>>>> execution modes could be combined with a synchronized execution, since
>>>> the synchronization would happen among the execution streams
>>>> themselves. The synchronization mechanism, usually called a "barrier",
>>>> won't be given too much focus here, since on the previous RFCs, it was
>>>> considered a somehow agreed and understood point.
>>>>
>>>> Termination
>>>> -----------
>>>>
>>>> By favoring asynchronous execution, execution streams need to also
>>>> have a default behavior for handling termination of termination
>>>> of resources. For instance, for a process based execution stream,
>>>> if the following code is executed::
>>>>
>>>> from avocado import Test
>>>> from avocado.streams.code import shell
>>>> import time
>>>>
>>>> class MyTest(avocado.Test):
>>>> def test(self):
>>>> self.streams[0].run(shell("sleep 100"))
>>>> time.sleep(10)
>>>>
>>>> The process created as part of the execution stream would run for
>>>> 10 seconds, and not 100 seconds. This reflects that execution streams
>>>> are, by definition, **disposable** execution environments.
>>>>
>>>> Execution streams are thus limited to the scope of one test, so
>>>> implementations will need to terminate and clean up all associated
>>>> resources.
>>>>
>>>> .. note:: based on initial experiments, this will usually mean that a
>>>> ``__del__`` method will be written to handle the cleanup.
>>> I'd suggest the runner should after the test execution run:
>>>
>>> for stream in test_instance.streams:
>>> stream.close()
>>>
>>> which should explicitly take care of closing the streams (and
>>> terminating it's processes) as `__del__` might not be executed on all
>>> occasions and I'd suggest doing this during/after `tearDown`.
>>>
>>
>> Right. Let's keep that in mind. At the end of the day, it's an
>> implementation level detail, and as long as the cleanup is tested and
>> works, we're good to go.
>>
>>>>
>>>> Avocado Utility Libraries
>>>> -------------------------
>>>>
>>>> Based on initial evaluation, it looks like most of the features
>>>> necessary
>>>> to implement multi stream execution support can be architected as a set
>>> to implement multi stream execution support can be __designed__ as a set
>>>
>>>> of utility libraries.
>>>>
>>>> One example of pseudo code that could be possible with this design::
>>>>
>>>> from avocado import Test
>>>> from avocado.streams import get_implementation
>>>> from avocado.streams.code import shell
>>>>
>>>> class Remote(Test):
>>>>
>>>> def test_filtering(self):
>>>> klass = get_implementation("remote")
> This is mainly what I dislike about the `get_implementation`. The
> `get_implementation` should IMO get all the available info about the
> stream and based on that it should decide which implementation is it. I
> don't see a point in having dict-like-method to map "remote" to
> "RemoteStream" class, I can do that by `getattr(streams, type)`.
>
Not really. The idea is to have a well defined interface, and different
(plugable) implementations. `get_implementation()` would know about
valid registered implementations that don't live in the
"avocado.streams" (or similar) namespace, but there are still valid
implementations.
> So as mentioned later I'd suggest creating `get_stream` method which
> would guess the most appropriate implementation based on the input and
> report already initialized stream.
>
This can be yet another method, but it doesn't conflict with the idea of
`get_implementation()`.
And by "guess the most appropriate" you probably mean having an ordered
list of default implementations, right? I think we should avoid
heuristics and complex guesswork here.
>>>
>>> Well this is not really scalable. Imagine that in one execution you want
>>> to use local, in next remote and then docker container. In your example
>>> you'd have to change the source code of the test. How about this:
>>>
>>> get_implementation(params.get("first_stream"))
>>> get_implementation(params.get("second_stream"))
>>>
>>
>> Your suggestion is really a "how to write an Avocado test". Yes, we
>> absolutely should use parameters in tests, and that's why the "klass"
>> parameters come from parameters. The use of "remote" here is just to
>> make things clearer with regards to how one would refer to
>> implementations (by name).
>>
>>> where:
>>>
>>> first_stream = "localhost"
>>> second_stream = "test:123456 at 192.168.122.10"
>>> third_stream = "docker://create=yes,image=fedora,remove=always"
>>> fourth_stream = "libvirt://create=no,domain=test_machine,start=yes"
>>> ...
>>>
>>
>> These URLs are just painful to read IMO.
>>
>>> Another option would be to allow `path` of the Avocado Test `params` and
>>> the class would get the details from the provided params+path. The cons
>>> would be it'd be hard to use anything but `params` for that:
>>>
>>> streams:
>>> first:
>>> type: localhost
>>> second:
>>> hostname: 192.168.122.10
>>> user: test
>>> password: 123456
>>> third:
>>> type: docker
>>> create: yes
>>> ...
>>>
>>
>> I like this structure better.
>>
>>> get_implementation(self.params, "/streams/first/*") => would use
>>> `params.get(..., "/streams/first/*")` to get all necessary parameters
>>>
>>
>> The idea of `get_implementation` is pretty simple: get an Execution
>> Stream implementation by its name. What you're describing here (with
>> regards to getting the "right" parameters and instantiating/activating
>> the execution stream easily/automatically) is exactly what I propose
>> later.
>>
> This is not really about the automatic creation. As for today it's
> impossible to map tests-to-variants and I saw some people are actually
> writing tests which depends on certain path so they can supply different
> params to different tests:
>
First of all, the design shouldn't be limited by current limitations
that can be addressed (although it's a very good thing to mention and
keep track of them).
Furthermore, I'd say the problem here is not mapping tests to variants
at all. We're using YAML (yaml_to_mux) simply as a didactic way of
setting variables at specific paths, and that is the essence here.
Maybe I'm missing something, but I don't see a reason why the same
"*/avocado/stream/*" parameters couldn't be available to all tests as
"Default params":
http://avocado-framework.readthedocs.io/en/48.0/TestParameters.html#default-params
And still be overridden by specific tests.
If you agree that's achievable but currently limited or non functional
because of addressable limitations, then I believe we're fine, and just
have another work item to deliver multi stream test support.
> boot:
> timeout: 10
> check:
> timeout: 1
> shutdown:
> timeout: 5
>
> and in boot.py:
>
> timeout = self.params.get("timeout", "*/boot/*")
>
> in check.py:
>
> timeout = self.params.get("timeout", "*/check/*")
>
> ...
>
>
> So the comment above tried to describe how one would create descriptions
> of streams on different places and be still able to get them in various
> tests from different locations. Imagine in test1.py:
>
> get_implementation(self.params, "*/test1/streams/first/*")
>
> in test2.py:
>
> get_implementation(self.params, "*/test2/streams/server/*")
>
> and so on. The automatic way would be shared for all tests, while this
> would allow the same but using different locations. Anyway this is
> really painful and I'd prefer using the single-line definition, or the
> list of tuples as described later...
>
>>>> if klass is not None:
>>>> stream = klass(host=self.params.get("remote_hostname"),
>>>>
>>>> username=self.params.get("remote_username")
>>>>
>>>> password=self.params.get("remote_password"))
>>>> cmd = "ping -c 1 %s" %
>>>> self.params.get("test_host_hostname")
>>>> stream.run(shell(cmd))
>>> I do like the rest of the example (only the klass would be already
>>> initialized by the `get_implementation`.
>>>
>>
>> This conflicts with the idea of "utility library to be used by test
>> developer that wants full control of the individual creation and use of
>> the execution streams". I mean, if some implementations are made
>> available at the `avocado.utils` namespace, then they could even be
>> referred to directly by module/name.
> Yes, by
> `get_implementation("docker://create=yes,image=fedora,remove=always") or
> `get_implementation(host=host, username=username, ...)`, which returns
> already initialized object of the detected type. Alternatively the user
> would import directly the specific class and use
> `DockerStream(image="fedora")`.
>
> Again, the better name for `get_implementation` in this sense would be
> `get_stream`.
>
I think you missed the point about `get_implementation()`. Again,
`get_stream()` could be implemented on top of it. And remember
that this is the "verbose, lengthy and boilerplate-full" method that we
hope most users won't need or use.
I'd really focus most of our energy on giving the majority of users (the
majority of use cases) a very easy to use set of tools.
>>
>> Again, since we're talking about making whatever makes sense in the
>> utility namespace, there could a function such as
>> "get_stream_parameters(name)" that would return the parameters to an
>> Execution Stream, that is:
>>
>> stream_name = "server"
>> path = "/avocado/streams/%s/*" % stream_name
>> impl = self.params.get("type", path=path)
>> stream = get_implementation(impl)(**get_stream_parameters(path))
>>
>
> Actually it should probably be `self.streams.get_stream()` as we do want
> to register it in the `Streams` to get it cleaned automatically. This
> also works for libraries as you'd first initialize `Streams` object and
> then do the `get_stream()`, `close()` and so on directly on the
> `Streams` object.
>
Can you explain a use case where you'd write code to manually create a
stream using the object that is supposed to contain the ready to use
streams? If you need to manually create it, then it's probably a good
idea for you to manually destroy it. Of course we could add mechanisms
to register streams created outside of `self.streams`, but I don't feel
the urge to do so unless proven wrong.
Anyway, this seems a corner case to me, and the most difficult and
important task here is to try to get the "recommended" way right, so
that most multi stream tests are really straightforward to read and write.
>>>>
>>>> Please note that this is not the intended end result of this proposal,
>>>> but
>>>> a side effect of implementing it using different software layers. Most
>>>> users should favor the simplified (higher level) interface.
>>>>
>>>> Writing a Multi-Stream test
>>>> ===========================
>>>>
>>>> As mentioned before, users have not yet been given tools **and
>>>> guidelines** for writing multi-host (multi-stream in Avocado lingo)
>>>> tests. By setting a standard and supported way to use the available
>>>> tools, we can certainly expect advanced multi-stream tests to become
>>>> easier to write and then much more common, robust and better supported
>>>> by Avocado itself.
>>>>
>>>> Mapping from parameters
>>>> -----------------------
>>>>
>>>> The separation of stream definitions and test is a very important goal
>>>> of this proposal. Avocado already has a advanced parameter system, in
>>> of this proposal. Avocado already has __an__ advanced parameter
>>> system, in
>>>
>>>> which a test received parameters from various sources.The most common
>>>> way of passing parameters at this point is by means of YAML files, so
>>>> these will be used as the example format.
>>> Well this might be quite hard to understand, how about just saying:
>>> "Avocado supports test parametrisation via Test Parameters system and
>>> the most common way is to use a YAML file by using `yaml_to_mux`
>>> plugin."
>>>
>>>>
>>>> Parameters that match a predefined schema (based on paths and node
>>>> names) will be by evaluated by a tests' ``streams`` instance
>>>> (available as ``self.streams`` within a test).
>>>>
>>>> For instance, the following snippet of test code::
>>>>
>>>> from avocado import Test
>>>>
>>>> class MyTest(Test):
>>>> def test(self):
>>>> self.streams[1].run(python("import mylib; mylib.action()"))
>>>>
>>>> Together with the following YAML file fed as input to the parameter
>>>> system::
>>>>
>>>> avocado:
>>>> streams:
>>>> - 1:
>>>> type: remote
>>>> host: foo.example.com
>>> This is currently not supported by our yaml parser as any dictionary is
>>> mapped to multiplex structure and I'm not sure it'd be possible (in a
>>> sane manner) to treat dict inside lists differently. Anyway as I
>>> mentioned earlier we could use:
>>>
>>
>> Oops, I may have used an incorrect syntax or idea (or both).
>>
>>> avocado:
>>> streams:
>>> 1: ssh://foo.example.com
>>>
>>> or:
>>>
>>> avocado:
>>> streams:
>>> 1:
>>> type: remote
>>> host: foo.example.com
>>>
>>
>> This one looks good IMO. I'm all in for more explicitly naming **when**
>> you go to the lengths of defining them.
>>
> Yes, plus we do use the OrderedDicts so we can still let people use:
>
> self.streams[0:4]
>
> together with:
>
> self.streams["server"]
>
> just beware the:
>
> self.streams["1"]
>
> is different then:
>
> self.streams[1]
>
> and the node-names (therefor even the 1: in streams) becomes string in
> Avocado, so it actually works well and in the example above the:
>
> self.streams["1"] == self.streams[0]
>
Nice, looks like OrderedDicts, or an even "smarter" version, could be
the basis or our slicing support.
>>> or:
>>>
>>> avocado:
>>> streams:
>>> - type: remote
>>> host: foo.example.com
>>>
>>> Another thing is I'd probably prefer names to ints so "1" or "server" or
>>> "worker1" etc, which goes nicely with the first 2 examples. The last
>>> example goes well with indexes, but it starts with 0 (which would be my
>>> recommendation anyway if we decided to go with indexes).
>>>
>>
>> I think I mentioned somewhere "if only integers"... I actually share
>> your fondness of names. I did not say it explicitly, but I think we can
>> support both, as the slicing examples make a lot of sense to me.
>>
>> But, the slicing examples can be expanded to its own dialect, such as
>> supporting regexes. For instance, `self.streams["client-\d+"]` makes a
>> lot of sense IMO.
>>
> Yes, this makes sense and is quite simple to do...
>
>>>>
>>>> Would result in the execution of ``import mylib; mylib.action()``
>>>> in a Python interpreter on host ``foo.example.com``.
>>>>
>>>> If test environments are refered to on a test, but have not been
>>>> defined
>>> If test environments are __referred__ to on a test, but have not been
>>> defined
>>>
>>
>> OK, thanks.
>>
>>>> in the outlined schema, Avocado's ``streams`` attribute implementation
>>>> can use a default Execution Stream implementation, such as a local
>>>> process
>>>> based one. This default implementation can, of course, also be
>>>> configured
>>>> at the system and user level by means of configuration files, command
>>>> line
>>>> arguments and so on.
>>>>
>>>> Another possibility is an "execution stream strict mode", in which no
>>>> default implementation would be used, but an error condition would be
>>>> generated. This may be useful on environments or tests that are
>>>> really tied to their execution stream types.
>>> I'd solve this by supporting `__len__` where `len(self.streams)` should
>>> report number of defined streams.
>>>
>>> Note the number of defined streams changes based on how many streams are
>>> defined __OR__ used by the test. So:
>>>
>>> avocado:
>>> streams:
>>> - 0:
>>> - 1:
>>>
>>>
>>> len(self.streams) => 2
>>> self.streams[5].run(cmd)
>>> len(self.streams) => 6
>>>
>>> where the streams 2-5 are the default streams.
>>>
>>
>> I have mixed feelings here. In my understanding, all of the streams are
>> created on demand, that is, when they're used. A configuration that
>> defines thousands of them will not cause an empty test (think of
>> `passtest.py`) to initialize them.
> Yes, that is the point. `__len__` reports the "defined", not necessarily
> "initialized" streams.
>
> Adding streams dynamically is definitely must-have-feature and one note
> the code above would actually only have the `streams[5]` initialized,
> the rest would wait for the first usage.
>
OK, I'm fine with that.
>>
>> But, the meaning of `len(self.streams)`, that is, defined or
>> initialized, is something we can further discuss later.
>>
> Sure, another possibility would be to support `streams.defined_streams`
> but I think the `__len__` would make more sense. It's similar to a list:
>
> streams = list(params.get("streams", "/avocado/*")
> len(streams) == 2
> streams.extend([2,3,4,5])
> len(streams) == 6
>
Sure, as long as we document that `len()` returns the defined streams,
it makes perfect sense to me.
>>>>
>>>> Intercommunication Test Example
>>>> -------------------------------
>>>>
>>>> This is a simple example that exercises the most important aspects
>>>> proposed here. The use case is to check that different hosts can
>>>> communicate among themselves. To do that, we define two streams as
>>>> parameters (using YAML here), backed by a "remote" implementation::
>>>>
>>>> avocado:
>>>> streams:
>>>> - 1:
>>>> type: remote
>>>> host: foo.example.com
>>>> - 2:
>>>> type: remote
>>>> host: bar.example.com
>>>>
>>>> Then, the following Avocado Test code makes use of them::
>>>>
>>>> from avocado import Test
>>>> from avocado.streams.code import shell
>>>>
>>>> class InterCommunication(Test):
>>>> def test(self):
>>>> self.streams[1].run(shell("ping -c 1 %s" %
>>>> self.streams[2].host))
>>>> self.streams[2].run(shell("ping -c 1 %s" %
>>>> self.streams[1].host))
>>>> self.streams.wait()
>>>> self.assertTrue(self.streams.success)
>>> Brainstorming here, how about letting `wait` raise exception when it
>>> fails unless we use `wait(ignore_failure)`. The exception would contain
>>> all the information so it'd be THE exception which failed the test?
>>>
>>
>> Yep, this is a valid question. I think the answer will depend on how
>> much we want the **test** result to be bound to what happens on the
>> streams. Right now it's obvious that I decided to keep them pretty much
>> separate.
>>
> Sure, the more I think about it I also share your vision.
>
Good!
>>> As for the `streams.success`, I guess it'd be a property, which would go
>>> through all streams results, and report `any(_.failure for _ in
>>> self.streams)`, right?
>>>
>>
>> Exactly.
>>
>>>>
>>>> The ``streams`` attribute provide a aggregated interface for all the
>>>> streams.
>>>> Calling ``self.streams.wait()`` waits for all execution streams (and
>>>> their
>>>> block of code) to finish execution.
>>>>
>>>> Support for slicing, if execution streams names based on integers only
>>>> could
>>>> be added, allowing for writing tests such as::
>>>>
>>>> avocado:
>>>> streams:
>>>> - 1:
>>>> type: remote
>>>> host: foo.example.com
>>>> - 2:
>>>> type: remote
>>>> host: bar.example.com
>>>> - 3:
>>>> type: remote
>>>> host: blackhat.example.com
>>>> - 4:
>>>> type: remote
>>>> host: pentest.example.com
>>>>
>>>> from avocado import Test
>>>> from avocado.streams.code import shell
>>>>
>>>> class InterCommunication(Test):
>>>> def test(self):
>>>> self.streams[1].run(shell("ping -c 1 %s" %
>>>> self.streams[2].host))
>>>> self.streams[2].run(shell("ping -c 1 %s" %
>>>> self.streams[1].host))
>>>> self.streams[3].run(shell("ping -c 1 %s" %
>>>> self.streams[1].host))
>>>> self.streams[4].run(shell("ping -c 1 %s" %
>>>> self.streams[1].host))
>>>> self.streams.wait()
>>>> self.assertTrue(self.streams[1:2].success)
>>>> self.assertFalse(self.streams[3:4].success)
>>> As mentioned earlier I'd prefer names to indexes, anyway I see the
>>> indexes useful as well. How about supporting a name or index?
>>>
>>
>> Yep, also thought of that.
>>
>>> As for the slices, I'd prefer list-like slice to stream-like slices as
>>> it'd be more natural to me to interact with a list of individual streams
>>> rather than a Stream object with a limited subset of streams. Anyway
>>> that's a matter of taste and I can definitely live with this as well.
>>>
>>
>> See my previous comments.
>>
>>> Now about this example, it's really limited. Again you are hard-coding
>>> the scenario and changing it is really complicated. I'd prefer something
>>> like:
>>>
>>> self.streams[0].run(server_cmd)
>>> self.streams[1:].run(contact_server_cmd)
>>> self.assertTrue(self.streams.success)
>>>
>>
>> Real tests will probably (hopefully) use better (symbolic) names. The
>> goal here is to focus on the mechanisms, which on yours and on my
>> version are identical.
>>
>>>>
>>>> Support for synchronized execution also maps really well to the
>>>> slicing example. For instance, consider this::
>>>>
>>>> from avocado import Test
>>>> from avocado.streams.code import shell
>>>>
>>>> class InterCommunication(Test):
>>>> def test(self):
>>>> self.streams[1].run(shell("ping -c 60 %s" %
>>>> self.streams[2].host)
>>>> self.streams[2].run(shell("ping -c 60 %s" %
>>>> self.streams[1].host))
>>>> ddos = shell("ddos --target %s" self.streams[1].host)
>>>> self.streams[3:4].run(ddos, synchronized=True)
>>>> self.streams[1:2].wait()
>>>> self.assertTrue(self.streams.success)
>>>>
>>>> This instructs streams 1 and 2 to start connectivity checks as soon as
>>>> they **individually** can, while, for a full DDOS effect, streams 3
>>>> and 4 would start only when they are both ready to do so.
>>> OK so this is about before-start-synchronisation. Well again, I'm not
>>> much fond of boundling the streams so I'd prefer allowing to define the
>>> workload (which returns when the workload is ready), trigger it (which
>>> triggers it and reports immediately) and then wait for it. The
>>> difference in usage is:
>>>
>>>
>>> self.streams[3].run(ddos, stopped=True)
>>> self.streams[4].run(ddos, stopped=True)
>>> self.streams[3].start()
>>> self.streams[4].start()
>>>
>>> The result is the same (unless you create processes per each stream and
>>> in `self.streams[3:4]` you use signals to synchronize the execution) but
>>> it allows greater flexibility like synchronizing other tasks then just
>>> streams...
>>>
>>> Or we can create methods `establish(cmd)`, `start()` and `wait()` which
>>> might better describe the actions.
>>
>> I think you missed my point here. Streams #3 and #4, in my example,
>> wait for *each other*. Using a mechanism such as barriers.
>>
> Oh you mean that the `synchronized=True` means it'd make a `barrier`
> mechanism available between those two streams and you'd be able to use
> this barrier from the streams? That would be rather limiting as you
> might want to create multiple barriers between several streams. I think
> you should re-describe your synchronization here as well as the
> structure changed a bit...
>
> In my RFC I basically passed the information about barriers by test
> params, which is obviously not available here as the Code could be
> anything. Anyway a barrier requires:
>
> 1. name
Name on my example is implicit, as it can be created from the unique
names of the stream themselves.
> 2. server
Also implicit, as most use cases should be able to use the default
barrier server provided by `self.streams`. To be more precise, an also
on-demand barrier server running alongside the avocado test process
(what you refer to as as "main", later).
> 3. number of clients
>
Sure, but I don't see how **code** run on streams have to care about
this. The idea is that they will only have their `run()` methods called
by the respective execution streams when the barrier is lifted (say, all
clients have checked in).
The point here is that individual stream execution in a "slice" will be
**synchronized** from the outside of the **block of code**. How the
synchronization is done, is a different matter.
> When creating synchronized streams by a slice, you have all of these
> available, but it might not be always like that and you might want to
> define different groups of barriers, for example:
>
> ssh:
>
> start_ssh()
> barrier("ssh_started", worker, 2)
> barrier("worker_finished", main, 3)
>
These 3 lies, IIUC, are "block of code" content, right?
> http:
>
> start_http()
> barrier("http_started", worker, 2)
> barrier("worker_finished", main, 3)
Same here.
>
> worker:
>
> barrier("ssh_started", worker, 2)
> barrier("http_started", worker, 2)
> connect_to_both()
> barrier("worker_finished", main, 3)
>
Same here for these 4 lines, right?
> Here you see 3 streams, they use 3 different barriers, 2 are
> stream->stream and 1 is stream->main. You might ask in this simple
> example why would you care about where the synchronization happens as
> you could always use `main` (which is the main Avocado host) but imagine
> testing private network between the 2 interfaces which might result in
> something like:
>
> stop_eth0()
> # now main can't communicate with this stream
> start_ssh()
> barrier("ssh_started", worker, 2)
> barrier("worker_finished", worker, 2)
> start_eth0()
> # now this stream reports the status to main
>
> where temporarily the connection stream->main is impossible. This was
> supported and used in Autotest.
>
The use case for temporarily disconnected streams shouldn't be a
problem. For instance, "self.streams" (let's call it, for now, the
"stream master") can just pass different configurations to the
individual execution streams instances. For example:
...
self.streams["worker[1-9]"].run(code,
synchronized=True,
sync_server="worker1")
do_something_else()
...
The "stream master" will run *all of this* asynchronously, just as
before. It will initialize stream "worker1", (on demand, just as
before) and will tell it to have a sync server ready.
After the confirmation that the sync_server is ready on "worker1", the
remaining **execution streams** on the "worker[1-9]" slice will be
initialized and will be given information about who their sync server
is. Up to this point, no **(block of) code** has run, only the
execution stream machinery.
Once the "stream master" has verified that all streams have checked in
to the sync_server, it can `run()` them. The **block of code** to be
run can include all kinds of temporary network shutdown, or even
permanent ones. Doing something like this should be fine:
...
self.streams["worker[1-9]"].run(code,
synchronized=True,
sync_server="worker1")
while time.time() < timeout:
if not self.streams["worker[1-9]"].success:
self.log("waiting for the workers to report success")
...
Until the stream master can get success on all individual execution
streams, which in turn can verify the outcome of the **block of code**,
it will not report `success`. Execution stream implementations can be
resilient to temporary connectivity loss, simply by implementing
adequate timeouts. A non-recoverable scenario could be signaled by an
exception, that could translate into a test ERROR.
Sidetracking: this actually raises a point about whether all execution
streams implementations should implement a sync server, or if that
should be optional. If optional, an error such as
"SyncServerNotCapable" could be raised. Test writers using
`synchronized=True`, would of course have to choose a capable server.
> Now the tricky part, how to support it. In the original RFC I used
> params to do that, which works well but is quite demanding on the test
> writer (especially with dynamic number of streams). I thought about it
> and one way is to create a streams representation and pass it to all
> streams. Now this is simple for pre-defined streams, but for ad-hock
> streams it would require either post-execution synchronization (which is
> available for python processes, but unless we do some master-slave
> architecture for bash or other scripts, it's not possible), or we could
> allow streams preparation before the actual execution (something like
> the `establish(cmd)` and `start()`. Let's use the 3 tests I described
> earlier and write a main test for it, deliberately using ad-hock streams
> rather then pre-defined configuration:
Talking about support for other languages: the execution stream
implementation doesn't care about what it's going to run. If it
supports synchronization, it will set it up and run the **block of
code** `run()` afterwards.
>
> no_workers = len(self.streams) # all pre-defined streams are workers
> assert no_workers > 0, "Not enought workers to run this test"
>
> # define servers
> server_types = params.get("server_types", default=["ssh", "http"])
> for type in server_types:
> self.streams.get_stream(stream_definition % type)
> for stream in self.streams:
> stream.start() # which forwards the definition of all defined
> streams
>
> where the streams definition is something like `Streams` only with
> static data, allowing:
>
>
> streams.barrier(name, no_clients) => contact main test to enter the
> barrier
> streams[0].barrier(name, no_clients) => contact stream0 to enter
> the barrier
> len(streams["client.*"]) => get number of "client.*" matching
> streams (to be able to calculate how many clients should enter the barrier)
> ...
>
> Basically from the Code worker I expect it to establish the connection
> on `get_stream()`, get some info like `ip_addr` and so on. Then on
> `start()` I expect it to get the important info about streams and send
> that along with the command where it is executed (for BASH as env
> variables or a file stored somewhere) so the executed stream knows what
> is going on around.
>
> Quite important note I don't expect all of this to do for all streams as
> quite often no synchronization is needed, sometimes stream->main
> synchronization is enough and sometimes all_streams->main
> synchronization is needed. All of these can be treated as a special
> situations and could be way simpler to develop, anyway the ultimate goal
> should be the free-flexibility in synchronization between all streams
> and our solution should scale to that (yes, as in my previous PR we can
> leave it up to the test writers and only allow passing params to the
> streams - env-variables, dict, ...).
>
> PS: The stream->main sync should actually be as simple as waiting till
> the main test reports `self.streams[0].barrier("name")`, which puts
> additional requirements on the main test's code, but is really simple to
> develop. The only reason why we need worker->worker (or
> worker->some_server) synchronization is when we start disabling network.
>
I absolutely agree. Even though we should make "hard things possible",
I believe our main goal here is to make the "simple things easy". Why?
Because test writers can resort to **any** other code pattern or
external libraries to implement very specific stuff. The sweet spot
here is how we can easily allow the vast majority of tests to be easily
written, and be run reliably and consistently.
>>>
>>>>
>>>> Feedback and future versions
>>>> ============================
>>>>
>>>> This being an RFC, feedback is extremely welcome. Also, exepect new
>>>> versions
>>>> based on feedback, discussions and further development of the ideas
>>>> initially
>>>> exposed here.
>>>>
>>> Overall I like it, I don't really see it as a counter-proposal as I
>>> think they are pretty similar, only you defined the block-of-code to be
>>> more generic and refined the high-level interface.
>>>
>>
>> I guess it's a good thing that you like it and that this is just a
>> "only" (small and simple) kind of thing. I guess my goal was attained
>> at some level :).
>>
> Sure, I becoming to like even the `Streams` object with slicing support,
> basically I see this as an `avocado.utils` library which would be
> (lazily) available as `self.streams` in each test along with a
> first-time initialization of streams from `self.params` (during the lazy
> init).
>
> I'm a bit afraid regarding the Code blocks as I think simple bash
> scripts can be better dealt with by aexpect-like libraries, python ones
> with python's distributed computing (yes, in python2 there is no barrier
> mechanism, but we can add utils for that), but for the complex tasks I
> see the benefit.
>
> I see some issues regarding the synchronization and flexibility as you
> are focused on rather simple example. I remember some multi-host tests
> from Autotest which disabled/hanged networks during the test executions
> and I believe we ought to support it as well. We can leave the params
> passing on the test writers, but maybe we can pass the definitions
> automatically.
>
Cool. I'll work on a v2 that includes more examples on "synchronization
among streams" which would support disconnected use cases.
>>> As for the implementation I think the `self.stream` description and all
>>> the details about comes a bit too early. I'd start with the low level
>>> interface, which is a library allowing to create `Streams()`, defines
>>> `Code()` and `Stream()` objects independently on Avocado and later when
>>> we see the usage I'd standardized the usage and embedded it into the
>>> `Test` class. Anyway I know we don't share this vision and I'm fine with
>>> doing it the other way around as the result is the same, only we might
>>> found some limits later which might be hard to solve in the current
>>> schema. But based on what I know from virtualization this (together with
>>> the barrier synchronization) should be enough to support the tests we
>>> know from Autotest which is a good start.
>>>
>>
>> Sure. I mentioned that it's impossible to define a "freeze" of any sort
>> on the interfaces. Still, talking about them, helps to shape the
>> features they may have and how they'd be used.
>>
>> But the most important thing here is your acknowledgment that this seems
>> to fit the needs we have on virtualization tests.
>>
>>> Last remark regarding my and your RFC is that I deliberately defined the
>>> block-of-code like (not as) Test, because executing scripts is possible
>>> nowadays via `aexpect`, `Remoter`, `remote_commander` or other standard
>>> python libraries, but combining existing tests and get not just the
>>
>> Right, but in no standard and supported way. This is one of the major
>> goals here. The very first line in this RFC is:
>>
>> "Avocado currently does not provide test writers with standard tools
>> or guidelines for developing tests that spawn multiple machines."
>>
>>> executed results but also to gather remote environment and so on, that I
>>> see beneficial. Anyway if I understand this implementation correctly
>>> it'd be possible to create `AvocadoTest` inherited from `Code` which
>>> would allow such executions and the `Streams` could support environment
>>> gathering (optionally) and that is all I care about. For simpler stuff
>>
>> Right.
>>
>>> I'd simply use `aexpect` (as I personally like it a lot) and I know of
>>> people who use `remote_commander` to synchronize and distribute tasks
>>> across multiple machines in `avocado-vt` so I assume they'll stick to
>>> their working solution as well. I see the `Streams` as a library to
>>
>> Our goal is to come up with useful innovations that will motivate users
>> to adopt them, so this is a bit negative. Unless you don't really
>> believe in the value of this proposal, I would expect the opposite
>> attitude.
>>
>>> support complex tasks, not just a simple command execution, even though
>>> some abstraction might be useful.
>>>
>>
>> Can you describe complex tasks? I imagine that the ideas your have are
>> based on "interacting with a remote shell/machine/console/application"
>> as aexpect and other tools allow, right?
>>
> Complex tasks are basically all the written tests with setUp test and
> tearDown (note the stream does not know about them, it's just the inner
> structure which results to success/failure including cleaned
> environment). One benefit in being able to run these is the
> code-reusability and the other is uniform way of writing units.
>
> You can do most of this via some interactive shell (as a lot of
> framework does) but when the test itself starts having many lines
> embedding it into another one makes it harder to follow.
>
>>> Actually now after the summary I noticed that what I'd really need is
>>> either the `AvocadoTest`-like command to be able to combine basic tests
>>> into a complex scenarios and then I'd need a unique way of interacting
>>> with different streams. So what I'd probably need is an
>>> `aexpect`-concentrator which would allow me to ask for a session based
>>> on the description:
>>>
>>> aexpect.RemoteShellSession(url)
>>>
>>> where `url` is something like:
>>>
>>> "localhost"
>>> "test:123456 at 192.168.122.10"
>>> "docker://create=yes,image=fedora,remove=always"
>>> "libvirt://create=no,domain=test_machine,start=yes"
>>>
>>> which would establish the connection (creating the container/vm first if
>>> asked for) and than I'd interact with it as with other
>>> `aexpect.ShellSession` (therefor not just a single command, but full
>>> expect-like behavior)
>>>
>>
>> OK, this matches my previous comment. Yes, what you're describing is
>> probably not something this RFC contemplates, but at least parts of
>> could be common for both use cases.
>>
> Yep, I agree. I'm saying that for the simple blocks of code (like shell
> script) this could be actually a better technology. For a binary
> execution I'd chose Remoter. But for the complex tasks which includes
> several lines of preps and some cleanups I'd definitely want something
> like this RFC. This note here was more a brainstorm about a possible
> improvement of aexpect (and Remoter-like implementations) which should
> probably share the same definition like streams in order to be able to
> learn it once and re-use it everywhere (yes, I do think all of these
> could share eg. the connection-code, container-creation-code and so on..)
>
Agreed.
> Btw for the `Avocado-test`-like code-block (if developed as it can
> simply be executed as s standalone script) I'd like to see sharing the
> sources with the remote-test-runner, so most of the Avocado sources
> would actually benefit from a clear reusable interfaces.
>
Absolutely.
>>> This last note is probably outside the scope of multi-stream tests and
>>> could (hopefully should) be implemented in parallel to serve different
>>> purpose. Anyway with this in mind I don't see much point in having
>>> `Bash` or `PythonModule`-like code-blocks in parallel test execution and
>>> I'd only focus on the full-blown complex parallel tasks.
>>>
>>
>> Sure, something like `self.interactives` available at the test level,
>> working similar to the streams could be a nice addition here.
>>
> Well they are utils so I don't see a point in propagating them in
> `Test`. Anyway yes, we should share the code of this RFC and then expand
> our documentation to help users to pick the right technology for their
> use-case and they all should be similar to use with some benefits
> regarding their usages.
>
>>> Anyway, hopefully this feedback is understandable, I have been writing
>>> it for 2 days so feel free to ask for some hints ...
>>>
>>> Lukáš
>>>
>>
>> Yes, I think I understood all your points. Let me know if my response
>> was clear enough.
> Yes, as mentioned earlier we are (probably) on the same boat, the main
> concern here is regarding the synchronization...
>
Cool. I'll work on a v2, which will hopefully makes the other use cases
clearer.
>>
>> And this for the feedback!
>>
>
On the previous email, I meant *thanks* for the feedback, so I'll
correct/repeat myself here: thanks for the feedback!
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170407/20db71f9/attachment.sig>
More information about the Avocado-devel
mailing list