[Avocado-devel] RFC: Multi-host tests
Cleber Rosa
crosa at redhat.com
Wed Mar 30 13:52:47 UTC 2016
On 03/30/2016 09:31 AM, Lukáš Doktor wrote:
> Dne 29.3.2016 v 20:25 Cleber Rosa napsal(a):
>>
>>
>> On 03/29/2016 04:11 AM, Lukáš Doktor wrote:
>>> Dne 28.3.2016 v 21:49 Cleber Rosa napsal(a):
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Cleber Rosa" <crosa at redhat.com>
>>>>> To: "Lukáš Doktor" <ldoktor at redhat.com>
>>>>> Cc: "Amador Pahim" <apahim at redhat.com>, "avocado-devel"
>>>>> <avocado-devel at redhat.com>, "Ademar Reis" <areis at redhat.com>
>>>>> Sent: Monday, March 28, 2016 4:44:15 PM
>>>>> Subject: Re: [Avocado-devel] RFC: Multi-host tests
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Lukáš Doktor" <ldoktor at redhat.com>
>>>>>> To: "Ademar Reis" <areis at redhat.com>, "Cleber Rosa"
>>>>>> <crosa at redhat.com>,
>>>>>> "Amador Pahim" <apahim at redhat.com>, "Lucas
>>>>>> Meneghel Rodrigues" <lookkas at gmail.com>, "avocado-devel"
>>>>>> <avocado-devel at redhat.com>
>>>>>> Sent: Saturday, March 26, 2016 4:01:15 PM
>>>>>> Subject: RFC: Multi-host tests
>>>>>>
>>>>>> Hello guys,
>>>>>>
>>>>>> Let's open a discussion regarding the multi-host tests for avocado.
>>>>>>
>>>>>> The problem
>>>>>> ===========
>>>>>>
>>>>>> A user wants to run netperf on 2 machines. To do it manually he does:
>>>>>>
>>>>>> machine1: netserver -D
>>>>>> machine1: # Wait till netserver is initialized
>>>>>> machine2: netperf -H $machine1 -l 60
>>>>>> machine2: # Wait till it finishes and report store the results
>>>>>> machine1: # stop the netserver and report possible failures
>>>>>>
>>>>>> Now how to support this in avocado, ideally as custom tests, ideally
>>>>>> even with broken connections/reboots?
>>>>>>
>>>>>>
>>>>>> Super tests
>>>>>> ===========
>>>>>>
>>>>>> We don't need to do anything and leave everything on the user. He is
>>>>>> free to write code like:
>>>>>>
>>>>>> ...
>>>>>> machine1 = aexpect.ShellSession("ssh $machine1")
>>>>>> machine2 = aexpect.ShellSession("ssh $machine2")
>>>>>> machine1.sendline("netserver -D")
>>>>>> # wait till the netserver starts
>>>>>> machine1.read_until_any_line_matches(["Starting netserver"],
>>>>>> 60)
>>>>>> output = machine2.cmd_output("netperf -H $machine1 -l
>>>>>> $duration")
>>>>>> # interrupt the netserver
>>>>>> machine1.sendline("\03")
>>>>>> # verify netserver finished
>>>>>> machine1.cmd("true")
>>>>>> ...
>>>>>>
>>>>>> the problem is it requires active connection and the user needs to
>>>>>> manually handle the results.
>>>>>
>>>>> And of course the biggest problem here is that it doesn't solve the
>>>>> Avocado problem: providing a framework and tools for tests that span
>>>>> multiple (Avocado) execution threads, possibly on multiple hosts.
>>>>>
>>> Well it does, each "ShellSession" is a new parallel process. The only
>>> problem I have with this design is that it does not allow easy code
>>> reuse and the results strictly depend on the test writer.
>>>
>>
>> Yes, *aexpect* allows parallel execution in an asynchronous fashion. Not
>> targeted to tests *at all*. Avocado, as a test framework, should deliver
>> more. Repeating the previous wording, it should be "providing a
>> framework and tools for tests that span multiple (Avocado) execution
>> threads, possibly on multiple hosts."
>>
> That was actually my point. You can implement multi-host-tests that way,
> but you can't share the tests (only include some shared pieces from
> libraries).
>
Right, then not related to Avocado, just an example of how a test writer
could do it (painfully) today.
>>>>>>
>>>>>>
>>>>>> Triggered simple tests
>>>>>> ======================
>>>>>>
>>>>>> Alternatively we can say each machine/worker is nothing but yet
>>>>>> another
>>>>>> test, which occasionally needs a synchronization or data-exchange.
>>>>>> The
>>>>>> same example would look like this:
>>>>>>
>>>>>> machine1.py:
>>>>>>
>>>>>> process.run("netserver")
>>>>>> barrier("server-started", 2)
>>>>>> barrier("test-finished", 2)
>>>>>> process.run("killall netserver")
>>>>>>
>>>>>> machine2.py:
>>>>>>
>>>>>> barrier("server-started", 2)
>>>>>> self.log.debug(process.run("netperf -H %s -l 60"
>>>>>> % params.get("server_ip"))
>>>>>> barrier("test-finished", 2)
>>>>>>
>>>>>> where "barrier(name, no_clients)" is a framework function which makes
>>>>>> the process wait till the specified number of processes are waiting
>>>>>> for
>>>>>> the same barrier.
>>>>>
>>>>> The barrier mechanism looks like an appropriate and useful utility
>>>>> for the
>>>>> example given. Even though your use case example explicitly
>>>>> requires it,
>>>>> it's worth pointing out and keeping in mind that there may be valid
>>>>> use cases
>>>>> which won't require any kind of synchronization. This may even be
>>>>> true to
>>>>> the executions of tests that spawn multiple *local* "Avocado runs".
>>>>>
>>> Absolutely, this would actually allow Julio to run his "Parallel
>>> (clustered) testing".
>>
>> So, let's try to identify what we're really looking for. For both the
>> use case I mentioned and Julio's "Parallel (clustered) testing", we need
>> a (the same) test run by multiple *runners*. A runner in this context is
>> something that implements the `TestRunner` interface, such as the
>> `RemoteTestRunner`:
>>
>> https://github.com/avocado-framework/avocado/blob/master/avocado/core/remote/runner.py#L37
>>
>>
>>
>> The following (pseudo) Avocado Test could be written:
>>
>> from avocado import Test
>>
>> # These are currently private APIs that could/should be or
>>
>> # be exposed under another level. Also, the current API is
>>
>> # very different from what is used here, please take it as
>>
>> # pseudo code that might look like a future implementation
>>
>> from avocado.core.remote.runner import RemoteTestRunner
>>
>> from avocado.core.runner import run_multi
>>
>> from avocado.core.resolver import TestResolver
>>
>> from avocado.utils.wait import wait_for
>>
>> class Multi(Test):
>>
>> def test(self):
>>
>> worker1 = RemoteTestRunner('worker1')
>>
>> worker2 = RemoteTestRunner('worker2')
>>
>> # Resolve a local test to send it to be run on multiple machines
>>
>> test = TestResolver().resolve('bonnie.py')
> not actually needed, it's part of the RemoteTestRunner
From the comments: "please take it as pseudo code that might look like
a future implementation".
>
>>
>> # run_multi is asynchronous, and results can be queried about
>> its status
>>
>> results = run_multi([worker1, worker2], test)
>>
>> wait_for(results.finished, self.timeout)
> Where are the results stored? The RemoteTestRunner only returns json
> results, so the user has to do it manually (or we provide yet another
> API to store results in-place)
>
And what prevents us from fetching and including the complete remote job
result? Again: "please take it as pseudo code that might look like a
future implementation".
>>
>> # combine remote whiteboard (with performance results) keyed by
>> worker name
>>
>> whiteboard = {}
>>
>> for worker_result in results:
>>
>> whiteboard[worker_result.name] = worker_result.whiteboard
>>
>> self.whiteboard = whiteboard
> This is sort-of-minimal functional version of your proposal:
>
> ```
> #!/usr/bin/env python
>
> from avocado import main
> from avocado import Test
>
>
> class PrintCalls(object):
>
> def __getattr__(*args, **kwargs):
> def printme(*args2, **kwargs2):
> print args, kwargs, args2, kwargs2
> return printme
>
>
> class PassTest(Test):
>
> """
> Example test that passes.
> """
>
> def test(self):
> """
> A test simply doesn't have to fail in order to pass
> """
> from avocado.core.remote.runner import RemoteTestRunner
> from flexmock import flexmock
> import threading
> import logging
> args = args=flexmock(multiplex_files=None, remote_no_copy=False,
> remote_username=None,
> remote_hostname="127.0.0.1",
> remote_password=None, remote_port=None,
> remote_timeout=999, show_job_log=None)
This shows that our RemoteTestRunner, and many core functionality
depends heavily on "args", and lack a clear API. Even if it would be
kept internal/core only, it could use a redesign with a clear API. If we
decide to leverage it for other uses (such as inside tests), then those
kind of changes are pretty much mandatory.
> job1 = flexmock(args=args, urls=["passtest", "failtest"],
> log=logging,
> logdir="/tmp",
>
> unique_id="60eade7a5a9c4df64681c5fbea16818a6bcb2adc")
> remote1 = RemoteTestRunner(job1, PrintCalls())
> failures = remote1.run_suite(None, None, None, 999)
> print failures
>
>
> if __name__ == "__main__":
> main()
> ```
>
We don't want users to have to write this kind of code, right?
Let's not try to solve the problem based on the code we have. Let's try
to solve the problem with an ideal design, and code that we can write.
> it requires additional work on the logging as currently it only reports
> failures, but it should also store per-worker results.
>
> I tried running multiple threads but it resulted into paramiko failures.
>
You obviously wrote this ugly code because it's *currently* the only way
to make it (almost) work. This is the wrong direction to be using on an
RFC like this. Let's focus on the experience we want to have/give, then
the best way we can deliver it.
>>
>>
>> If any kind of synchronization was necessary between workers, the
>> barrier utility library could be used, maybe even
>> transparently as part of "run_multi". Parameter passing to tests is also
>> a layered issue. My point is that this seems to
>> include the features needed to allow "tests that span multiple machines".
>>
>> Does it look reasonable?
>>
>> A lot of sugar coating can (and should) be added on top. Creating
>> workers automatically, having a superclass for tests
>> that span multiple machines, plugins that take worker names directly
>> from command line options and what not are
>> likely natural additions.
>
> Your proposal is very similar to the "control" files RFC and the result
> is not that different from what I proposed.
>
> [RemoteTestRunner]
>
> Allows people limited set of currently internal API which allows them to
> create several jobs inside another job, execute them and then parse
> results and report whatever they want to.
>
> [Super test]
>
> Helps people to construct ordinary "avocado run" commands, adds the
> results-dirs inside this job, executes them in parallel and reports
> pass/fail.
The "RemoteTestRunner" does not exclude the possibility of including the
remote results. In the abstract example I gave, the "results" object can
indeed contain it all. Quoting myself again:
"A lot of sugar coating can (and should) be added on top. Creating
workers automatically, having a superclass for tests that span multiple
machines..."
>
> [conclusion]
>
> They both share the same parts, only "RemoteTestRunner" approach uses
> (currently) internal API to run the jobs, "Super test" uses command line
> (stable+supported) interface known to users.
Let's not try to solve the problems based on the code/architecture we
currently have.
>
> So to me the "Super test" has only one drawback and that is running yet
> another avocado instance. On the contrary it gains the well known
> results per each worker inside the results and it depends on interface
> we support and keep stable.
>
It's not clear to me what you mean by "Super test" by now. Originally,
this is what you called super test:
Super tests
===========
We don't need to do anything and leave everything on the user. He is
free to write code like:
...
machine1 = aexpect.ShellSession("ssh $machine1")
machine2 = aexpect.ShellSession("ssh $machine2")
machine1.sendline("netserver -D")
# wait till the netserver starts
machine1.read_until_any_line_matches(["Starting netserver"], 60)
output = machine2.cmd_output("netperf -H $machine1 -l $duration")
# interrupt the netserver
machine1.sendline("\03")
# verify netserver finished
machine1.cmd("true")
But based on your description, it now looks like you mean something
closer to the "MHT" example. Can you shed some light here?
> Also note that I think some users will run separate avocado instances
> instead of the custom tests (eg. from jenkins, which picks the machines
> and you run local avocado instances over multiple machines and the only
> thing you use is the --sync to synchronize the execution). So it comes
> to me that using the same approach should the best option.
As I said before: "plugins that take worker names directly from command
line options and what not are likely natural additions."
My point here is that the Avocado command line app does not have the
right "API" granularity, simply because it's command line tool, and as
any command line tool, lacks the introspection and communication power
of a proper programming language API. Building this very important
feature on top of the command line *alone* is, besides a layer
violation, a very fragile thing.
Have you thought that we may find ourselves adding command line options
that wouldn't otherwise benefits users just to accommodate our own needs
to develop the multi host test feature? Would that make sense? This
kind of question shows very clearly at what layer we should be focusing on.
- Cleber.
>
>
>>
>>>
>>>>>>
>>>>>> The barrier needs to know which server to use for communication so we
>>>>>> can either create a new service, or simply use one of the
>>>>>> executions as
>>>>>> "server" and make both processes use it for data exchange. So to
>>>>>> run the
>>>>>> above tests the user would have to execute 2 avocado commands:
>>>>>>
>>>>>> avocado run machine1.py --sync-server machine1:6547
>>>>>> avocado run machine2.py --remote-hostname machine2 --mux-inject
>>>>>> server_ip:machine1 --sync machine1:6547
>>>>>>
>>>>>> where:
>>>>>> --sync-server tells avocado to listen on ip address machine1
>>>>>> port 6547
>>>>>> --remote-hostname tells the avocado to run remotely on machine2
>>>>>> --mux-inject adds the "server_ip" into params
>>>>>> --sync tells the second avocado to connect to machine1:6547 for
>>>>>> synchronization
>>>>>
>>>>> To be honest, apart from the barrier utility, this provides little
>>>>> value
>>>>> from the PoV of a *test framework*, and possibly unintentionally,
>>>>> competes
>>>>> and overlaps with "remote" tools such as fabric.
>>>>>
>>>>> Also, given that the multiplexer is an optional Avocado feature, such
>>>>> a feature should not depend on it.
>>> It does not, these are only used to demonstrate this particular
>>> feature. You can hardcode the values in the tests, you can use
>>> env-variables or any other feature.
>>>
>>> Basically this "mht" format is nothing more, than list of "avocado
>>> run" commands to be executed in parallel and it's focus was on
>>> simplicity, maybe even only for demonstration purposes.
>>>
>>>>>
>>>>>>
>>>>>> Running those two tests has only one benefit compare to the previous
>>>>>> solution and that is it gathers the results independently and makes
>>>>>> allows one to re-use simple tests. For example you can create a 3rd
>>>>>> test, which uses different params for netperf, run it on "machine2"
>>>>>> and
>>>>>> keep the same script for "machine1". Or running 2 netperf senders
>>>>>> at the
>>>>>> same time. This would require libraries and more custom code when
>>>>>> using
>>>>>> "Super test" approach.
>>>>>>
>>>>>> There are additional benefits for this solution. When we introduce
>>>>>> the
>>>>>> locking API, tests running on a remote machine will be actually
>>>>>> directly
>>>>>> executed in avocado, therefor the locking API will work for them,
>>>>>> avoiding problems with multiple tests using the same shared resource.
>>>>>>
>>>>>> Another future benefit would be system reboot/lost connection when we
>>>>>> introduce this support for individual tests. The way it'd work is
>>>>>> that
>>>>>> user triggers the jobs, the master remembers the test ids and would
>>>>>> poll
>>>>>> for results until they finish/timeout.
>>>>>>
>>>>>> All of this we get for free thanks to re-using the existing
>>>>>> infrastructure (or the future infrastructure), so I believe this is
>>>>>> the
>>>>>> right way to go and in this RFC I'm describing details of this
>>>>>> approach.
>>>>>>
>>>>>
>>>>> All of the benefits listed are directly based on the fact that
>>>>> tests on
>>>>> remote systems would be run under the Avocado test runner and would
>>>>> have
>>>>> it's runtime libraries available. This is a valid point, but again it
>>>>> doesn't bring a significant change in the user experience wrt running
>>>>> tests that span multiple "Avocado runs" (possibly on remote machines).
>>>>>
>>> Basically this is the key part of this RFC. I like the idea of running
>>> avocado processes for each test, instead of yet another remote
>>> execution handling. The biggest benefit are the test results in well
>>> known format and the possibility to run/combine all the tests
>>> supported by avocado.
>>>
>>> Actually I have avocado-in-avocado script in my CI testing, it just
>>> waits for the long-names fix to be applied as it generates too long
>>> test names. But I tested it with the fix and the results are very nice
>>> and easy to analyze as you simply go through results you know from
>>> simple testing.
>>>
>>>>>>
>>>>>> Triggering the jobs
>>>>>> -------------------
>>>>>>
>>>>>> Previous example required the user to run the avocado 2 times (per
>>>>>> each
>>>>>> machine) and sharing the same sync server. Additionally it resulted
>>>>>> into
>>>>>> 2 separated results. Let's try to eliminate this problem.
>>>>>>
>>>>>>
>>>>>> Basic tests
>>>>>> ~~~~~~~~~~~
>>>>>>
>>>>>> For basic setups, we can come up with very simple format to describe
>>>>>> which tests should be triggered and avocado should take care of
>>>>>> executing it. The way I have in my mind is to simply accept list of
>>>>>> "avocado run" commands:
>>>>>>
>>>>>> simple_multi_host.mht:
>>>>>>
>>>>>> machine1.py
>>>>>> machine2.py --remote-hostname machine2 --mux-inject
>>>>>> server_ip:machine1
>>>>>>
>>>>>> Running this test:
>>>>>>
>>>>>> avocado run simple_multi_host.mht --sync-server 0.0.0.0
>>>>>>
>>>>>> avocado would pick a free port and start the sync server on it.
>>>>>> Then it
>>>>>> would prepend "avocado run" and append "--sync $sync-server
>>>>>> --job-results-dir $this-job-results" to each line in
>>>>>> "simple_multi_host.mht" and run them in parallel. Afterward it'd wait
>>>>>> till both processes finish and report pass/fail depending on the
>>>>>> status.
>>>>>>
>>>>>> This way users get overall results as well as individual ones and
>>>>>> simple
>>>>>> way to define static setups.
>>>>>>
>>>>>
>>>>> First, the given usage example would require Avocado to introduce:
>>>>>
>>>>> * A brand new file format
>>>>> * A new test type (say MULTI_HOST_TEST, in addition to the SIMPLE,
>>>>> INSTRUMENTED, etc).
>>>>>
>>>>> Introducing a brand new file format may look like a very simple thing
>>>>> to do, but it's not. I can predict that we'd learn very quickly that
>>>>> our original file format definition is very limited. Then we'd either
>>>>> have to live with that, or introduce new file format versions, or just
>>>>> break the initial definition or compatibility. These are all problems
>>>>> related to file formats, not really to your proposed file format.
>>>>>
>>>>> Then, analogous to the "remote tools (fabric)" example I gave before,
>>>>> this looks to be outside of the problem scope of Avocado, in the sense
>>>>> that "template" tools can do it better.
>>>>>
>>>>> Introducing a new test type, and a test resolver/loader, would be a
>>>>> mandatory step to achieve this design, but it looks like a necessary
>>>>> action only to make the use of "MHT" file format possible.
>>>>>
>>>>> Please note that having a design that allow users to fire multiple
>>>>> Avocado command line instances executions in their own scripts is a
>>>>> bad
>>>>> thing, but as a test framework, I believe we can deliver a better,
>>>>> more
>>>>> focused experience.
>>>>
>>>> I meant "is *not* a bad thing".
>>>>
>>> I think you have a point here. My idea was to support new-line
>>> separated list of avocado executions as a simple wrapper to run
>>> processes in parallel as it's very simple to develop and it's not
>>> promising anything. It simply takes whatever you hand it over, spawns
>>> multiple processes and gives you results.
>>>
>>> Then to add some value I added the --sync handling as it's one
>>> problematic thing. Basically it can be written in a generic way, but I
>>> see your point with hard-to-debug failures or unexpected behavior.
>>>
>>> It was meant to be a very simple and easy to understand way to promote
>>> multi-host-testing but it can as well become very painful thing if
>>> people start relying on it. So maybe we should only introduce the real
>>> thing below.
>>>
>>>>>
>>>>>>
>>>>>> Contrib scripts
>>>>>> ~~~~~~~~~~~~~~~
>>>>>>
>>>>>> The beauty of executing simple lines is, that users might create
>>>>>> contrib
>>>>>> scripts to generate the "mht" files to get even better flexibility.
>>>>>
>>>>> Since I don't think a new file format and test type is a good thing,
>>>>> this
>>>>> also becomes a bad idea IMHO.
>>>>>
>>>>>>
>>>>>>
>>>>>> Advanced tests
>>>>>> ~~~~~~~~~~~~~~
>>>>>>
>>>>>> The above might still not be flexible enough. But the system
>>>>>> underneath
>>>>>> is very simple and flexible. So how about creating instrumented
>>>>>> tests,
>>>>>> which generate the setup? The same simple example as before:
>>>>>>
>>>>>> multi_host.py
>>>>>>
>>>>>> runners = ["machine1.py"]
>>>>>> runners.append("machine2.py --remote-hostname machine2
>>>>>> --mux-inject
>>>>>> server_ip:machine1")
>>>>>> self.execute(runners)
>>>>>>
>>>>>
>>>>> A major plus here is that there's no attempt to define new file
>>>>> formats,
>>>>> test types and other items that are necessary only to fulfill a use
>>>>> case
>>>>> requirement. Since Avocado's primary language of choice is Python, we
>>>>> should stick to it, given that it's expressive enough and well
>>>>> maintained
>>>>> enough. This is of course a lesson we learned with Autotest itself,
>>>>> let's
>>>>> not forget it.
>>>>>
>>>>> Then, a couple of things I dislike here:
>>>>>
>>>>> 1) First runner is special/magical (sync server would be run here)
>>>>> 2) Interface with runner execution is done by command line
>>>>> parameters
>>>>>
>>> Well the 0-st runner is special (the one which executes the
>>> multi-host-instrumented-test). It needs to listen on any free port and
>>> pass this port to all executed tests (if they use barriers/sync).
>>>
>>> I'll talk about the 2nd point later....
>>>
>>>
>>>>>> where the "self.execute(tests)" would take the list and does the
>>>>>> same as
>>>>>> for basic tests. Optionally it could return the json results per each
>>>>>> tests so the test itself can react and modify the results.
>>>>>>
>>>>>> The above was just a direct translation of the previous example,
>>>>>> but to
>>>>>> demonstrate the real power of this let's try a PingPong multi host
>>>>>> test:
>>>>>>
>>>>>> class PingPong(MultiHostTest):
>>>>>> def test(self):
>>>>>> hosts = self.params.get("hosts", default="").split(";")
>>>>>> assert len(hosts) >= 2
>>>>>> runners = ["ping_pong --remote-hostname %s" % _
>>>>>> for _ in hosts]
>>>>>> # Start creating multiplex tree interactively
>>>>>> mux = MuxVariants("variants")
>>>>>> # add /run/variants/ping with {} values
>>>>>> mux.add("ping", {"url": hosts[1], "direction": "ping",
>>>>>> "barrier": "ping1"})
>>>>>> # add /run/variants/pong with {} values
>>>>>> mux.add("pong", {"url": hosts[-1], "direction": "pong",
>>>>>> "barrier": "ping%s" % len(hosts) + 1})
>>>>>> # Append "--mux-inject mux-tree..." to the first
>>>>>> command
>>>>>> runners[0] += "--mux-inject %s" % mux.dump()
>>>>>> for i in xrange(1, len(hosts)):
>>>>>> mux = MuxVariants("variants")
>>>>>> next_host = hosts[i+1 % len(hosts)]
>>>>>> prev_host = hosts[i-1]
>>>>>> mux.add("pong", {"url": prev_host, "direction":
>>>>>> "pong",
>>>>>> "barrier": "ping%s" % i})
>>>>>> mux.add("ping", {"url": next_host, "direction":
>>>>>> "ping",
>>>>>> "barrier": "ping%s" % i+1})
>>>>>> runners[i] += "--mux-inject %s" % mux.dump()
>>>>>> # Now do the same magic as in basic multihost test on
>>>>>> # the dynamically created scenario
>>>>>> self.execute(runners)
>>>>>>
>>>>>> The `self.execute` generates the "simple test"-like list of "avocado
>>>>>> run" commands to be executed. But the test writer can define some
>>>>>> additional behavior. In this example it generates
>>>>>> machine1->machine2->...->machine1 chain of ping-pong tests.
>>>>>
>>>>> You mean that this would basically generate a "shell script like" list
>>>>> of avocado runs? This looks to be a very strong design decision, and
>>>>> I fail to see how it would lend itself to be flexible enough and
>>>>> deliver
>>>>> the "test writer can define some additional behavior" requirement.
>>>>>
>>> Explanation below...
>>>
>>>>>>
>>>>>> When running "avocado run pingpong --mux-inject
>>>>>> hosts:machine1;machine2"
>>>>>> this generates 2 jobs, both running just a single "ping_pong" test
>>>>>> with
>>>>>> 2 multiplex variants:
>>>>>>
>>>>>> machine1:
>>>>>>
>>>>>> variants: !mux
>>>>>> ping:
>>>>>> url: machine2
>>>>>> direction: pong
>>>>>> barrier: ping1
>>>>>> pong:
>>>>>> url: machine2
>>>>>> direction: pong
>>>>>> barrier: ping2
>>>>>> machine2:
>>>>>>
>>>>>> variants: !mux
>>>>>> pong:
>>>>>> url: machine1
>>>>>> direction: pong
>>>>>> barrier: ping1
>>>>>> ping:
>>>>>> url: machine1
>>>>>> direction: ping
>>>>>> barrier: ping2
>>>>>>
>>>>>> The first multiplex tree for three machines looks like this:
>>>>>>
>>>>>> variants: !mux
>>>>>> ping:
>>>>>> url: machine2
>>>>>> direction: pong
>>>>>> barrier: ping1
>>>>>> pong:
>>>>>> url: machine3
>>>>>> direction: pong
>>>>>> barrier: ping
>>>>>>
>>>>>> Btw I simplified the format for the sake of this RFC. I think
>>>>>> instead of
>>>>>> generating the strings we should support API to specify test,
>>>>>> multiplexer, options... and then turn them into the parallel executed
>>>>>> jobs (usually remotely). But these are just details to be solved
>>>>>> if we
>>>>>> decide to work on it.
>>>>>
>>>>> This statement completely changes what you have proposed up to this
>>>>> point.
>>>>>
>>>>> IMHO it's far from being just details, because that would define the
>>>>> lowest
>>>>> and commonest level of this feature set that we would advertise and
>>>>> support.
>>>>> The design should really be from this level up, and not from the
>>>>> opposite
>>>>> direction.
>>>>>
>>>>> If external users want to define file formats (say your own MHT
>>>>> proposal) on
>>>>> top of our "framework for running tests that span multiple execution
>>>>> threads"
>>>>> at once, they should be able to do so.
>>>>>
>>>>> If you ask me, having sound Avocado APIs that users could use to fire
>>>>> multiple
>>>>> portions of their *tests* at once and have their *results* coalesced
>>>>> into a
>>>>> single
>>>>> *test* result is about what Avocado should focus on.
>>> And this was suppose to be the answer. In the end yes, I think it
>>> should generate the "avocado run" command with result-dir based inside
>>> this test's results. The reason is it gives you the results you know
>>> per each worker and they can run independently (survive the network
>>> issues, system reboots when we add the support for it in avocado)
>>>
>>
>> This would really be an implementation detail of the chosen runner. The
>> current remote runner actually run Avocado, but that's its
>> (RemoteTestRunner) own internal design decision. It does have a lot of
>> pluses, but that is not the focus of this conversation. Another runner,
>> say, ThinTestRunner, could choose to do things differently.
>>
>> Having said that, I completely agree that we should, unless proven
>> wrong, reuse the RemoteTestRunner for multi-host tests.
>>
>>> The alternative is to create a client worker, which executes code on
>>> demand, but that's more complex and it'd double the effort if we
>>> decide to support system reboots/connection issues.
>>
>> Agreed. Having an agent/broker on the remote side does not seem to be
>> necessary or beneficial at this point.
>>
>>>
>>> What this paragraph was about is that it should not probably directly
>>> generate the arguments, but we should define an API which adds
>>> individual pieces of information and is translated into the command at
>>> the end.
>>>
>>> I decided not to go into details here as I thought it's better to
>>> focus on part1 (--sync --sync-server) which already has a proof of
>>> concept version out there. Then I wanted to create the "mht" file,
>>> which would demonstrate how the results could look like, and how it
>>> all goes together and when we have those results and issues, we can
>>> introduce the instrumented-test API which would evolve from the
>>> real-world issues.
>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Results and the UI
>>>>>> ==================
>>>>>>
>>>>>> The idea is, that the user is free to run the jobs separately, or to
>>>>>> define the setup in a "wrapper" job. The benefit of using the
>>>>>> "wrapper"
>>>>>> job are the results in one place and the `--sync` handling.
>>>>>>
>>>>>> The difference is that running them individually looks like this:
>>>>>>
>>>>>> 1 | avocado run ping_pong --mux-inject url:192.168.1.58:6001
>>>>>> --sync-server
>>>>>> 1 | JOB ID : 6057f4ea2c99c43670fd7d362eaab6801fa06a77
>>>>>> 1 | JOB LOG :
>>>>>> /home/medic/avocado/job-results/job-2016-01-22T05.33-6057f4e/job.log
>>>>>> 1 | SYNC : 0.0.0.0:6001
>>>>>> 1 | TESTS : 1
>>>>>> 1 | (1/1) ping_pong: \
>>>>>> 2 | avocado run ping_pong --mux-inject :url::6001
>>>>>> direction:pong
>>>>>> --sync 192.168.1.1:6001 --remote-host 192.168.1.1
>>>>>> 2 | JOB ID : 6057f4ea2c99c43670fd7d362eaab6801fa06a77
>>>>>> 2 | JOB LOG :
>>>>>> /home/medic/avocado/job-results/job-2016-01-22T05.33-6057f4e/job.log
>>>>>> 2 | TESTS : 1
>>>>>> 2 | (1/1) ping_pong: PASS
>>>>>> 1 | (1/1) ping_pong: PASS
>>>>>>
>>>>>> and you have 2 results directories and 2 statuses. By running them
>>>>>> wrapped inside simple.mht test you get:
>>>>>>
>>>>>> avocado run simple.mht --sync-server 192.168.122.1
>>>>>> JOB ID : 6057f4ea2c99c43670fd7d362eaab6801fa06a77
>>>>>> JOB LOG :
>>>>>> /home/medic/avocado/job-results/job-2016-01-22T05.33-6057f4e/job.log
>>>>>> TESTS : 1
>>>>>> (1/1) simple.mht: PASS
>>>>>> RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>>>>>> INTERRUPT 0
>>>>>> TIME : 0.00 s
>>>>>>
>>>>>> And single results:
>>>>>>
>>>>>> $ tree $RESULTDIR
>>>>>>
>>>>>> └── test-results
>>>>>> └── simple.mht
>>>>>> ├── job.log
>>>>>> ...
>>>>>> ├── 1
>>>>>> │ └── job.log
>>>>>> ...
>>>>>> └── 2
>>>>>> └── job.log
>>>>>> ...
>>>>>>
>>>>>> tail -f job.log:
>>>>>> running avocado run ping pong ping pong
>>>>>> running avocado run pong ping pong ping --remote-hostname
>>>>>> 192.168.122.53
>>>>>> waiting for processes to finish...
>>>>>> PASS avocado run ping pong ping pong
>>>>>> FAIL avocado run pong ping pong ping --remote-hostname
>>>>>> 192.168.122.53
>>>>>> this job FAILED
>>>>>>
>>>>>
>>>>> I won't spend much time here, since the UI is bound to follow other
>>>>> design
>>>>> ideas/decisions.
>>>>>
>>> Sure, the important part here is the results format.
>>>
>>>>>>
>>>>>> Demonstration
>>>>>> =============
>>>>>>
>>>>>> While considering the design I developed a WIP example. You can
>>>>>> find it
>>>>>> here:
>>>>>>
>>>>>> https://github.com/avocado-framework/avocado/pull/1019
>>>>>>
>>>>>> It demonstrates the `Triggered simple tests` chapter without the
>>>>>> wrapping tests. Hopefully it helps you understand what I had in
>>>>>> mind. It
>>>>>> contains modified "examples/tests/passtest.py" which requires 2
>>>>>> concurrent executions (for example if you want to test your server
>>>>>> and
>>>>>> run multiple concurrent "wget" connections). Feel free to play with
>>>>>> it,
>>>>>> change the number of connections, set different barriers, combine
>>>>>> multiple different tests...
>>>>>>
>>>>>>
>>>>>> Autotest
>>>>>> ========
>>>>>>
>>>>>> Avocado was developed by people familiar with Autotest, so let's just
>>>>>> mention here, that this method is not all that different from
>>>>>> Autotest
>>>>>> one. The way Autotest supports parallel execution is it let's
>>>>>> users to
>>>>>> create the "control" files inside the multi-host-control-file and
>>>>>> then
>>>>>> run those in parallel. For synchronization it contains master->slave
>>>>>> barrier mechanism extended of SyncData to send pickled data to all
>>>>>> registered runners.
>>>>>>
>>>>>> I considered if we should re-use the code, but:
>>>>>>
>>>>>> 1. we do not support control files, so I just inspired by passing the
>>>>>> params to the remote instances
>>>>>
>>>>> One of the wonderful things about Autotest control files is that
>>>>> it's not a custom file format. This can not be underestimated. While
>>>>> other frameworks have had huge XML based file formats to drive their
>>>>> jobs, Autotest control files are infinitely more capable and their
>>>>> readability is a lot more scalable.
>>>>>
>>>>> The separation of client and server test types (and control files) is
>>>>> actually what prevents control files from nearing perfection IMHO.
>>> Yep
>>>
>>>>>
>>>>> The server API allows you to run client control files on given hosts.
>>>>> These client control files usually need tweaking for each host. Then
>>>>> you're suddenly doing code generation (control files Python code).
>>>>> That
>>>>> is not nice.
>>> The tests I saw usually generated simple "runTest" with different
>>> params. So what I'm proposing is actually similar, let's run avocado
>>> and allow params passing.
>>>
>>>>>
>>>>> I believe that, if Avocado provides such an API that allows regular
>>>>> Python
>>>>> code to operate similarly to server control files, while giving more
>>>>> control
>>>>> and granularity to what is run on the individual job executions (say
>>>>> on remote machines), and help to coalesce the individual portions
>>>>> into a
>>>>> single test result, it would be a very attractive tool.
>>> I think the multi-host test should only pick existing normal tests and
>>> run the set of tests they need to perform the task using barriers to
>>> synchronize it.
>>>
>>> Actually there is one thing which is significantly limiting the usage
>>> and that's the multiplexer. I'd like to run:
>>>
>>> "avocado run boot migrate recievemigrate migrate recievemigrate
>>> shutdown" tests and use different params for each tests. Currently
>>> this is not possible and it's something I'd been proposing all the
>>> time. (mapping params to individual tests).
>>>
>>> Anyway even without this mapping we can do all kinds of setups and
>>> when we add such feature we can always start using it in
>>> multi-host-testing as multi-host-testing is just triggering
>>> avocado-jobs in terms of this RFC so all features available in avocado
>>> are available to each worker in multi-host-testing.
>>>
>>> PS: The multiplexer is not needed for multi-host-tests, you're free to
>>> hard-code the values inside tests or to use whatever way to tell the
>>> test what it should do. The barriers are using the server from
>>> "--sync" cmdline argument so the test is the only component which
>>> might need to be parametric.
>>
>> I will, on purpose, not explore the parameter passing problems until we
>> are more or less on the same page about the bare bones of "running a
>> test that span multiple machines". Then we can explore this optional but
>> very important aspect.
>>
>>>
>>>>>
>>>>>> 2. the barriers and syncdata are quite hackish, master->slave
>>>>>> communication. I think the described (and demonstrated) approach does
>>>>>> the same in a less hackish way and is easy to extend
>>>>>>
>>>>>> Using this RFC we'd be able to run autotest-multi-host tests, but
>>>>>> it'd
>>>>>> require rewriting the control files to "mht" (or contrib) files.
>>>>>> It'd be
>>>>>> probably even possible to write a contrib script to run the control
>>>>>> file
>>>>>> and generate the "mht" file which would run the autotest test. Anyway
>>>>>> the good think for us is, that this does not affect "avocado-vt",
>>>>>> because all of the "avocado-vt" multi-host tests are using a single
>>>>>> "control" file, which only prepares the params for simple avocado-vt
>>>>>> executions. The only necessary thing is a custom "tests.cfg" as by
>>>>>> default it disallows multi-host tests (or we can modify the
>>>>>> "tests.cfg"
>>>>>> and include the filter inside the "avocado-vt" loader, but these are
>>>>>> just the details to be sorted when we start running avocado-vt
>>>>>> multi-host tests.
>>>>>>
>>>>>> Conclusion
>>>>>> ==========
>>>>>>
>>>>>> Multi-host testing was solved many times in the history. Some
>>>>>> hardcode
>>>>>> tests with communication, but most framework I had seen support
>>>>>> triggering "normal/ordinary" tests and add some kind of barrier
>>>>>> (either
>>>>>> inside the code or between the tests) mechanism to synchronize the
>>>>>> execution. I'm for the flexibility and easy test sharing and that
>>>>>> is how
>>>>>> I described it here.
>>>>>>
>>>>>> Kind regards,
>>>>>> Lukáš
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Avocado-devel mailing list
>>>>> Avocado-devel at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/avocado-devel
>>>>>
>>>
>>
>
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
More information about the Avocado-devel
mailing list