[Avocado-devel] Multi Stream Test Support

[Jeff Nelson]

On Wed, 12 Apr 2017 18:32:57 -0400
Cleber Rosa <crosa at redhat.com> wrote:

> On 04/10/2017 01:56 PM, Jeff Nelson wrote:
> > On Mon, 3 Apr 2017 09:48:16 -0400
> > Cleber Rosa <crosa at redhat.com> wrote:
> >   
> >> Note: this document can be view in rendered format at:
> >>
> >> https://github.com/clebergnu/avocado/blob/RFC_multi_stream_v1/docs/source/rfcs/multi_stream.rst
> >>  
> > 
> > Here's my feedback. Please let me know if you have any questions.

As I was working on the response you've given to my feedback (good job
by the way) I realized I have an extreme, unstated bias that I should
mention. This comment is being written last but it appears up front to
provide context to what follows.

When I first read the RFC I was approaching it from the viewpoint of a
test writer ONLY. I expected to find:

a) the definition of a stream from the point of view of a test writer;
b) the promises that I--as a test writer--could expect from avocado
   regarding the behavior of streams;
c) how I--as a test writer--could manipulate and be informed about
   streams; and
d) what I--as a test writer--could (and couldn't) do in the code 
   I would be writing as the developer of the stream.

However, it has became clear to me that my expectations were wrong.
This RFC contains a much broader description of multi stream test
support. In particular, it describes streams not only from the test
writer's viewpoint, it describes changes to and enhancements of avocado
internals. These are things that a test writer doesn't need to know,
but that an avocado developer does.

If I were writing this RFC I would want to make a clear distinction
between these two viewpoints. I would actually separate them in the
text, focusing first on the test writer's view and then the avocado
developer's view (things like the stream manager and send()). And I
would make it clear in the introduction that the RFC is going to talk
about both viewpoints. But that's just me; this is the first RFC I've
studied in great detail and I'll know better next time.

This realization is repeated below, in the spot where it first occurred
to me that it might be happening. You can choose to do with this what
you like.


> > 
> >   
> >> ===========================
> >>  Multi Stream Test Support
> >> ===========================
> >>
> >> Introduction
> >> ============
> >>
> >> Avocado currently does not provide test writers with standard tools
> >> or guidelines for developing tests that spawn multiple machines.
> >>  
> > 
> > 
> > s/spawn multiple machines/require multiple, cooperating execution
> > contexts/ ?
> > 
> >   
> 
> The reason for using "multiple machine" here is that, up to the this
> point, that's how we have seen users referring to this particular
> feature.  I felt that an introductory line without such reference would
> fail to establish a connection to it.
> 
> Still, "multiple execution contexts" sounds good, and I'll try to
> incorporate both ideas, if at all possible, in the next version.

OK.

 
> >> Since these days the concept of a "machine" is blurring really
> >> quickly, this proposal for Avocado's version of "multi machine" test
> >> support is more abstract (that's an early and quick explanation of
> >> what a "stream" means).  One of the major goal is to be more flexible
> >> and stand the "test" (pun intended) of time.  
> > 
> > This introduction mentions "machine" multiple times. I found this
> > puzzling, because the proposal is titled "multi stream test support".
> > The machine view is an old, historical idea and I know next to nothing
> > about it, so it's not very helpful to me.
> >   
> 
> Right, you're spot on here.  I say that because "multiple machine" may
> mean *a lot* to users with previous involvement and engagement in
> similar tools.
> 
> > I suggest writing an intro for a target audience that knows nothing
> > about the past but is familiar with how avocado operates today. The
> > intro should describe the goal or objective of the design. Reference to
> > older implementations and ideas can be done in the Background section.
> > 
> >   
> 
> OK, makes sense.
> 
> >> This is a counter proposal to a previous RFC posted and discussed on
> >> Avocado development mailing list.  Many of the concepts detailed here
> >> were introduced there:
> >>
> >> * https://www.redhat.com/archives/avocado-devel/2016-March/msg00025.html
> >> * https://www.redhat.com/archives/avocado-devel/2016-March/msg00035.html
> >> * https://www.redhat.com/archives/avocado-devel/2016-April/msg00042.html
> >> * https://www.redhat.com/archives/avocado-devel/2016-April/msg00072.html  
> > 
> > Are there other related RFCs that this proposal affects? For example,
> > is there an RFC that describes the avocado callable test interface? If
> > so, referencing it might be useful for readers.
> > 
> >    
> 
> I don't think so.  The implementation of the ideas given here should not
> affect the concepts that are already consolidated in Avocado.
> 
> A current Avocado user hoping to execute his/her tests, should not be
> affected in any way, shape or form, by upgrading to a version that
> contains the implementation of the ideas proposed here.

OK. Is it worth stating this as an explicit goal of the proposal?


> >> Background
> >> ==========
> >>
> >> The prior art that influences Avocado the most is Autotest.  The
> >> reason is that many of the Avocado developers worked on Autotest
> >> before, and both share various common goals.  Let's use Autotest,
> >> which provided support for multiple machine test support as a basis
> >> for comparison.
> >>
> >> Back in the Autotest days, a test that would spawn multiple machines
> >> was a very particular type of test.  To write such a test, one would
> >> write a **different** type of "control file" (a server one).  Then, by
> >> running a "server control file" with an **also different** command
> >> line application (``autoserv``, A.K.A. ``autotest-remote``), the
> >> server control file would have access to some special variables, such
> >> as the ``machines`` one.  By using an **also different** type of job
> >> implementation, the control file could run a given **Python function**
> >> on these various ``machines``.
> >>
> >> An actual sample server control file (``server/samples/reboot.srv``)
> >> for Autotest looks like this::
> >>
> >>    1  def run(machine):
> >>    2     host = hosts.create_host(machine)
> >>    3     host.reboot()
> >>    4
> >>    5  job.parallel_simple(run, machines)
> >>
> >> Line #5 makes use of the different (server) job implementation to run
> >> function ``run`` (defined in line #1) in parallel on machines given by
> >> the special variable ``machines`` (made available by the also special
> >> ``autoserv`` tool).
> >>
> >> This quick background check shows two important facts:  
> > 
> > s/two/three/ (or four, if you adopt the suggestion that follows)
> > 
> >   
> 
> Right!  I wish I mastered my text editor enough to write an automatic
> check for list counts like this. :)
> 
> 
> >> 1) The functionality is not scoped to tests.  It's not easy to understand
> >>    where a test begins or ends by looking at such a control file.
> >>
> >> 2) Users (and most importantly test writers) have to learn about
> >>    different tools and APIs when writing "multi machine" code;
> >>
> >> 3) The machines are defined outside the test itself (in the form of
> >>    arguments to the ``autoserv`` command line application);  
> > 
> > 4. The "machine" abstraction is insufficient.
> > 
> > Basically, the "multiple machines" view has evolved into a "multiple
> > execution streams" view. With multiple streams, the machine is just one
> > of several properties.
> > 
> >   
> 
> I'd say that a "machine" is just one of the possible backends (or
> implementations, or models, or types) when using the "execution stream"
> abstract interface.  How does that sound?

Sure, give it a try.


> >> Please keep these Autotest characteristics in mind: Avocado's multi
> >> stream test support goals will be presented shortly, and will detail
> >> how they contrast with those.
> >>
> >> Avocado's Multi Stream Test Support Goals
> >> =========================================
> >>
> >> This is a hopefully complete summary of our goals:
> >>
> >> 1) To not require a different type of test, that is, allow users
> >>    to *write* a plain `avocado.Test` while still having access to
> >>    multi stream goodies;
> >>
> >> 2) To allow for clear separation between the test itself and its
> >>    execution environment (focus here on the execution streams
> >>    environment);
> >>
> >> 3) To allow increased flexibility by abstracting the "machines"
> >>    concept into "excution streams";
> >>
> >> 4) To allow for even increased flexibility by allowing test writers to
> >>    use not only Python functions, but other representations of code to
> >>    be executed on those separate streams;  
> > 
> > I'm not sure what this means, specifically the phrase "other
> > representations of code".
> > 
> >   
> 
> You mean the 4th item, or the whole list?  If it's about the 4th item,
> let me give some context.  Virtually all the Python based frameworks for
> executing code on multiple machines rely on the execution of plain and
> simple Python functions.

I mean the 4th item.

 
> Now, still with regards to the 4th item, let me try to rephrase it:
> 
> 4) Code to be run on execution streams can be of various types.

Now that I've studied this item a bit more I think I understand the
original definition of item 4, but there was a part I stumbled over.
Here's a suggested wording change that might improve it a bit:

s/To allow for even increased flexibility by allowing test writers
to use/To give test writers greater flexibility by permitting/

They key transformation is changing the phrase "even increased" to
"greater," which makes more sense to me grammatically.

 
> >> Comparison with prior art
> >> -------------------------
> >>
> >> When compared to the Autotest version of multiple machine support for
> >> tests, Avocado's version is similar in that it keeps the separation of
> >> machine and test definition.  
> > 
> > This is slightly contradictory to the autotest example given above,
> > where one of the deficiencies was, "it's not easy to understand where a
> > test begins and ends". In other words, it seems as though you're
> > claiming avocado has a characteristic similar to autotest, but it's a
> > characteristic we don't really like.
> > 
> >   
> >>                                That means that tests written in
> >> accordance to the official guidelines, will not contain reference to
> >> the machines ("execution streams") on which they will have portions of
> >> themselves executed on.  
> > 
> > OK, I see the point.
> > 
> >   
> >> But, a major difference from the Autotest version is that this
> >> proposal attempts to provide the **same basic tools and test APIs** to
> >> the test writers needing the multiple stream support.  Of course,
> >> additional tools and APIs will be available, but they will not
> >> incompatible with traditional Avocado INSTRUMENTED tests.  
> > 
> > Yes.
> > 
> > This comparison section is a bit light. One could almost read it as a
> > restatement of Goal 1. And there are 4 goals in total. If the key
> > points of this comparison were incorporated into Goal 1, you could
> > probably drop this section. (Or you could enhance the section by adding
> > comparisons to the other goals.)
> > 
> >   
> 
> I'll try to expand it.  As I mentioned before, a lot of users have the
> Autotest concepts and approach in mind, so I think it's useful to make a
> more detailed comparison.
> 
> >> Core concepts
> >> =============
> >>
> >> Because the first goal of this RFC is to set the general scope and
> >> approach to Multi Stream test support, it's important to properly
> >> describe each of the core concepts (usually abstractions) that will be
> >> used in later parts of this document.
> >>
> >> Execution Stream
> >> ----------------
> >>
> >> An *Execution Stream* is defined as a disposable execution environment,
> >> different and ideally isolated from the main test execution environment.  
> > 
> > What about the relationship between execution streams? Is there any
> > requirement or expectation that the execution streams are different and
> > ideally isolated from *each other* as well?
> > 
> >   
> 
> One execution stream should have no knowledge about another execution
> stream.  No dependencies across execution streams will be supported in
> any way.

I agree and suggest that this be explicitly specified so that test
writers know that they can't do this.


> Having said that, there are some examples about synchronization in a
> portion (called slice) of execution stream instances.  The execution
> streams manager, available in a test as `self.streams`, will coordinate
> synchronization.

True, but the execution streams manager is not something that test
writers implement. When I wrote the above I was thinking about
the architecture as seen from the point of view of a test writer.

I feel like the RFC is being written for two audiences: the test writer
and the avocado developer. The two audiences have different needs and
sometimes the RFC blurs these together.

A test writer needs to know: what is a stream, how does it behave, what
are it's properties, what operations can be performed on it? Also
important: what can't be done? These are the things that a test writer
needs to understand if they are going to write a test that uses
streams.

>From the point of view of the avocado developer, details like the
streams manager is a concept that they need to understand (in addition
to everything that a test writer needs to know). The streams manager
handles the implementation details of streams and it helps provide the
stream behaviors that are promised to the test writer. But, does the
test writer need to understand these things? I'm suggesting that perhaps
they don't.

This is a tricky thing to sort out and you may choose not to do so. But
I'll try to point out these details to help identify them so you can
decide.



> >> A simplistic but still valid implementation of an execution
> >> environment could be based on an Operating System level process.
> >> Another valid implementation would be based on a lightweight
> >> container.  Yet another valid example could be based on a remote
> >> execution interface (such as a secure shell connection).
> >>
> >> These examples makes it clear that level of isolation is determined
> >> solely by the implementation.
> >>
> >>  .. note:: Even though the idea is very similar, the term *thread* was
> >>            intentionally avoided here, so that readers are not led to think
> >>            that the architecture is based on an OS level thread.
> >>
> >> An execution stream is the *"where"* to execute a "Block Of Code"
> >> (which is the *"what"*).
> >>
> >> Block of Code
> >> -------------
> >>
> >> A *Block of Code* is defined as computer executable code that can run
> >> from start to finish under a given environment and is able to report
> >> its outcome.  
> > 
> > Self-referential definition (uses the word code in term being defined
> > and in the definition). Instead of "computer executable code" how about
> > something like "a sequence of executable statements"?
> > 
> >   
> 
> Sounds good.  One question, though: does "statement" makes one think
> about human readable content?  For instance, if we decide to support an
> implementation based on byte-code, would that qualify as "a sequence of
> executable statements"?

Yes. "statement" is an abstract term that means anything from a byte
opcode to an entire function call. It entirely depends upon the context
and in this case, I think it's an appropriate word.


> >> For instance, a command such as ``grep -q vmx /proc/cpuinfo; echo $?``
> >> is valid computer executable code that can run under various shell
> >> implementations.  A Python function or module, a shell command, or
> >> even an Avocado INSTRUMENTED test could qualify as a block of code,
> >> given that an environment knows how to run them.
> >>
> >> Again, this is the *what* to be run on a "Execution Streams" (which,
> >> in turn, is *"where"* it can be run).
> >>
> >> Basic interface
> >> ===============
> >>
> >> Without initial implementation attempts, it's unreasonable to document
> >> interfaces at this point and do not expect them to change.  Still, the
> >> already existing understanding of use cases suggests an early view of
> >> the interfaces that would be made available.  
> > 
> > Personal preference: The first sentence contains two double-negative
> > phrases (unreasonable, do not) which makes the meaning hard to
> > understand. I think the statement would read better if one of the
> > negatives is transformed into a positive.
> > 
> > For example:
> >   s/and do not expect them/because they are likely/
> > 
> >    
> 
> Let me try a version based on your suggestion:
> 
> It's only reasonable to **document** interfaces after real
> implementation attempts take place.  Before that, the interfaces are
> subject to change.  Still, the current understanding of use cases can
> suggest an early preview of the necessary interfaces.

Looks good.

 
> >> Execution Stream Interface
> >> --------------------------
> >>
> >> One individual execution stream, within the context of a test, should
> >> allow its users (test writers) to control it with a clean interface.  
> > 
> > "Control" to me implies being able to manipulate an active, executing
> > stream. Perhaps a better term instead of "control" is "define"?
> >   
> 
> Yes, control is indeed intended here.  Definition would happen elsewhere.
> 
> > [After reading on, I see that in fact the word "control" does make
> > sense given the actions you've listed below. I guess I was expecting to
> > see this section describe the "where" and the "what" properties of an
> > execution stream.]
> > 
> >   
> 
> OK.  I'll check how I can fill in the gaps you've found here.
> 
> >> Actions that an execution stream implementation should provide:
> >>
> >> * ``run``: Starts the execution of the given block of code (async,
> >>   non-blocking).
> >> * ``wait``: Block until the execution of the block of code has
> >>   finished.  ``run`` can be given a ``wait`` parameter that will
> >>   automatically block until the execution of code has finished.
> >> * ``terminate``: Terminate the execution stream, interrupting the
> >>   execution of the block of code and freeing all resources
> >>   associated with this disposable environment  
> > 
> > I'm trying to think about how these actions are different than the
> > actions avocado already defines for a test. So one thing that I think
> > could be clarified is how the multi stream proposal affects the
> > existing test architecture.
> >   
> 
> Right.  The missing link here was obvious the "avocado.Test"
> architecture diagram presented later.  In short, the "avocado.Test"
> itself remains unchanged as long as users don't choose to use the multi
> stream functionality.  Let me break down a few scenarios here:
> 
> 1) Test does not reference "self.streams": test is run in a single
> process, as this is the only execution environment (the isolation model)
> that the Avocado test runner provides for each executed test.
> 
> 2) Test references "self.streams": test is run in a single process (as
> created by the Avocado test runner for isolation purposes).  Executable
> statements provided to each referenced execution stream will run on a
> separate context.

OK.

 
> > For example:
> > * execution streams: today, tests have a single execution stream. In the
> >   future, tests have one or more execution streams.
> >   
> 
> There's no execution stream Today.  There's only the test process, as
> mentioned in scenario #1.
> 
> > * run: today, tests run themselves. In the future, (something) runs
> >   execution streams.
> >   
> 
> Tests will continue to run themselves.  By referring to "self.streams",
> AKA the "streams manager", portions of the test (clearly specified
> "blocks of code") will be run indirectly by the test, and directly by
> the "streams manager".
> 
> >   * what is this "something?" Is it the test runner? Is it a stream?
> >     If it's a stream, then what launches the first stream? I don't
> >     have a clear picture of how this looks (I suspect this reflects
> >     more on me than on you).
> >   
> 
> The Avocado test runner continues to run a test.  The "streams manager"
> will run "blocks of code" on a number of "execution streams".
> 
> >   * this question arises in part because I'm trying to understand what
> >     happens today to the existing code that makes up a test, in
> >     particular the part that makes up the 'mainline'. Does it turn into
> >     a stream? If not, then perhaps it should have an official name
> >     (assuming it doesn't) so that we can reason about it separately
> >     from streams and to avoid people thinking that it's a stream (in
> >     case it's not).
> >   
> 
> An `avocado.Test` does not become an "execution stream", nor will it be
> executed in one.  It continues to be executed by the Avocado test runner.
> 
> > * async: today, test (stream) execution is synchronous. In the future,
> >   streams can execute asynchronously.
> >   
> 
> Today tests are executed synchronously, and will continue to be so (with
> regards to this proposal).  Execution streams can run either
> asynchronous (default) or synchronous (wait=True).
> 
> > * wait: doesn't really exist today and it wouldn't make sense
> >   anyway: a single test (stream) cannot wait on itself. In the future,
> >   streams can wait on other streams. However, a stream cannot wait on
> >   itself and a stream cannot wait on a stream in another test.
> >   
> 
> I guess for now it's clear that a "a single test (stream)" is not a
> valid idea proposed here.  A test and an execution stream are very
> different things.

Agreed. Thanks for explaining the differences.


>  About waiting:
> 
> * It doesn't exist today, because execution streams do not exist today
> and tests are not execution streams.
> 
> * Streams cannot wait on other streams.  Still, given the
> synchronization feature, a group (slice) of streams can be kept from
> `run()`ning their "block of code" until the execution stream manager
> says so.

Right. The first statement ("streams cannot wait on other streams") is
an architectural definition of streams that I think a test writer needs
to know. The rest of the paragraph is a detail that an avocado
developer should know.

 
> * Streams should not be able to wait, at all, for other streams in other
> tests.  The point here is that the "streams manager", which lives under
> the scope of **one** test, is responsible for synchronizing execution
> streams.

Same here: the first statement is a promise to test writers about what
they cannot do with streams. But they don't care about what piece of
code (the streams manager) performs the synchronization nor do they
care how it's done.

> 
> >   * I realize I made a possibly invalid assumption: that any stream in
> >     a test can get information about any other stream in the same test.
> >     That may or may not be true. It implies being able to pass some
> >     sort of handle to an execution stream so that streams can learn
> >     about each other. That's another--possibly bad--assumption.
> >   
> 
> Agreed, and this is being actively avoided on this proposal.

OK.

 
> > * terminate: doesn't really exist today, but there is a similar
> >   operation called 'cancel'. Does the meaning of 'cancel' change?
> > 
> >   
> 
> As execution streams are not tests (or the other way around) then they
> mean two different things.  If a test fires a number of execution
> streams asynchronously, terminate() will be called to (try to) clean up
> the resources created **by the execution streams themselves**.

If an execution stream forks a process, will that be cleaned up by
terminate()? What about shared memory that the execution stream
allocated with shmget?

I think a clear distinction should be made between what avocado and the
execution streams are each responsible for. For example:

* avocado will tell the execution stream to terminate;

* test writers are responsible for writing execution streams that clean
  up after themselves; and

* avocado will manage its own resources that it uses to track execution
  streams

 
> >> The following properties should be provided to let users monitor the
> >> progress and outcome of the execution:
> >>
> >> * ``active``: Signals with True or False wether the block of code
> >>   given on ``run`` has finished executing.  This will always return
> >>   False if ``wait`` is used, but can return either True or False when
> >>   running in async mode.  
> > 
> > s/wether/whether/
> >   
> 
> Yep, thanks.
> 
> > I think if you use the word "signal" then the reader will be thinking
> > in terms of exceptions that are signaled. Using the word "return" seems
> > better, especially since that's what you use in the rest of the
> > description.
> >   
> 
> Good point.
> 
> > Consider that 'active' is only one possibility; there could be a whole
> > set of values that describe the execution state of the stream.
> > 
> >   
> 
> This is actually something that I'm attempting to avoid.  Unless proven
> wrong, I feel there's a lot to gain here by keeping the interface as
> simple as possible.  A test should **not** be in the business of
> checking whether, say, a VM that backs a given execution stream is
> booting or reverting from a snapshot.
> 
> IMO, the `active` property should be set when the *execution stream*
> starts running (including any kind of self setup), and unset when it
> finishes running the *block of code*.  As simple as that.

OK.


> >> * ``success``: A simplistic but precise view of the outcome of the
> >>   execution.  
> > 
> > Likewise, 'success' seems to be one of many possible outcomes.
> > 
> >   
> 
> I see it differently.  The flow would be something like:
> 
> 1) `active` initially set to False, `success` also set to False
> 2) `run()` is called, `success` still set to False
> 3) `run()` finishes running, and given the expected output, sets
> `success` to True

OK.

 
> >> * ``output``: A dictionary of various outputs that may have been
> >>   created by ``run``, keyed by a descriptive name.
> >>
> >> The following properties could be provided to transport block of code
> >> payloads to the execution environment:
> >>
> >> * ``send``: Sends the given content to the execution stream
> >>   environment.
> >>
> >> Block of Code Interface for test writers
> >> ----------------------------------------
> >>
> >> When a test writer intends to execute a block code, he must choose from
> >> one of the available implementations.  Since the test writer must know
> >> what type of code it's executing, the user inteface with the implementation
> >> can be much more flexible.  
> >   
> 
> A typo of mine here: s/inteface/interface/
> 
> > I'm stuck on wording again. Are these equivalent statements?
> > 
> > a) When a test writer intends to execute a block of code
> > b) When a test writer intends to define a block of code
> > c) When a test author intends to write a block of code
> >   
> 
> No, because focus here is on the execution.  Of course, the test writer
> may end up writing the "executable statements" as he writes the test,
> but the focus where is how he'd specify the type of "executable
> statements" he wants executed.

In my mind, there is a distinction between definition and
execution. I think this distinction is blurred in the paragraph that
introduces this section.

In my view of the world, the test writer defines the test code that
avocado executes. Test writers don't execute tests, avocado does.
Similarly, avocado executes streams, not test writers; test writers
define the stream that avocado executes.

Using my terminology, the best paragraph I can write looks like this:

  When a test writer intends to define a block code, he must choose
  the type of code to write. The type can be: a python function, a
  shell command, and so on. Because avocado knows the type of code it's
  executing, the 'block of code' user interface with the implementation
  can be much more flexible.  

I got stuck again, this time on the last sentence, because I don't see
how the conclusion follows. I think it may be pointing out something to
an avocado developer and not to a test writer. Maybe this helps explain
the perspective I have and will give you some ideas.


> 
> > If so I would prefer either (c) or (b) instead of (a).
> > 
> > I have a similar difficulty parsing the phrase, "the test writer must
> > know what type of code it's executing". First, machines typically
> > execute code, not people. Second, "it's" isn't a valid reference to a
> > person in the English language, the gender-specific terms "he" and
> > "she" terms are used.
> > 
> >   
> 
> The "it" in "it's executing" is indeed ambiguous, and should be "the
> execution stream".
> 
> >> For instance, suppose a Block Of Code implementation called
> >> ``PythonModule`` exists.  This implementation would possibly run
> >> something like
> >> ``python -m <modulename>`` and collect its outcome.
> >>
> >> A user of such an implementation could write a test such as::
> >>
> >>   from avocado import Test
> >>   from avocado.streams.code import PythonModule
> >>
> >>   class ModuleTest(Test):
> >>     def test(self):
> >>         self.streams[1].run(PythonModule("mymodule",
> >>                                          path=["/opt/myproject"]))
> >>
> >> The ``path`` interface in this example is made available and supported
> >> by the ``PythonModule`` implementation alone and will not be used the
> >> execution stream implementations. As a general rule, the "payload"
> >> should be the first argument to all block of code implementations.
> >> Other arguments can follow.  
> > 
> > This all seems OK to me.
> > 
> >   
> >> Another possibility related to parameters is to have the Avocado's own
> >> test parameters ``self.params`` passed through to the block of code
> >> implementations, either all of them, or a subset based on path.  This
> >> could allow for example, a parameter signaling a "debug" condition to
> >> be passed on to the execution of the block of code.  Example::
> >>
> >>   from avocado import Test
> >>   from avocado.streams.code import PythonModule
> >>
> >>   class ModuleTest(Test):
> >>     def test(self):
> >>         self.streams[1].run(PythonModule("mymodule",
> >>                                          path=["/opt/myproject"],
> >>                                          params=self.params))  
> > 
> > This makes sense.
> > 
> >    
> >> Block of Code Interface for Execution Stream usage
> >> --------------------------------------------------
> >>
> >> Another type of public interface, in the sense that it's well known
> >> and documented, is the interface that Execution Stream implementations
> >> will use to interact with Block of Code implementations.  This is not
> >> intended to be used by test writers, though.  
> > 
> > Then who would use it?
> > 
> >    
> 
> It's mentioned in the section title: "for *Execution Stream* usage", but
> I have to agree that it was a too abrupt transition which made the
> reference to "execution streams" too subtle (almost invisible?).

Agreed. I think I had expectations about what was going to be in this
section and as a result, I completely missed what was actually being
described. I read the phrase "for execution stream usage" and thought,
"oh good, I'm going to be told what I, as a test writer, am allowed to
use (with respect to avocado APIs) in my execution stream".

Instead, I think this section is written more for the avocado developer (in
order to implement the code to support execution streams).


> 
> >> Again, it's too early to define a frozen implementation, but this is
> >> how it could look like:
> >>
> >> * ``send_self``: uses the Execution Stream's ``send`` interface to properly
> >>   populate the payload or other necessary assets for its execution.  
> > 
> > I'm afraid I'm lost again. The above section on Execution Streams says
> > that its interfaces are used by users (test writers).
> > 
> > Here we have a definition of a send_self interface which is described
> > as using 'send', one of the interfaces of Execution Streams. So does
> > that mean the person writing a send_self method (that calls 'send') is
> > a test writer?
> >   
> 
> The "block of code" implementation will almost always need to send
> content to a location accessible to the execution stream (usually the
> same host).  One way to solve this, would be make the execution stream
> itself look at the block of code for a list for "file manifest" of
> sorts.  IMO, this is limited, so the proposal here is to let the block
> of code implementation itself do that.  To actually send the content
> over, the **execution stream** `send()` interface would be used.

Thanks for the explanation; it makes sense to me now. In my
understanding, this is a description of what avocado has to do, not
what a user of avocado (a test writer) needs to do. In other words,
this is describing the avocado internal architecture that supports
blocks of code.

As before, my confusion arose from my bias of expecting the RFC to
describe the multi stream proposal only from the test writer's point of
view, not the test writer and the avocado developer's viewpoints.

 
> For instance, consider a block of code implementation based on Python's
> binary "wheel" packages.  Those packages do not require recompilation
> (which would impose delays or limitations) on the execution stream, and
> will also pack dependencies.  Pseudo code:
> 
> class PythonWheel(BlockOfCode):
> 
>     def __init__(self, name):
>         self.name = name
> 
>     def send_self(self, execution_stream):
>         process.run("pip wheel -w OUTPUT_DIR %s" % self.name)
>         for wheel in os.glob("OUTPUT_DIR/*.whl"):
>             execution_stream.send(wheel)
> 
> > I think an architecture diagram would help me see how these various
> > layers fit together and interact with each other.
> > 
> >   
> 
> Sure, I will try to draw a diagram on this specific interaction.

Sure, or you can ignore it. I think I understand the concepts now so
it's not really necessary any longer.


> >> * ``run``: Starts the execution of the payload, and waits for the outcome
> >>   in a synchronous way.  The asynchronous support is handled at the
> >> Execution
> >>   Stream side.
> >> * ``success``: Reports the positive or negative outcome in a
> >>   simplistic but precise way.
> >> * ``output``: A dictionary of various outputs that may be generated by the
> >>   execution of the code.  The Execution Stream implementation may merge this
> >>   content with its own ``output`` dictionary, given an unified view of the
> >>   output produced there.
> >>
> >> Advanced topics and internals
> >> =============================
> >>
> >> Execution Streams
> >> -----------------
> >>
> >> An execution stream  was defined as a "disposable execution
> >> environment".  A "disposable execution environment", currently in the
> >> form of a fresh and separate process, is exactly what the Avocado
> >> test runner gives to a test in execution.
> >>
> >> While there may be similarities between the Avocado Test Process
> >> (created by the test runner) and execution streams, please note that
> >> the execution streams are created *by* one's test code.  The following
> >> diagram may help to make the roles clearer::
> >>
> >>    +-----------------------------------+
> >>    |       Avocado Test Process        |  <= created by the test runner
> >>    | +-------------------------------+ |
> >>    | | main execution stream         | |  <= executes your `test*()` method
> >>    | +-------------------------------+ |
> >>    | | execution stream #1           | |  <= initialized on demand by one's
> >>    | | ...                           | |     test code.  utilities to do so
> >>    | | execution stream #n           | |     are provided by the framework
> >>    | +-------------------------------+ |
> >>    +-----------------------------------+  
> > 
> > Yay, a picture! :-)
> > 
> > Some of my earlier questions about the test (is it a stream or not) are
> > now coming into focus. In particular, that the existing test code has a
> > particular status.
> > 
> >    
> 
> Yep, I'll try to come up with a few more.
> 
> >> Even though the proposed mechanism is to let the framework create the
> >> execution lazily (on demand), the use of the execution stream is the
> >> definitive trigger for its creation.  With that in mind, it's accurate
> >> to say that the execution streams are created by one's test code
> >> (running on the "main execution stream").
> >>
> >> Synchronous, asynchronous and synchronized execution
> >> ----------------------------------------------------
> >>
> >> As can be seen in the interface proposal for ``run``, the default
> >> behavior is to have asynchronous executions, as most observed use
> >> cases seem to fit this execution mode.
> >>
> >> Still, it may be useful to also have synchronous execution.  For that,
> >> it'd be a matter of setting the ``wait`` option to ``run``.
> >>
> >> Another valid execution mode is synchronized execution.  This has been
> >> thoroughly documented by the previous RFCs, under sections named
> >> "Synchronization".  In theory, both synchronous and asynchronous
> >> execution modes could be combined with a synchronized execution, since
> >> the synchronization would happen among the execution streams
> >> themselves.  The synchronization mechanism, usually called a "barrier",
> >> won't be given too much focus here, since on the previous RFCs, it was
> >> considered a somehow agreed and understood point.
> >>
> >> Termination
> >> -----------
> >>
> >> By favoring asynchronous execution, execution streams need to also
> >> have a default behavior for handling termination of termination
> >> of resources.  
> > 
> > Nit: It's the fact that asynchronous execution is offered at all that
> > causes this problem to arise. It doesn't matter whether it's the
> > favored execution model or not.
> > 
> > Suggestion: "By favoring asynchronous execution" --> "By offering an
> > asynchronous execution environment"
> >   
> 
> True.
> 
> > "termination of termination" is a circular phrase. Maybe something like
> > this would be better: "...execution streams need to also have a default
> > behavior for their termination that includes releasing of resources."
> >   
> 
> That was actually a typo.  I did not mean for the second "termination"
> to be there.  The sentence should be something like:
> 
> "By providing asynchronous execution model, execution streams need to
> also have a default behavior for handling termination of resources."
> 
> > However, please be very explicit about exactly what resources you are
> > reclaiming. (I recall a recent request to terminate child processes
> > created by a test; that request was denied. I don't think you mean to
> > reverse that decision, which is why I'm making this point.)
> >   
> 
> I think it's pretty hard to define them at this point, since they're
> supposed to be execution stream implementation specific.  For instance,
> one implementation may be backed by an SSH connection only.  In that
> case, what is created by the execution stream is a connection, and that
> should be terminated.
> 
> Another implementation may still use an SSH connection for transport,
> but also have support for provisioning of VMs to which it will connect
> to.  In that case, if it provisions a VM and connects to it, it should
> clean up both the VM and the connection it created.

Yes, but still you should have a dividing line beyond which
avocado will not cross (the shmem example given earlier). This goes
back to my bias of reading the RFC from the point of view of the test
writer.


> > Here's another suggested wording. It's probably too much,
> > but maybe you can get something useful from it:
> > 
> >   Synchronous execution streams are designed to terminate when their
> >   block of code exits; because they are synchronous, the termination of
> >   the test is blocked until the execution stream has completed.  
> 
> Correct, and good suggestion.
> 
> >   However, asynchronous execution streams may not explicitly terminate,
> >   but run "forever". Therefore, avocado must terminate an asynchronous
> >   execution stream when the test that created it terminates.
> > 
> >   
> 
> Nice! Pretty easy to read and understand.
> 
> >>                For instance, for a process based execution stream,
> >> if the following code is executed::
> >>
> >>   from avocado import Test
> >>   from avocado.streams.code import shell
> >>   import time
> >>
> >>   class MyTest(avocado.Test):
> >>       def test(self):
> >>           self.streams[0].run(shell("sleep 100"))
> >>           time.sleep(10)
> >>  
> > 
> > Nit: I thought stream numbers started at 1. s/streams[0]/streams[1]/.
> > 
> >   
> 
> As you can probably see on the other examples and discussions on other
> threads, we're still trying to shape the exact way to refer to execution
> streams from the (now so called) "streams manager".

Yep.

 
> >> The process created as part of the execution stream would run for
> >> 10 seconds, and not 100 seconds.  This reflects that execution streams
> >> are, by definition, **disposable** execution environments.  
> > 
> > This is twice that the characteristic "disposable" has been used and
> > I'm still not sure what it means.
> > 
> >   
> 
> It means that users should not expect to find the same execution stream
> across multiple tests.  They should not get attached to them.  There's
> no provision for persisting files on execution streams or even to bring
> back files from them.

OK.

 
> >> Execution streams are thus limited to the scope of one test, so
> >> implementations will need to terminate and clean up all associated
> >> resources.  
> > 
> > "all associated resources" -- again, I think you need to be precise
> > about what resources are being discussed. Saying "all" implies more
> > than I think you mean.
> > 
> >    
> 
> I think the proper wording here would probably be: all resources created
> by the execution stream itself.

I think by now I've beat this topic to death. :-)


> >> .. note:: based on initial experiments, this will usually mean that a
> >>           ``__del__`` method will be written to handle the cleanup.
> >>
> >> Avocado Utility Libraries
> >> -------------------------
> >>
> >> Based on initial evaluation, it looks like most of the features necessary
> >> to implement multi stream execution support can be architected as a set
> >> of utility libraries.
> >>
> >> One example of pseudo code that could be possible with this design::
> >>
> >>   from avocado import Test
> >>   from avocado.streams import get_implementation
> >>   from avocado.streams.code import shell
> >>
> >>   class Remote(Test):
> >>
> >>       def test_filtering(self):
> >>           klass = get_implementation("remote")
> >>           if klass is not None:
> >>               stream = klass(host=self.params.get("remote_hostname"),
> >>                              username=self.params.get("remote_username")
> >>                              password=self.params.get("remote_password"))
> >>               cmd = "ping -c 1 %s" % self.params.get("test_host_hostname")
> >>               stream.run(shell(cmd))
> >>  
> >   
> >> Please note that this is not the intended end result of this proposal, but
> >> a side effect of implementing it using different software layers.  Most
> >> users should favor the simplified (higher level) interface.  
> > 
> > I am trying to understand the purpose of this illustration. Based on
> > the remarks in the last paragraph above, I think you are trying to
> > demonstrate how to construct a test using the stream interface
> > described by this RFC (one that would be simpler to define using the
> > existing test interface).
> >   
> 
> This is basically a statement that the foundations of the multi stream
> feature will be, as much as possible, be implemented as reusable sets of
> utility modules.  By having those utility methods available, advanced
> users could leverage them to write tests using most of the same
> machinery, but in different ways then suggested by the "Avocado Multi
> Stream Tests" documentation (when such a document comes to existence).

This is a good approach.


> > Given this context, I'm trying to see if I understand how this code
> > behaves. And I remember (from the above definition) that run() defaults
> > to asynchronous execution, so should the 'wait' option be specified?
> > The ping must terminate since it has a count of 1. If wait is not
> > passed, then it's possible for the test to terminate before the ping
> > has been sent (or the response received).
> > 
> >   
> 
> Right, the example is indeed broken.  Actually, when used like that, it
> would need both the added `wait` (parameter or function) and a check of
> the result of `run()`.

Good, because it points out that at least I'm understanding some of
what the RFC describes! :-)


> >> Writing a Multi-Stream test
> >> ===========================
> >>
> >> As mentioned before, users have not yet been given tools **and
> >> guidelines** for writing multi-host (multi-stream in Avocado lingo)
> >> tests.  By setting a standard and supported way to use the available
> >> tools, we can certainly expect advanced multi-stream tests to become
> >> easier to write and then much more common, robust and better supported
> >> by Avocado itself.
> >>
> >> Mapping from parameters
> >> -----------------------
> >>
> >> The separation of stream definitions and test is a very important goal
> >> of this proposal.  Avocado already has a advanced parameter system, in
> >> which a test received parameters from various sources.The most common
> >> way of passing parameters at this point is by means of YAML files, so
> >> these will be used as the example format.  
> > 
> > Suggested:
> > s/stream definitions and test/stream and test abstractions/
> >   
> 
> IMO this suggestion actually brings a different meaning than the one I
> intended.

So maybe there's a grammar change needed instead. I got stuck on the
comparison of "stream definitions" (plural) with "test" (singular).

Would it hurt to drop this first sentence entirely? Maybe the
paragraph doesn't need it.


> 
> > Grammar changes:
> > s/a advanced/an advanced/
> > s/a test received parameters/a test receives/
> > 
> >   
> 
> ACK.
> 
> >> Parameters that match a predefined schema (based on paths and node
> >> names) will be by evaluated by a tests' ``streams`` instance
> >> (available as ``self.streams`` within a test).
> >>
> >> For instance, the following snippet of test code::
> >>
> >>   from avocado import Test
> >>
> >>   class MyTest(Test):
> >>       def test(self):
> >>           self.streams[1].run(python("import mylib; mylib.action()"))
> >>
> >> Together with the following YAML file fed as input to the parameter
> >> system::
> >>
> >>   avocado:
> >>      streams:
> >>       - 1:
> >>           type: remote
> >>           host: foo.example.com
> >>
> >> Would result in the execution of ``import mylib; mylib.action()``
> >> in a Python interpreter on host ``foo.example.com``.
> >>
> >> If test environments are refered to on a test, but have not been defined
> >> in the outlined schema, Avocado's ``streams`` attribute implementation
> >> can use a default Execution Stream implementation, such as a local process
> >> based one.  This default implementation can, of course, also be configured
> >> at the system and user level by means of configuration files, command line
> >> arguments and so on.  
> > 
> > Grammar & typo change:
> > s/refered to on a test/referred to by a test/
> > s/process based/process-based/
> >   
> 
> ACK.
> 
> >> Another possibility is an "execution stream strict mode", in which no
> >> default implementation would be used, but an error condition would be
> >> generated.  This may be useful on environments or tests that are
> >> really tied to their execution stream types.
> >>
> >> Intercommunication Test Example
> >> -------------------------------
> >>
> >> This is a simple example that exercises the most important aspects
> >> proposed here.  The use case is to check that different hosts can
> >> communicate among themselves.  To do that, we define two streams as
> >> parameters (using YAML here), backed by a "remote" implementation::
> >>
> >>   avocado:
> >>      streams:
> >>       - 1:
> >>           type: remote
> >>           host: foo.example.com
> >>       - 2:
> >>           type: remote
> >>           host: bar.example.com
> >>
> >> Then, the following Avocado Test code makes use of them::
> >>
> >>   from avocado import Test
> >>   from avocado.streams.code import shell
> >>
> >>   class InterCommunication(Test):
> >>       def test(self):
> >>           self.streams[1].run(shell("ping -c 1 %s" % self.streams[2].host))
> >>           self.streams[2].run(shell("ping -c 1 %s" % self.streams[1].host))
> >>           self.streams.wait()
> >>           self.assertTrue(self.streams.success)
> >>
> >> The ``streams`` attribute provide a aggregated interface for all the
> >> streams.
> >> Calling ``self.streams.wait()`` waits for all execution streams (and their
> >> block of code) to finish execution.  
> > 
> > Grammar:
> > s/a aggregated/an aggregated/
> > 
> >   
> 
> ACK.
> 
> >> Support for slicing, if execution streams names based on integers only could
> >> be added, allowing for writing tests such as::  
> > 
> > Slicing is a new concept that has not yet been defined. What is it?
> > 
> >   
> 
> Sorry for not mentioning.  I borrowed the Python slicing concept here:
> 
> * https://docs.python.org/2.3/whatsnew/section-slices.html
> * https://docs.python.org/2/tutorial/introduction.html#lists
> 
> But, on another thread, we've talked a lot about extending the
> capabilities here.
> 

Yes, I intentionally wrote my feedback on the original proposal without
looking at the comments in the other thread.


> >>   avocado:
> >>      streams:
> >>       - 1:
> >>           type: remote
> >>           host: foo.example.com
> >>       - 2:
> >>           type: remote
> >>           host: bar.example.com
> >>       - 3:
> >>           type: remote
> >>           host: blackhat.example.com
> >>       - 4:
> >>           type: remote
> >>           host: pentest.example.com
> >>
> >>   from avocado import Test
> >>   from avocado.streams.code import shell
> >>
> >>   class InterCommunication(Test):
> >>       def test(self):
> >>           self.streams[1].run(shell("ping -c 1 %s" % self.streams[2].host))
> >>           self.streams[2].run(shell("ping -c 1 %s" % self.streams[1].host))
> >>           self.streams[3].run(shell("ping -c 1 %s" % self.streams[1].host))
> >>           self.streams[4].run(shell("ping -c 1 %s" % self.streams[1].host))
> >>           self.streams.wait()
> >>           self.assertTrue(self.streams[1:2].success)
> >>           self.assertFalse(self.streams[3:4].success)
> >>
> >> Support for synchronized execution also maps really well to the
> >> slicing example.  For instance, consider this::
> >>
> >>   from avocado import Test
> >>   from avocado.streams.code import shell
> >>
> >>   class InterCommunication(Test):
> >>       def test(self):
> >>           self.streams[1].run(shell("ping -c 60 %s" % self.streams[2].host)
> >>           self.streams[2].run(shell("ping -c 60 %s" % self.streams[1].host))
> >>           ddos = shell("ddos --target %s" self.streams[1].host)
> >>           self.streams[3:4].run(ddos, synchronized=True)
> >>           self.streams[1:2].wait()
> >>           self.assertTrue(self.streams.success)
> >>
> >> This instructs streams 1 and 2 to start connectivity checks as soon as
> >> they **individually** can, while, for a full DDOS effect, streams 3
> >> and 4 would start only when they are both ready to do so.
> >>
> >> Feedback and future versions
> >> ============================
> >>
> >> This being an RFC, feedback is extremely welcome.  Also, exepect new
> >> versions
> >> based on feedback, discussions and further development of the ideas
> >> initially
> >> exposed here.
> >>  
> > 
> > Final thoughts: I like this proposal and think it has a lot of good
> > ideas. It really does a lot to advance the architecture and
> > usefulness of avocado.
> > 
> > One thing I started noticing towards the middle of the RFC is that the
> > term 'execution stream' seems to be overloaded; it has two meanings.
> > First, it's used as the overall name given to this entire concept.
> > Second, it's used as the name of a particular attribute (the "where")
> > so that it can be reasoned about separately from the block of code (the
> > "what"). Usually I can determine which meaning is intended by the
> > context in which it is used. But this was sometimes challenging. I'm
> > wondering if one or the other concepts should have a different name. 
> >   
> 
> I think I know how to address that.  Some key players on this first
> version, such as the test's stream manager, were missing formal names.
> 
> Also, a few more architecture diagrams may help to show the difference
> between different components that may look overlapping.
> 
> > Thanks.
> > 
> > -Jeff
> >   
> 
> Thank you for the thorough feedback!

You're welcome. Thanks for addressing the comments!

-Jeff