[Avocado-devel] RFC: Plugin execution order

Tue Oct 4 15:00:56 UTC 2016

On Fri, Sep 30, 2016 at 07:24:53AM -0300, Cleber Rosa wrote:
> 
> On 09/29/2016 04:14 PM, Jeff Nelson wrote:
> > On Thu, Sep 29, 2016 at 01:24:22PM -0300, Cleber Rosa wrote:
> >> As detailed in the following card:
> >>
> >>  https://trello.com/c/oWzrV48E/837-execution-order-support-in-plugins
> >>
> >> It should be possible to specify a custom order for plugins to be
> >> executed by setting specific configuration.
> >>
> >> The first observed approach would be to create a section
> >> called `[plugins.<type>]` where the `<type>`  conforms to the
> >> description on fully qualified plugin names given here:
> >>
> >>
> >> https://github.com/avocado-framework/avocado/pull/1495/commits/193a10ce98cb5747395eefcb485dd452696b4b11#diff-0f4f89ace79fa15278d9b283c2d9d9b2R84
> >>
> >>
> >> Then, by creating a key named `order`, containing the short names as a
> >> list.  Enabled plugins not listed will be executed *after* plugins
> >> listed, but in non-determined order.
> > 
> > The phrase "enabled plugins" implies that there can be disabled
> > plugins as well.
> > 
> >
> 
> 
> Yeah, support for that has just been added:
> 
> http://avocado-framework.readthedocs.io/en/latest/Plugins.html#disabling-a-plugin
> 
> > 
> >> For instance, consider the following entry points::
> >>
> >>  'avocado.plugins.result' : [
> >>     'xunit = avocado.plugins.xunit:XUnitResult',
> >>     'json = avocado.plugins.jsonresult:JSONResult',
> >>     'archive = avocado.plugins.archive:Archive',
> >>     'mail = avocado.plugins.mail:Mail',
> >>     'html = avocado_result_html:HTMLResult'
> >>   ]
> >>
> >> We can say that:
> >>
> >> * The plugin type, according to the fully qualified plugin name
> >>  definition here is `result`.
> >>
> >> * The plugin fully qualified names are:
> >>  - result.xunit
> >>  - result.json
> >>  - result.archive
> >>  - result.mail
> >>  - result.html
> >>
> >> * The short names for plugins of type "result" are:
> >>  - xunit
> >>  - json
> >>  - archive
> >>  - mail
> >>  - html
> > 
> > I like this use of examples. The illustrations are clear and easy to
> > understand.
> > 
> > 
> 
> Thanks.
> 
> >> To make sure that the mail plugin is run after (and thus includes)
> >> the HTML result, the following configuration entry can be set::
> >>
> >>  [plugins.result]
> >>  order = html, archive
> > 
> > What does it mean for a plugin to "include" the results of an earlier
> > plugin?
> > 
> > 
> 
> I meant that a imaginary "mail" plugin, would include all previously
> generated results, so it would include the HTML report.  That wasn't a
> really good example when it comes to *core* functionality, that is, the
> plugin system wouldn't by its own do any of those inclusions.
> 
> >> The other result plugins, namely xunit, json and mail, will still
> >> be run.  It's guaranteed they'll be run *after* the other result
> >> plugins.  The order in which they'll run after the explicitly
> >> ordered plugins is undefined.
> > 
> > Can a plugin determine what plugin(s) have run before it? (I don't
> > think it's necessary.)
> > 
> > 
> 
> In theory, a plugin could look at the configuration system and get that
> order.  But, I don't think that's going to be a common use case.
> 
> >> Other possible approach
> >> -----------------------
> >>
> >> The other approach possible, would require a default order value
> >> for plugins.  This would still preferably be done in configuration
> >> rather than in code.  Then, the fully qualified name for a plugin could
> >> be used as part of the configuration section.  Example::
> >>
> >>  [plugin.result.archive]
> >>  order = 50
> >>
> >>  [plugin.result.html]
> >>  order = 30
> > 
> > Small typo: "plugins.result" not "plugin.result" (two lines).
> > 
> > 
> 
> Yep, thanks!
> 
> >> This would make the `html` plugin run before the `archive` plugin.
> >> While more verbose, it would allow for external plugins to ship with
> >> stock configuration files that would set, by default, its ordering.
> > 
> > Order is here is used as a numerical value to indicate a relative
> > ordering, correct? I'm not sure I like the name "order"; how about
> > "sequence"?
> > 
> > A drawback to this approach is that you still have to come up with a
> > rule for what happens when two plugins have the same sequence number.
> > 
> 
> Then the order is undefined.  I don't see a clean way around this.
> 
> >> Feedback is highly appreciated!
> > 
> > Another way to specify the order is to use an attribute at the
> > [plugins] or [plugins.result] level with certain expected values:
> > 
> > [plugins]
> > execution-order = random | lexical | user-defined
> > 
> > where 'user-defined' would require yet another attribute
> > that defines the sequence. 'user-defined' would also have to
> > handle the 'unspecified' condition.
> > 
> > I think this is too complicated, but I offer it as a counter-example.
> > 
> 
> Random is default, and I fail to see the practical use of lexical.
> About user-defined, I was trying to avoid (at this point) code based
> ordering.

Please be careful with the wording here: I think you mean
arbitrary or undefined, not random.

If you're not purposefully implementing random ordering (by
reading from an entropy source such as /dev/random), then it's
simply undefined or arbitrary (dependent of an external,
undefined source, such as the file-system ordering, or the
behavior of a library).

The problem with undefined behavior is that users might start
trusting it if they always see the same behavior in practice. For
example, plugins will always execute in the same order if the
user is not touching the file-system. Then, the order *might*
change when the user upgrades avocado.

If you're implementing plugin ordering, the default should be
something stable and predictable.

Thanks.
   - Ademar

> 
> > Finally, the typo got me to thinking: should it be "plugin" or
> > "plugins"? I don't care one way or the other. However, I do believe
> > that the attribute should be named consistent with the rest of
> > avocado. If there isn't a style guide for avocado proposals and naming
> > conventions, it would be good to have one. Consistency is going to be
> > hard to establish and maintain without it.
> > 
> 
> I would go with "plugins", because we already have a global section that
> configures "all plugins behavior", and that section is called "plugins".
>  Then, I see "plugins.<type>" as "sub section" (logically only) of that
> main one, that configures plugins of a given type.
> 
> > -Jeff
> > 
> > 
> 
> Thanks for the feedback!
> 
> >>
> >> -- 
> >> Cleber Rosa
> >> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> >> [ Avocado Test Framework - avocado-framework.github.io ]
> >>
> 
> -- 
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> 

-- 
Ademar Reis
Red Hat

^[:wq!