[Avocado-devel] Post 82.0 LTS plans

[Cleber Rosa]

On Sat, Sep 26, 2020 at 11:06:44PM +0300, Plamen Dimitrov wrote:
> Hi Cleber,
> 
> First of all thanks for summarizing the 82.0 LTS aftermaths and your
> current perspective on what comes next.
> 
> On 2020-09-16 22:02, Cleber Rosa wrote:
> > Hi Avocado community,
> > 
> > This is an invitation to a conversation about Avocado's present and
> > future.
> > 
> > If you missed Avocado's latest LTS release (82.0 \o/), these are the
> > two documents you may want to look at:
> > 
> >  * https://avocado-framework.readthedocs.io/en/82.0/releases/lts/82_0.html
> > 
> >  * https://avocado-framework.readthedocs.io/en/82.0/releases/82_0.html
> > 
> > There are many co-related topics, so instead of firing multiple
> > e-mails, I'll try to address all of them here, while using section
> > names to keep individual topics in check.  Feel free to comment on the
> > section that matters to you and ignore others.
> > 
> > 69.x LTS
> > ========
> > 
> > The general rule that the previous LTS version will be maintained for
> > a minimum of 6 months following another LTS release.  Following that
> > rule, we would be maintaining the 69.x LTS series until March 2021.
> > 
> > One important characteristic of the 69.x series is that it supports
> > Python 2, which has been declared EOL on Dec 2019.  So users stuck
> > with Python 2 can only use 69.x LTS.  In theory, the March 2021 date
> > far exceeds the Python 2 EOL date, so we suppose that it's a safe
> > date.
> 
> +1
> 
> > Avocado-VT and Avocado Compatibility
> > ====================================
> > 
> > Avocado-VT is Avocado's primary "customer" and the two share a common
> > origin, and one retro feeds improvements into the other.  Having said
> > that, a lot of energy has been spent into keeping Avocado-VT compatible
> > with multiple Avocado of versions.  At any given moment, Avocado-VT is
> > supposed to be compatible with:
> > 
> > 1. Avocado 69.x LTS
> > 2. Avocado Latest release (currently 82.0)
> > 3. Avocado development branch
> > 
> > With Avocado 82.0 LTS, there's an opportunity to improve this
> > situation.  The first drastic change I'll propose is to drop as many
> > of the compatibility requirements above, and pin Avocado-VT for the
> > time being to require compatibility Avocado 82.0 LTS *only*.  This
> > would greatly simplify Avocado-VT's development, as there would be
> > one and only way to deal with Avocado.
> 
> What doesn't remain clear to me is what is the planned versioning to
> follow from such an arrangement? Is Avocado VT going to be tagged in
> synchrony with Avocado releases still considering that the such tagging
> does not imply compatibility or give any other information? Or will
> Avocado VT follow different numbering? Or just the LTS tags? How about
> packaging to follow from such tagging? Will we have for instance an
> 82.0-versioned VT RPM to install alongside an 83 Avocado RPM?
>

That's an interesting point.  So far, the releases of Avocado-VT are
not done in a way that a roadmap is strictly followed or a number of
specific issues have to be completed before its releases.  Avocado-VT
releases are just a convenience tag/release events that go through
some compatibility testing.

So, even though this proposal means that Avocado-VT will be used with
the LTS release only, the same convenience tag/release and
compatibility testing (beyond the CI level) can continue to be done.

In short, I think Avocado-VT 83.0 (and then 84.0, 85.0, etc), can go
through the same release process, and be made available to users in
packaged form, even if it's meant to be used with Avocado 82.0 LTS.

> > On the other hand, this could cause:
> > 
> > 1. Bitrot, where Avocado-VT would at a given point not work with
> >    newer Avocado, say the next LTS version.
> > 
> > 2. A "lock down", where improvements to "avocado.utils" libraries
> >    would not be easily available to Avocado-VT tests.
> 
> I am also concerned about the opposite direction - how easy it will be
> to migrate fully implemented and application stable utilities from VT
> to Avocado? In a previous conversation some of us have had but I also
> say here for clarity to everyone in the mailing list, I think it is a
> good idea to migrate more utilities from Avocado VT to the larger usage
> scope of Avocado. This will (1) reduce the volume and maintenance cost
> of Avocado VT, (2) make the possibility of compatibility with Avocado
> even easier (as more of the code and improvements will immediately end
> up on the Avocado side), and (3) make it easier to differentiate which
> functionality of Avocado VT still remains irreplaceable by pure Avocado
> like VM isolation model, etc.
>

ACK, these are all valid concerns and noteworthy points.

> > Avoiding bitrot
> > ---------------
> > 
> > Let's assume that the next LTS release is version 95.0.  The worst
> > possible scenario would be "waking up" and finding out that making
> > Avocado-VT compatible Avocado 95.0 would be overwhelming and
> > prohibitive mission.
> > 
> > One mitigation would be to extend 82.0 LTS life time, but this would
> > only push the problem further away and not really solve it.
> > 
> > To avoid such a situation, I propose that a proactive effort is
> > made every few Avocado releases, with the goal of:
> > 
> > * evaluate the current compatibility
> > * plan and develop compatibility tasks
> > 
> > Those should be done not only in light of the *current* Avocado state,
> > but also in sync with Avocado's planned features.  For example,
> > consider the following release time line:
> > 
> > * Avocado 83.0 released
> > * Avocado 84.0 released
> > * Avocado 85.0 released
> > * Avocado-VT compatibility evaluated with Avocado 85.0
> >  - Compatibility with newer Avocado "runner API" identified and
> >    issue created
> >  - Compatibility with newer Avocado "result API" identified and
> >    issue created
> > * Avocado 86.0 is released
> >  - Compatibility with "runner API" resolved
> > * Avocado 87.0 is released
> >  - Compatibility with "result API" resolved
> > * Avocado 88.0 released
> > * Avocado-VT compatibility evaluated with Avocado 88.0
> >   ...
> > * Avocado 94.0 released (pre LTS release)
> >  - All pending compatibility issues are addressed
> > * Avocado 95.0 released
> >  - Avocado-VT should be compatible with both current 95.0 LTS and 82.x
> >    LTS
> 
> I am not really sure how beneficial this process will end up being. On one
> side it is more complex than simply fixing smaller (if discovered earlier)
> incompatibilities between VT and Avocado. On the other it does reduce the
> number of checks one has to do. I guess one of the main benefits of going
> this way remains reduction of context switching and more clumped batch-oriented
> fixes at certain checkpoints that won't require divided focus. At the same
> time it remains true that with longer periods between syncs the issues will
> pile up much more and to some extend may become harder to resolve and overcome.
>

Besides the context switching you described, the problem with the
current per-commit lockstep compatibility approach requires extra
development that may not be needed at all with a coarser compatibility
requirement.  So, changes that happen between, say, Avocado 82.0 and
83.0 may require compatibility changes to Avocado-VT.  Then, on
Avocado 84.0, more work on the same area is done, and more
compatibility related changes are required on the Avocado-VT side.
These could really be one set of changes to ensure Avocado-VT
compatibility with Avocado 84.0.

We could work around this on the Avocado side, by:

1) having very long sprints;

2) having a "next" branch that introduces changes without the promise
of compatibility, and delay sending changes to master

But this is really a bad way of faking the real impacts of what
happens when other software depends on another software master branch
and would impact all other Avocado users.

I know this proposal is not a solution that makes everybody thrilled,
but it's the best compromise I can think of.

> > Integration jobs on CI can also keep running with Avocado master, but
> > in non-gating state ("allow failures" in Travis CI terminology).  This
> > can serve as a "temperature check" for the level of compatibility and
> > tasks to records as issues and address in the near future.
> 
> Another thing this section doesn't clarify much are the really long term plans
> regarding Avocado VT. Is it planned to be supported many more LTS-es to come
> or wished to be replaced with the newer isolation models of Avocado? Knowing
> the end-goal can help us judge better what is a better direction in the present.
>

I really think Avocado-VT will be around for many more LTS-es.  The
value it has in itself, and in the form of the coverage provided by
tp-qemu and tp-libvirt (and others) can not be replicated easily.

But, we see a trend and push to have tests being written as early as
possible, if possible at the upstream projects by the original author
of the features.  In this case, it's *not* easy to leverage Avocado-VT.

The win-win situation is where both the upstream developer writing a
"bare Avocado" test, and Avocado-VT users can use as much as possible
common code, while preserving their inherent differences at the
framework level.  Here I'm talking specially about the the utility
libraries, but also some Avocado test runner (and related tooling).

> > Library lock downs
> > ------------------
> > 
> > Users running tests can usually wait a reasonable long time for new
> > features to land in the Avocado test runner.  In contrast, fixes or
> > new features in utility libraries are usually needed to write a test
> > and can not wait for too long.
> > 
> > With the proposal above to bind Avocado-VT to Avocado LTS versions
> > only, utility libraries present within Avocado that is,
> > "avocado.utils", would also be limited to the LTS version.  This is
> > not desirable, as it can:
> > 
> > * Slow down test development
> > 
> > * Create duplicate copies of utilities:
> >  - within different tests
> >  - across Avocado and Avocado-VT
> 
> Yes, even at present there are some problems like this - duplicated utilities
> and VT tests across repos. So making repos even more out of sync could only
> exacerbate this unless we have a proper remedy.
>

Yes.

> > Moving all utility libraries into Avocado-VT (such as the one currently
> > available in Avocado) is also not an option, as users of other tests
> > (for example avocado-misc-tests) and even Avocado itself need them.
> 
> The same doesn't hold for moving utilities from VT to Avocado though which
> also has the benefit of fewer projects to manage. Side repos for such components
> often end up under-maintained and impossible to contribute to.
>

The possibility of under-maintanance and overall low quality of such
code is exactly what I hope to address here.  The proposal of a new
repo, with *high* standards and the right tools to help achieve those
standards, is what I suggested below.

> > The most obvious solution is to move the utility libraries to its own
> > repository.  Let's suppose this new repository/module is called
> > "greenmatter" (my imagination ran wild, please do the same ;).  This
> > would allow for scenarios such as:
> > 
> > 1. a) Avocado-VT master
> >    b) Avocado LTS (with its builtin current version of "avocado.utils")
> >    c) new tp-qemu tests using newer "greenmatter" utility libraries
> > 
> > 2. a) avocado-misc-tests master
> >    b) Avocado (either LTS or latest)
> >    c) new tests using newer "greenmatter" utility libraries
> 
> So to get this straight, using another repository for utilities (called here
> "utilitarianism" to add to the diversity of names) will still require a certain
> level of compatibility between Avocado and VT, this time however indirect as it
> reduces to compatibility between VT and "utilitarianism" and compatibility between
> "utilitarianism" and Avocado.
>

Right, it will require compatibility handling, but only by the project
that uses it, and only on that direction.  What I mean is, once this
module releases version 1.0, Avocado is free to adopt version 1.0, and
switch from using some of its "avocado.utils" libraries to some of
those in the new module.  It may take Avocado-VT more time to adopt
it, and then adopt version 3.0 (or the other way around).

The end goal, however, is for all projects to move utility libraries
to this new repo, and update their compatibility often.

> Of course, maintaining such compatibility is not exactly half the cost but even
> less considering the typical changes such utilities undergo where breakage could
> be mostly API-related and not due to conceptual differences. In the end the choice
> of different repository vs using Avocado doesn't matter much for me but I wanted
> to point out some pros and cons.
>

Right.  The API changes are usually mechanic as you suggested and be
commited with the new version requirements.  It's quite different than
the current approach where Avocado-VT (and its tests) is supposed be
compatible with all Avocado version... which leads to keeping its own
libraries (in exchange for developer sanity :).

> > Goals of "greenmatter" libraries
> > --------------------------------
> > 
> > "tried, tested and trusted"
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > The current utility libraries suffer a big problem.  They are either:
> > 
> > * tested by superficial tests with lots of mocking
> > * "tested" by the tests that actually have another goal and simply
> >   use the libraries
> > * are not tested at all
> > 
> > It's challenging to test the kind of libraries in "virttest.utils_*"
> > or in "avocado.utils" because they require specific environments, and
> > can be destructive.  For instance, not many people will trust
> > (rightfully so) running tests on their local machines.
> > 
> > On the other hand, Avocado is in the business of:
> > 
> > 1) Running tests isolated (see the nrunner spanwers)
> > 2) Checking, fulfilling and caching test requirements (see BP002)
> > 
> > The idea is to leverage and extend Avocado's features so that every
> > single new utility library in this new repository gets real test
> > coverage.  Examples:
> > 
> > 1. a) "avocado.utils.lv_utils" becomes "greenmatter.lv"
> >    b) "greenmater.lv" gets a "tests/test_lv.py" module
> >    c) "tests/test_lv.py" describes its requirements, such as:
> >       - A software package (from the distribution): lvm2
> >       - A specific isolation level for the spawner: vm
> >       - An empty block device of around 10G
> > 
> > 2. a) "avocado.utils.distro" becomes "greenmatter.distro"
> >    b) "tests/test_distro.py:Distro.test_fedora" describe its requirements:
> >       - Operating system: fedora:32
> >    c) "tests/test_distro.py:Distro.test_centos" describe its requirements:
> >       - Operating system: centos:8
> > 
> > For example #2, the requirement for "fedora:32" could be fulfilled by
> > the current environment, and the spawner selected could end up being
> > "process".  In that case, the requirement for "centos:8" would need
> > to be fulfilled by another spawner, such as a container or vm based
> > spawner.
> 
> This might couple the choice of spawner too much with the given test requirement
> (for instance a vm spawner should satisfy pretty much all requirements above in
> the most isolated manner possible and the code should make some decisions about
> reduction of isolation in order to choose a process) and we might rather be
> willing to give the choice of spawner and thus isolation model to the user. However,

Agreed... Firing up a separate process for each test is already very
costly when compared to how, say, what Python's unittest runner does.
If you extrapolate that to VMs, you can end up with test jobs so slow
that you don't want to run them.

So I think the user must have control of the spawner/isolation, but a
sane default that uses the least complex but satisfiable
implementation should be used (if at all possible to decide on that).

> I digress here and focusing back on testing the utilities themselves, this could
> be viable if we don't reach a chicken-or-egg problem. We could avoid it by using
> k-fold self-testing where all k-1 (but the tested) utilities could be used in
> the testing of a given utility or use some incremental self-compiling technique.
>

This sounds interesting.  TBH, my original, poor guy's approach, was
to think of the tooling (say Avocado + Job definition + Spawners code)
as a separate instalation.  So the tooling used to test the libraries
would be using a non-development version of the libraries.  It could
be interesting indeed to add a testing in which the tooling *also*
uses the libraries under test, to get a sense of the compatibility,
and maybe even use that result as a self-promotion (bumping the
version) of the libraries used on the tooling.

Is this more or less what you had in mind?

> > The resulting test coverage for each module should be generated and
> > presented automatically, such as it's currently being done with the
> > (per release) coverage for the "avocado.utils.vmimage" library:
> > 
> >   https://avocado-framework.readthedocs.io/en/82.0/guides/writer/libs/vmimage.html#supported-images
> > 
> > It should be possible for a user to "shop" for a library, such as
> > "lv", and find out that in which environments it has been tested on.
> 
> This is something new that wasn't clear above: I guess you mean testing a utility
> in more than one environments? Is such testing planned to be part of the CI for
> these utilities? Or a devoted hardware? If I wish to inspect output of tests for
> a given utility where should I look for them?
>

Yes, I absolutely think we should test the utility libraries on more
than one environment, and yes, until proven impossible, at the CI level.
For instance, a matrix like the following:

   +--------+-----------+--------------+------------+
   | module | fedora 31 | ubuntu 18.04 | centos 8.2 |
   +--------+-----------+--------------+------------+
   | distro |     V     |       V      |      V     |
   +------------------------------------------------+
   |   lv   |     V     |       V      |      X     |
   +------------------------------------------------+

Could be compiled from the "results.json" of tests, with some extended
metadata about the test environments.  Most CI environments will let
you save some artifacts, and either the "results.json" itself or some
further processed file could be used for that.  Options for hosting
this:

 1) A dedicated web page that renders those results from a database
 2) A mechanism that generates a new pull request and updates the
    documentation

> Also, what is meant by "shop" more specifically? If one close the given repository
> they will obtain the full set of utilities, what does an action of "selecting" a
> utility entail in this sense?
>

By shopping for a library, I mean that, were I about to write a test
that deals with logical volumes, and I intend the test to run on
CentOS 8.2 and Fedora 31, I would be able to quickly confirm that,
instead of calling "pv*/vg*/lv*" command line utilities manually,
I can leverage the "lv" module.

> > Common API "look and feel"
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > Getting the design of APIs right is pretty hard, but here we have a
> > unique change of declaring a common set of guidelines from the start
> > and enforcing as libraries are added.
> > 
> > It's expected that a "BluePrint" would be written defining, among other
> > things:
> > 
> >  * how modules should be named
> >  * how classes should be named
> >  * how functions should be named if they return information
> >  * how functions should be named if they check or assert some condition
> >  * how functions should behave if running on a platform that doesn't
> >    apply (say "distro" is found running on a BSD or Windows system)
> 
> This sounds more like contribution guidelines than the current blueprints which
> discuss new features or large refactoring in Avocado. I guess you mean this as
> a venue for discussion in which sense I wonder do you mean this for the already
> established requirements for the Avocado and VT projects or for the utility repo?
>

You're right that the current Blue Prints on the Avocado project
describe technical features.  But, BP000 is due to describe the
overall usage of the Blue Prints.  Maybe Blue Prints can and should
include guidelines too.

The take away is that I'm initially interested in setting a much
better thought out, and consequently stricter, set of contribution
*requirements* for this new repo.

> > Concurrent usage and migration plan
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 
> > It should be possible to keep using the libraries available at
> > Avocado-VT's "virtest.utils_*" and Avocado's "avocado.utils"
> > concurrently with the libraries migrated to the new repository.
> 
> Yes, I think this is an essential part of any large scale migration.
> 
> > Suppose your test looks like:
> > 
> >    from avocado.utils import cpu
> >    from avocado.utils import lv_utils
> >    from avocado.utils import distro
> >    from virttest import utils_net
> > 
> > Once "greenmatter 1.0" gets release, let's say with nothing but "lv" and
> > "distro" libraries.  You project can add a new requirement:
> > 
> >    greenmatter==1.0
> > 
> > And the next version of a test can look like:
> > 
> >    from avocado.utils import cpu
> >    from virttest import utils_net
> >    from greenmatter import lv
> >    from greenmatter import distro
> > 
> > Gradually, as more modules are migrated, the requirement can eventually be:
> > 
> >    greenmatter==12.0
> > 
> > And the test can look like:
> > 
> >    from greenmatter import cpu
> >    from greenmatter import net
> >    from greenmatter import lv
> >    from greenmatter import distro
> 
> +1
> 
> > 
> > That's all for now.  All feedback is appreciated.
> > 
> > Thanks!
> 
> I hope some of my comments and raised questions are useful and we get some more
> opinions. Thanks for reaching out for feedback like this!
> 
> Plamen
> 

They do! And thanks a lot for the suggestions and very much valid
points!

- Cleber.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20200928/109ddf68/attachment.sig>