[Avocado-devel] Post 82.0 LTS plans

Sat Sep 26 20:06:44 UTC 2020

Hi Cleber,

First of all thanks for summarizing the 82.0 LTS aftermaths and your
current perspective on what comes next.

On 2020-09-16 22:02, Cleber Rosa wrote:
> Hi Avocado community,
> 
> This is an invitation to a conversation about Avocado's present and
> future.
> 
> If you missed Avocado's latest LTS release (82.0 \o/), these are the
> two documents you may want to look at:
> 
>  * https://avocado-framework.readthedocs.io/en/82.0/releases/lts/82_0.html
> 
>  * https://avocado-framework.readthedocs.io/en/82.0/releases/82_0.html
> 
> There are many co-related topics, so instead of firing multiple
> e-mails, I'll try to address all of them here, while using section
> names to keep individual topics in check.  Feel free to comment on the
> section that matters to you and ignore others.
> 
> 69.x LTS
> ========
> 
> The general rule that the previous LTS version will be maintained for
> a minimum of 6 months following another LTS release.  Following that
> rule, we would be maintaining the 69.x LTS series until March 2021.
> 
> One important characteristic of the 69.x series is that it supports
> Python 2, which has been declared EOL on Dec 2019.  So users stuck
> with Python 2 can only use 69.x LTS.  In theory, the March 2021 date
> far exceeds the Python 2 EOL date, so we suppose that it's a safe
> date.

+1

> Avocado-VT and Avocado Compatibility
> ====================================
> 
> Avocado-VT is Avocado's primary "customer" and the two share a common
> origin, and one retro feeds improvements into the other.  Having said
> that, a lot of energy has been spent into keeping Avocado-VT compatible
> with multiple Avocado of versions.  At any given moment, Avocado-VT is
> supposed to be compatible with:
> 
> 1. Avocado 69.x LTS
> 2. Avocado Latest release (currently 82.0)
> 3. Avocado development branch
> 
> With Avocado 82.0 LTS, there's an opportunity to improve this
> situation.  The first drastic change I'll propose is to drop as many
> of the compatibility requirements above, and pin Avocado-VT for the
> time being to require compatibility Avocado 82.0 LTS *only*.  This
> would greatly simplify Avocado-VT's development, as there would be
> one and only way to deal with Avocado.

What doesn't remain clear to me is what is the planned versioning to
follow from such an arrangement? Is Avocado VT going to be tagged in
synchrony with Avocado releases still considering that the such tagging
does not imply compatibility or give any other information? Or will
Avocado VT follow different numbering? Or just the LTS tags? How about
packaging to follow from such tagging? Will we have for instance an
82.0-versioned VT RPM to install alongside an 83 Avocado RPM?

> On the other hand, this could cause:
> 
> 1. Bitrot, where Avocado-VT would at a given point not work with
>    newer Avocado, say the next LTS version.
> 
> 2. A "lock down", where improvements to "avocado.utils" libraries
>    would not be easily available to Avocado-VT tests.

I am also concerned about the opposite direction - how easy it will be
to migrate fully implemented and application stable utilities from VT
to Avocado? In a previous conversation some of us have had but I also
say here for clarity to everyone in the mailing list, I think it is a
good idea to migrate more utilities from Avocado VT to the larger usage
scope of Avocado. This will (1) reduce the volume and maintenance cost
of Avocado VT, (2) make the possibility of compatibility with Avocado
even easier (as more of the code and improvements will immediately end
up on the Avocado side), and (3) make it easier to differentiate which
functionality of Avocado VT still remains irreplaceable by pure Avocado
like VM isolation model, etc.

> Avoiding bitrot
> ---------------
> 
> Let's assume that the next LTS release is version 95.0.  The worst
> possible scenario would be "waking up" and finding out that making
> Avocado-VT compatible Avocado 95.0 would be overwhelming and
> prohibitive mission.
> 
> One mitigation would be to extend 82.0 LTS life time, but this would
> only push the problem further away and not really solve it.
> 
> To avoid such a situation, I propose that a proactive effort is
> made every few Avocado releases, with the goal of:
> 
> * evaluate the current compatibility
> * plan and develop compatibility tasks
> 
> Those should be done not only in light of the *current* Avocado state,
> but also in sync with Avocado's planned features.  For example,
> consider the following release time line:
> 
> * Avocado 83.0 released
> * Avocado 84.0 released
> * Avocado 85.0 released
> * Avocado-VT compatibility evaluated with Avocado 85.0
>  - Compatibility with newer Avocado "runner API" identified and
>    issue created
>  - Compatibility with newer Avocado "result API" identified and
>    issue created
> * Avocado 86.0 is released
>  - Compatibility with "runner API" resolved
> * Avocado 87.0 is released
>  - Compatibility with "result API" resolved
> * Avocado 88.0 released
> * Avocado-VT compatibility evaluated with Avocado 88.0
>   ...
> * Avocado 94.0 released (pre LTS release)
>  - All pending compatibility issues are addressed
> * Avocado 95.0 released
>  - Avocado-VT should be compatible with both current 95.0 LTS and 82.x
>    LTS

I am not really sure how beneficial this process will end up being. On one
side it is more complex than simply fixing smaller (if discovered earlier)
incompatibilities between VT and Avocado. On the other it does reduce the
number of checks one has to do. I guess one of the main benefits of going
this way remains reduction of context switching and more clumped batch-oriented
fixes at certain checkpoints that won't require divided focus. At the same
time it remains true that with longer periods between syncs the issues will
pile up much more and to some extend may become harder to resolve and overcome.

> Integration jobs on CI can also keep running with Avocado master, but
> in non-gating state ("allow failures" in Travis CI terminology).  This
> can serve as a "temperature check" for the level of compatibility and
> tasks to records as issues and address in the near future.

Another thing this section doesn't clarify much are the really long term plans
regarding Avocado VT. Is it planned to be supported many more LTS-es to come
or wished to be replaced with the newer isolation models of Avocado? Knowing
the end-goal can help us judge better what is a better direction in the present.

> Library lock downs
> ------------------
> 
> Users running tests can usually wait a reasonable long time for new
> features to land in the Avocado test runner.  In contrast, fixes or
> new features in utility libraries are usually needed to write a test
> and can not wait for too long.
> 
> With the proposal above to bind Avocado-VT to Avocado LTS versions
> only, utility libraries present within Avocado that is,
> "avocado.utils", would also be limited to the LTS version.  This is
> not desirable, as it can:
> 
> * Slow down test development
> 
> * Create duplicate copies of utilities:
>  - within different tests
>  - across Avocado and Avocado-VT

Yes, even at present there are some problems like this - duplicated utilities
and VT tests across repos. So making repos even more out of sync could only
exacerbate this unless we have a proper remedy.

> Moving all utility libraries into Avocado-VT (such as the one currently
> available in Avocado) is also not an option, as users of other tests
> (for example avocado-misc-tests) and even Avocado itself need them.

The same doesn't hold for moving utilities from VT to Avocado though which
also has the benefit of fewer projects to manage. Side repos for such components
often end up under-maintained and impossible to contribute to.

> The most obvious solution is to move the utility libraries to its own
> repository.  Let's suppose this new repository/module is called
> "greenmatter" (my imagination ran wild, please do the same ;).  This
> would allow for scenarios such as:
> 
> 1. a) Avocado-VT master
>    b) Avocado LTS (with its builtin current version of "avocado.utils")
>    c) new tp-qemu tests using newer "greenmatter" utility libraries
> 
> 2. a) avocado-misc-tests master
>    b) Avocado (either LTS or latest)
>    c) new tests using newer "greenmatter" utility libraries

So to get this straight, using another repository for utilities (called here
"utilitarianism" to add to the diversity of names) will still require a certain
level of compatibility between Avocado and VT, this time however indirect as it
reduces to compatibility between VT and "utilitarianism" and compatibility between
"utilitarianism" and Avocado.

Of course, maintaining such compatibility is not exactly half the cost but even
less considering the typical changes such utilities undergo where breakage could
be mostly API-related and not due to conceptual differences. In the end the choice
of different repository vs using Avocado doesn't matter much for me but I wanted
to point out some pros and cons.

> Goals of "greenmatter" libraries
> --------------------------------
> 
> "tried, tested and trusted"
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> The current utility libraries suffer a big problem.  They are either:
> 
> * tested by superficial tests with lots of mocking
> * "tested" by the tests that actually have another goal and simply
>   use the libraries
> * are not tested at all
> 
> It's challenging to test the kind of libraries in "virttest.utils_*"
> or in "avocado.utils" because they require specific environments, and
> can be destructive.  For instance, not many people will trust
> (rightfully so) running tests on their local machines.
> 
> On the other hand, Avocado is in the business of:
> 
> 1) Running tests isolated (see the nrunner spanwers)
> 2) Checking, fulfilling and caching test requirements (see BP002)
> 
> The idea is to leverage and extend Avocado's features so that every
> single new utility library in this new repository gets real test
> coverage.  Examples:
> 
> 1. a) "avocado.utils.lv_utils" becomes "greenmatter.lv"
>    b) "greenmater.lv" gets a "tests/test_lv.py" module
>    c) "tests/test_lv.py" describes its requirements, such as:
>       - A software package (from the distribution): lvm2
>       - A specific isolation level for the spawner: vm
>       - An empty block device of around 10G
> 
> 2. a) "avocado.utils.distro" becomes "greenmatter.distro"
>    b) "tests/test_distro.py:Distro.test_fedora" describe its requirements:
>       - Operating system: fedora:32
>    c) "tests/test_distro.py:Distro.test_centos" describe its requirements:
>       - Operating system: centos:8
> 
> For example #2, the requirement for "fedora:32" could be fulfilled by
> the current environment, and the spawner selected could end up being
> "process".  In that case, the requirement for "centos:8" would need
> to be fulfilled by another spawner, such as a container or vm based
> spawner.

This might couple the choice of spawner too much with the given test requirement
(for instance a vm spawner should satisfy pretty much all requirements above in
the most isolated manner possible and the code should make some decisions about
reduction of isolation in order to choose a process) and we might rather be
willing to give the choice of spawner and thus isolation model to the user. However,
I digress here and focusing back on testing the utilities themselves, this could
be viable if we don't reach a chicken-or-egg problem. We could avoid it by using
k-fold self-testing where all k-1 (but the tested) utilities could be used in
the testing of a given utility or use some incremental self-compiling technique.

> The resulting test coverage for each module should be generated and
> presented automatically, such as it's currently being done with the
> (per release) coverage for the "avocado.utils.vmimage" library:
> 
>   https://avocado-framework.readthedocs.io/en/82.0/guides/writer/libs/vmimage.html#supported-images
> 
> It should be possible for a user to "shop" for a library, such as
> "lv", and find out that in which environments it has been tested on.

This is something new that wasn't clear above: I guess you mean testing a utility
in more than one environments? Is such testing planned to be part of the CI for
these utilities? Or a devoted hardware? If I wish to inspect output of tests for
a given utility where should I look for them?

Also, what is meant by "shop" more specifically? If one close the given repository
they will obtain the full set of utilities, what does an action of "selecting" a
utility entail in this sense?

> Common API "look and feel"
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Getting the design of APIs right is pretty hard, but here we have a
> unique change of declaring a common set of guidelines from the start
> and enforcing as libraries are added.
> 
> It's expected that a "BluePrint" would be written defining, among other
> things:
> 
>  * how modules should be named
>  * how classes should be named
>  * how functions should be named if they return information
>  * how functions should be named if they check or assert some condition
>  * how functions should behave if running on a platform that doesn't
>    apply (say "distro" is found running on a BSD or Windows system)

This sounds more like contribution guidelines than the current blueprints which
discuss new features or large refactoring in Avocado. I guess you mean this as
a venue for discussion in which sense I wonder do you mean this for the already
established requirements for the Avocado and VT projects or for the utility repo?

> Concurrent usage and migration plan
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> It should be possible to keep using the libraries available at
> Avocado-VT's "virtest.utils_*" and Avocado's "avocado.utils"
> concurrently with the libraries migrated to the new repository.

Yes, I think this is an essential part of any large scale migration.

> Suppose your test looks like:
> 
>    from avocado.utils import cpu
>    from avocado.utils import lv_utils
>    from avocado.utils import distro
>    from virttest import utils_net
> 
> Once "greenmatter 1.0" gets release, let's say with nothing but "lv" and
> "distro" libraries.  You project can add a new requirement:
> 
>    greenmatter==1.0
> 
> And the next version of a test can look like:
> 
>    from avocado.utils import cpu
>    from virttest import utils_net
>    from greenmatter import lv
>    from greenmatter import distro
> 
> Gradually, as more modules are migrated, the requirement can eventually be:
> 
>    greenmatter==12.0
> 
> And the test can look like:
> 
>    from greenmatter import cpu
>    from greenmatter import net
>    from greenmatter import lv
>    from greenmatter import distro

+1

> 
> That's all for now.  All feedback is appreciated.
> 
> Thanks!

I hope some of my comments and raised questions are useful and we get some more
opinions. Thanks for reaching out for feedback like this!

Plamen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20200926/ed981b3a/attachment.sig>