[Avocado-devel] Post 82.0 LTS plans

Wed Sep 30 13:29:44 UTC 2020

On Thu, Sep 17, 2020 at 3:03 AM Cleber Rosa <crosa at redhat.com> wrote:

> Hi Avocado community,
>
> This is an invitation to a conversation about Avocado's present and
> future.
>
> If you missed Avocado's latest LTS release (82.0 \o/), these are the
> two documents you may want to look at:
>
>  * https://avocado-framework.readthedocs.io/en/82.0/releases/lts/82_0.html
>
>  * https://avocado-framework.readthedocs.io/en/82.0/releases/82_0.html
>
> There are many co-related topics, so instead of firing multiple
> e-mails, I'll try to address all of them here, while using section
> names to keep individual topics in check.  Feel free to comment on the
> section that matters to you and ignore others.
>
> 69.x LTS
> ========
>
> The general rule that the previous LTS version will be maintained for
> a minimum of 6 months following another LTS release.  Following that
> rule, we would be maintaining the 69.x LTS series until March 2021.
>
> One important characteristic of the 69.x series is that it supports
> Python 2, which has been declared EOL on Dec 2019.  So users stuck
> with Python 2 can only use 69.x LTS.  In theory, the March 2021 date
> far exceeds the Python 2 EOL date, so we suppose that it's a safe
> date.
>

+1

>
> Avocado-VT and Avocado Compatibility
> ====================================
>
> Avocado-VT is Avocado's primary "customer" and the two share a common
> origin, and one retro feeds improvements into the other.  Having said
> that, a lot of energy has been spent into keeping Avocado-VT compatible
> with multiple Avocado of versions.  At any given moment, Avocado-VT is
> supposed to be compatible with:
>
> 1. Avocado 69.x LTS
> 2. Avocado Latest release (currently 82.0)
> 3. Avocado development branch
>
> With Avocado 82.0 LTS, there's an opportunity to improve this
> situation.  The first drastic change I'll propose is to drop as many
> of the compatibility requirements above, and pin Avocado-VT for the
> time being to require compatibility Avocado 82.0 LTS *only*.  This
> would greatly simplify Avocado-VT's development, as there would be
> one and only way to deal with Avocado.
>
> On the other hand, this could cause:
>
> 1. Bitrot, where Avocado-VT would at a given point not work with
>    newer Avocado, say the next LTS version.
>
> 2. A "lock down", where improvements to "avocado.utils" libraries
>    would not be easily available to Avocado-VT tests.
>
> Avoiding bitrot
> ---------------
>
> Let's assume that the next LTS release is version 95.0.  The worst
> possible scenario would be "waking up" and finding out that making
> Avocado-VT compatible Avocado 95.0 would be overwhelming and
> prohibitive mission.
>
> One mitigation would be to extend 82.0 LTS life time, but this would
> only push the problem further away and not really solve it.
>
> To avoid such a situation, I propose that a proactive effort is
> made every few Avocado releases, with the goal of:
>
> * evaluate the current compatibility
> * plan and develop compatibility tasks
>
> Those should be done not only in light of the *current* Avocado state,
> but also in sync with Avocado's planned features.  For example,
> consider the following release time line:
>
> * Avocado 83.0 released
> * Avocado 84.0 released
> * Avocado 85.0 released
> * Avocado-VT compatibility evaluated with Avocado 85.0
>  - Compatibility with newer Avocado "runner API" identified and
>    issue created
>  - Compatibility with newer Avocado "result API" identified and
>    issue created
> * Avocado 86.0 is released
>  - Compatibility with "runner API" resolved
> * Avocado 87.0 is released
>  - Compatibility with "result API" resolved
> * Avocado 88.0 released
> * Avocado-VT compatibility evaluated with Avocado 88.0
>   ...
> * Avocado 94.0 released (pre LTS release)
>  - All pending compatibility issues are addressed
> * Avocado 95.0 released
>  - Avocado-VT should be compatible with both current 95.0 LTS and 82.x
>    LTS
>

This sounds very reasonable to me, the only one question is how would it
affect the release workflow of avocado-vt?

>
> Integration jobs on CI can also keep running with Avocado master, but
> in non-gating state ("allow failures" in Travis CI terminology).  This
> can serve as a "temperature check" for the level of compatibility and
> tasks to records as issues and address in the near future.
>
> Library lock downs
> ------------------
>
> Users running tests can usually wait a reasonable long time for new
> features to land in the Avocado test runner.  In contrast, fixes or
> new features in utility libraries are usually needed to write a test
> and can not wait for too long.
>
> With the proposal above to bind Avocado-VT to Avocado LTS versions
> only, utility libraries present within Avocado that is,
> "avocado.utils", would also be limited to the LTS version.  This is
> not desirable, as it can:
>
> * Slow down test development
>
> * Create duplicate copies of utilities:
>  - within different tests
>  - across Avocado and Avocado-VT
>
> Moving all utility libraries into Avocado-VT (such as the one currently
> available in Avocado) is also not an option, as users of other tests
> (for example avocado-misc-tests) and even Avocado itself need them.
>
> The most obvious solution is to move the utility libraries to its own
> repository.  Let's suppose this new repository/module is called
> "greenmatter" (my imagination ran wild, please do the same ;).  This
> would allow for scenarios such as:
>
> 1. a) Avocado-VT master
>    b) Avocado LTS (with its builtin current version of "avocado.utils")
>    c) new tp-qemu tests using newer "greenmatter" utility libraries
>
> 2. a) avocado-misc-tests master
>    b) Avocado (either LTS or latest)
>    c) new tests using newer "greenmatter" utility libraries
>
> Goals of "greenmatter" libraries
> --------------------------------
>
> "tried, tested and trusted"
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The current utility libraries suffer a big problem.  They are either:
>
> * tested by superficial tests with lots of mocking
> * "tested" by the tests that actually have another goal and simply
>   use the libraries
> * are not tested at all
>
> It's challenging to test the kind of libraries in "virttest.utils_*"
> or in "avocado.utils" because they require specific environments, and
> can be destructive.  For instance, not many people will trust
> (rightfully so) running tests on their local machines.
>
> On the other hand, Avocado is in the business of:
>
> 1) Running tests isolated (see the nrunner spanwers)
> 2) Checking, fulfilling and caching test requirements (see BP002)
>
> The idea is to leverage and extend Avocado's features so that every
> single new utility library in this new repository gets real test
> coverage.  Examples:
>
> 1. a) "avocado.utils.lv_utils" becomes "greenmatter.lv"
>    b) "greenmater.lv" gets a "tests/test_lv.py" module
>    c) "tests/test_lv.py" describes its requirements, such as:
>       - A software package (from the distribution): lvm2
>       - A specific isolation level for the spawner: vm
>       - An empty block device of around 10G
>
> 2. a) "avocado.utils.distro" becomes "greenmatter.distro"
>    b) "tests/test_distro.py:Distro.test_fedora" describe its requirements:
>       - Operating system: fedora:32
>    c) "tests/test_distro.py:Distro.test_centos" describe its requirements:
>       - Operating system: centos:8
>
> For example #2, the requirement for "fedora:32" could be fulfilled by
> the current environment, and the spawner selected could end up being
> "process".  In that case, the requirement for "centos:8" would need
> to be fulfilled by another spawner, such as a container or vm based
> spawner.
>
> The resulting test coverage for each module should be generated and
> presented automatically, such as it's currently being done with the
> (per release) coverage for the "avocado.utils.vmimage" library:
>
>
> https://avocado-framework.readthedocs.io/en/82.0/guides/writer/libs/vmimage.html#supported-images
>
> It should be possible for a user to "shop" for a library, such as
> "lv", and find out that in which environments it has been tested on.
>
> Common API "look and feel"
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Getting the design of APIs right is pretty hard, but here we have a
> unique change of declaring a common set of guidelines from the start
> and enforcing as libraries are added.
>
> It's expected that a "BluePrint" would be written defining, among other
> things:
>
>  * how modules should be named
>  * how classes should be named
>  * how functions should be named if they return information
>  * how functions should be named if they check or assert some condition
>  * how functions should behave if running on a platform that doesn't
>    apply (say "distro" is found running on a BSD or Windows system)
>

Exactly, to get "trusted" these libraries should be well-designed/defined,
but I am just curious how these guidelines would be taken in practice. In
my experience, especially for new initiatives, different abstractions could
bring different module/class/function names, and maintainers may hold
different opinions on that. So I wonder if the above things could cover it.

>
> Concurrent usage and migration plan
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> It should be possible to keep using the libraries available at
> Avocado-VT's "virtest.utils_*" and Avocado's "avocado.utils"
> concurrently with the libraries migrated to the new repository.
>
> Suppose your test looks like:
>
>    from avocado.utils import cpu
>    from avocado.utils import lv_utils
>    from avocado.utils import distro
>    from virttest import utils_net
>
> Once "greenmatter 1.0" gets release, let's say with nothing but "lv" and
> "distro" libraries.  You project can add a new requirement:
>
>    greenmatter==1.0
>
> And the next version of a test can look like:
>
>    from avocado.utils import cpu
>    from virttest import utils_net
>    from greenmatter import lv
>    from greenmatter import distro
>

During migration, a bad case I can see is we have to commit a change twice:
one for avocado.utils/virttest.utils_* and the other for the devel branch
of greenmatter, for example, a bug fix. But I think it could be acceptable
because we may not meet that too often.

>
> Gradually, as more modules are migrated, the requirement can eventually be:
>
>    greenmatter==12.0
>

Does it mean the latest release of greenmatter should be the only supported
release? or will there be LTS releases? (to avoid facing compatibility
issues, it is better to limit the usage as well.)

--
Xu Han

> And the test can look like:
>
>    from greenmatter import cpu
>    from greenmatter import net
>    from greenmatter import lv
>    from greenmatter import distro
>
>
> That's all for now.  All feedback is appreciated.
>
> Thanks!
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20200930/329c5401/attachment.htm>