[Avocado-devel] Post 82.0 LTS plans
Jeff Nelson
jen at redhat.com
Wed Sep 30 17:48:26 UTC 2020
On Wed, 16 Sep 2020 15:02:58 -0400
Cleber Rosa <crosa at redhat.com> wrote:
> Hi Avocado community,
>
> This is an invitation to a conversation about Avocado's present and
> future.
>
> If you missed Avocado's latest LTS release (82.0 \o/), these are the
> two documents you may want to look at:
>
> * https://avocado-framework.readthedocs.io/en/82.0/releases/lts/82_0.html
>
> * https://avocado-framework.readthedocs.io/en/82.0/releases/82_0.html
>
> There are many co-related topics, so instead of firing multiple
> e-mails, I'll try to address all of them here, while using section
> names to keep individual topics in check. Feel free to comment on the
> section that matters to you and ignore others.
>
> 69.x LTS
> ========
>
> The general rule that the previous LTS version will be maintained for
> a minimum of 6 months following another LTS release. Following that
> rule, we would be maintaining the 69.x LTS series until March 2021.
>
> One important characteristic of the 69.x series is that it supports
> Python 2, which has been declared EOL on Dec 2019. So users stuck
> with Python 2 can only use 69.x LTS. In theory, the March 2021 date
> far exceeds the Python 2 EOL date, so we suppose that it's a safe
> date.
+1
> Avocado-VT and Avocado Compatibility
> ====================================
>
> Avocado-VT is Avocado's primary "customer" and the two share a common
> origin, and one retro feeds improvements into the other. Having said
> that, a lot of energy has been spent into keeping Avocado-VT compatible
> with multiple Avocado of versions. At any given moment, Avocado-VT is
> supposed to be compatible with:
>
> 1. Avocado 69.x LTS
> 2. Avocado Latest release (currently 82.0)
> 3. Avocado development branch
>
> With Avocado 82.0 LTS, there's an opportunity to improve this
> situation. The first drastic change I'll propose is to drop as many
> of the compatibility requirements above, and pin Avocado-VT for the
> time being to require compatibility Avocado 82.0 LTS *only*. This
> would greatly simplify Avocado-VT's development, as there would be
> one and only way to deal with Avocado.
Reducing the sync points between avocado and avocado-vt is a desirable
goal. At the same time, I don't think this should give developers free
reign to intentionally break compatibility. It should always be done
intentionally and with a plan to address it somehow.
> On the other hand, this could cause:
>
> 1. Bitrot, where Avocado-VT would at a given point not work with
> newer Avocado, say the next LTS version.
>
> 2. A "lock down", where improvements to "avocado.utils" libraries
> would not be easily available to Avocado-VT tests.
>
> Avoiding bitrot
> ---------------
>
> Let's assume that the next LTS release is version 95.0. The worst
> possible scenario would be "waking up" and finding out that making
> Avocado-VT compatible Avocado 95.0 would be overwhelming and
> prohibitive mission.
>
> One mitigation would be to extend 82.0 LTS life time, but this would
> only push the problem further away and not really solve it.
>
> To avoid such a situation, I propose that a proactive effort is
> made every few Avocado releases, with the goal of:
>
> * evaluate the current compatibility
> * plan and develop compatibility tasks
+1
> Those should be done not only in light of the *current* Avocado state,
> but also in sync with Avocado's planned features. For example,
> consider the following release time line:
>
> * Avocado 83.0 released
> * Avocado 84.0 released
> * Avocado 85.0 released
> * Avocado-VT compatibility evaluated with Avocado 85.0
> - Compatibility with newer Avocado "runner API" identified and
> issue created
> - Compatibility with newer Avocado "result API" identified and
> issue created
> * Avocado 86.0 is released
> - Compatibility with "runner API" resolved
> * Avocado 87.0 is released
> - Compatibility with "result API" resolved
> * Avocado 88.0 released
> * Avocado-VT compatibility evaluated with Avocado 88.0
> ...
> * Avocado 94.0 released (pre LTS release)
> - All pending compatibility issues are addressed
> * Avocado 95.0 released
> - Avocado-VT should be compatible with both current 95.0 LTS and 82.x
> LTS
>
> Integration jobs on CI can also keep running with Avocado master, but
> in non-gating state ("allow failures" in Travis CI terminology). This
> can serve as a "temperature check" for the level of compatibility and
> tasks to records as issues and address in the near future.
>
> Library lock downs
> ------------------
>
> Users running tests can usually wait a reasonable long time for new
> features to land in the Avocado test runner. In contrast, fixes or
> new features in utility libraries are usually needed to write a test
> and can not wait for too long.
>
> With the proposal above to bind Avocado-VT to Avocado LTS versions
> only, utility libraries present within Avocado that is,
> "avocado.utils", would also be limited to the LTS version. This is
> not desirable, as it can:
>
> * Slow down test development
>
> * Create duplicate copies of utilities:
> - within different tests
> - across Avocado and Avocado-VT
>
> Moving all utility libraries into Avocado-VT (such as the one currently
> available in Avocado) is also not an option, as users of other tests
> (for example avocado-misc-tests) and even Avocado itself need them.
>
> The most obvious solution is to move the utility libraries to its own
> repository. Let's suppose this new repository/module is called
> "greenmatter" (my imagination ran wild, please do the same ;). This
> would allow for scenarios such as:
>
> 1. a) Avocado-VT master
> b) Avocado LTS (with its builtin current version of "avocado.utils")
> c) new tp-qemu tests using newer "greenmatter" utility libraries
>
> 2. a) avocado-misc-tests master
> b) Avocado (either LTS or latest)
> c) new tests using newer "greenmatter" utility libraries
Adding another repo of utilities creates its own set of problems: now
you have to sync across avocado, avocado-vt and "greenmatter". This
creates a matrix of combinations that need to be supported and tested,
new kinds of defects to be triaged, and new support issues to consider;
for example, will you now need to tag the "greenmatter" repo to
indicate what it supports? Will the tag be relative to what avocado
wants, avocado-vt wants, or (ugh) both?
As suggested by Plamen Dimitrov <pdimitrov at pevogam.com>, another option
would be to move the utilities into avocado itself. This collapses the
matrix back to just two, avocado and avocado-vt, and is a much more
straightforward configuration to reason about.
> Goals of "greenmatter" libraries
> --------------------------------
>
> "tried, tested and trusted"
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The current utility libraries suffer a big problem. They are either:
>
> * tested by superficial tests with lots of mocking
> * "tested" by the tests that actually have another goal and simply
> use the libraries
> * are not tested at all
>
> It's challenging to test the kind of libraries in "virttest.utils_*"
> or in "avocado.utils" because they require specific environments, and
> can be destructive. For instance, not many people will trust
> (rightfully so) running tests on their local machines.
>
> On the other hand, Avocado is in the business of:
>
> 1) Running tests isolated (see the nrunner spanwers)
> 2) Checking, fulfilling and caching test requirements (see BP002)
>
> The idea is to leverage and extend Avocado's features so that every
> single new utility library in this new repository gets real test
> coverage. Examples:
>
> 1. a) "avocado.utils.lv_utils" becomes "greenmatter.lv"
> b) "greenmater.lv" gets a "tests/test_lv.py" module
> c) "tests/test_lv.py" describes its requirements, such as:
> - A software package (from the distribution): lvm2
> - A specific isolation level for the spawner: vm
> - An empty block device of around 10G
>
> 2. a) "avocado.utils.distro" becomes "greenmatter.distro"
> b) "tests/test_distro.py:Distro.test_fedora" describe its requirements:
> - Operating system: fedora:32
> c) "tests/test_distro.py:Distro.test_centos" describe its requirements:
> - Operating system: centos:8
>
> For example #2, the requirement for "fedora:32" could be fulfilled by
> the current environment, and the spawner selected could end up being
> "process". In that case, the requirement for "centos:8" would need
> to be fulfilled by another spawner, such as a container or vm based
> spawner.
>
> The resulting test coverage for each module should be generated and
> presented automatically, such as it's currently being done with the
> (per release) coverage for the "avocado.utils.vmimage" library:
>
> https://avocado-framework.readthedocs.io/en/82.0/guides/writer/libs/vmimage.html#supported-images
>
> It should be possible for a user to "shop" for a library, such as
> "lv", and find out that in which environments it has been tested on.
I see a huge explosion of a test and support matrix, of which I am not
fond.
-Jeff
> Common API "look and feel"
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Getting the design of APIs right is pretty hard, but here we have a
> unique change of declaring a common set of guidelines from the start
> and enforcing as libraries are added.
>
> It's expected that a "BluePrint" would be written defining, among other
> things:
>
> * how modules should be named
> * how classes should be named
> * how functions should be named if they return information
> * how functions should be named if they check or assert some condition
> * how functions should behave if running on a platform that doesn't
> apply (say "distro" is found running on a BSD or Windows system)
>
> Concurrent usage and migration plan
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> It should be possible to keep using the libraries available at
> Avocado-VT's "virtest.utils_*" and Avocado's "avocado.utils"
> concurrently with the libraries migrated to the new repository.
>
> Suppose your test looks like:
>
> from avocado.utils import cpu
> from avocado.utils import lv_utils
> from avocado.utils import distro
> from virttest import utils_net
>
> Once "greenmatter 1.0" gets release, let's say with nothing but "lv" and
> "distro" libraries. You project can add a new requirement:
>
> greenmatter==1.0
>
> And the next version of a test can look like:
>
> from avocado.utils import cpu
> from virttest import utils_net
> from greenmatter import lv
> from greenmatter import distro
>
> Gradually, as more modules are migrated, the requirement can eventually be:
>
> greenmatter==12.0
>
> And the test can look like:
>
> from greenmatter import cpu
> from greenmatter import net
> from greenmatter import lv
> from greenmatter import distro
>
>
> That's all for now. All feedback is appreciated.
>
> Thanks!
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
More information about the Avocado-devel
mailing list