[Avocado-devel] Post 82.0 LTS plans
Cleber Rosa
crosa at redhat.com
Wed Sep 16 19:02:58 UTC 2020
Hi Avocado community,
This is an invitation to a conversation about Avocado's present and
future.
If you missed Avocado's latest LTS release (82.0 \o/), these are the
two documents you may want to look at:
* https://avocado-framework.readthedocs.io/en/82.0/releases/lts/82_0.html
* https://avocado-framework.readthedocs.io/en/82.0/releases/82_0.html
There are many co-related topics, so instead of firing multiple
e-mails, I'll try to address all of them here, while using section
names to keep individual topics in check. Feel free to comment on the
section that matters to you and ignore others.
69.x LTS
========
The general rule that the previous LTS version will be maintained for
a minimum of 6 months following another LTS release. Following that
rule, we would be maintaining the 69.x LTS series until March 2021.
One important characteristic of the 69.x series is that it supports
Python 2, which has been declared EOL on Dec 2019. So users stuck
with Python 2 can only use 69.x LTS. In theory, the March 2021 date
far exceeds the Python 2 EOL date, so we suppose that it's a safe
date.
Avocado-VT and Avocado Compatibility
====================================
Avocado-VT is Avocado's primary "customer" and the two share a common
origin, and one retro feeds improvements into the other. Having said
that, a lot of energy has been spent into keeping Avocado-VT compatible
with multiple Avocado of versions. At any given moment, Avocado-VT is
supposed to be compatible with:
1. Avocado 69.x LTS
2. Avocado Latest release (currently 82.0)
3. Avocado development branch
With Avocado 82.0 LTS, there's an opportunity to improve this
situation. The first drastic change I'll propose is to drop as many
of the compatibility requirements above, and pin Avocado-VT for the
time being to require compatibility Avocado 82.0 LTS *only*. This
would greatly simplify Avocado-VT's development, as there would be
one and only way to deal with Avocado.
On the other hand, this could cause:
1. Bitrot, where Avocado-VT would at a given point not work with
newer Avocado, say the next LTS version.
2. A "lock down", where improvements to "avocado.utils" libraries
would not be easily available to Avocado-VT tests.
Avoiding bitrot
---------------
Let's assume that the next LTS release is version 95.0. The worst
possible scenario would be "waking up" and finding out that making
Avocado-VT compatible Avocado 95.0 would be overwhelming and
prohibitive mission.
One mitigation would be to extend 82.0 LTS life time, but this would
only push the problem further away and not really solve it.
To avoid such a situation, I propose that a proactive effort is
made every few Avocado releases, with the goal of:
* evaluate the current compatibility
* plan and develop compatibility tasks
Those should be done not only in light of the *current* Avocado state,
but also in sync with Avocado's planned features. For example,
consider the following release time line:
* Avocado 83.0 released
* Avocado 84.0 released
* Avocado 85.0 released
* Avocado-VT compatibility evaluated with Avocado 85.0
- Compatibility with newer Avocado "runner API" identified and
issue created
- Compatibility with newer Avocado "result API" identified and
issue created
* Avocado 86.0 is released
- Compatibility with "runner API" resolved
* Avocado 87.0 is released
- Compatibility with "result API" resolved
* Avocado 88.0 released
* Avocado-VT compatibility evaluated with Avocado 88.0
...
* Avocado 94.0 released (pre LTS release)
- All pending compatibility issues are addressed
* Avocado 95.0 released
- Avocado-VT should be compatible with both current 95.0 LTS and 82.x
LTS
Integration jobs on CI can also keep running with Avocado master, but
in non-gating state ("allow failures" in Travis CI terminology). This
can serve as a "temperature check" for the level of compatibility and
tasks to records as issues and address in the near future.
Library lock downs
------------------
Users running tests can usually wait a reasonable long time for new
features to land in the Avocado test runner. In contrast, fixes or
new features in utility libraries are usually needed to write a test
and can not wait for too long.
With the proposal above to bind Avocado-VT to Avocado LTS versions
only, utility libraries present within Avocado that is,
"avocado.utils", would also be limited to the LTS version. This is
not desirable, as it can:
* Slow down test development
* Create duplicate copies of utilities:
- within different tests
- across Avocado and Avocado-VT
Moving all utility libraries into Avocado-VT (such as the one currently
available in Avocado) is also not an option, as users of other tests
(for example avocado-misc-tests) and even Avocado itself need them.
The most obvious solution is to move the utility libraries to its own
repository. Let's suppose this new repository/module is called
"greenmatter" (my imagination ran wild, please do the same ;). This
would allow for scenarios such as:
1. a) Avocado-VT master
b) Avocado LTS (with its builtin current version of "avocado.utils")
c) new tp-qemu tests using newer "greenmatter" utility libraries
2. a) avocado-misc-tests master
b) Avocado (either LTS or latest)
c) new tests using newer "greenmatter" utility libraries
Goals of "greenmatter" libraries
--------------------------------
"tried, tested and trusted"
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The current utility libraries suffer a big problem. They are either:
* tested by superficial tests with lots of mocking
* "tested" by the tests that actually have another goal and simply
use the libraries
* are not tested at all
It's challenging to test the kind of libraries in "virttest.utils_*"
or in "avocado.utils" because they require specific environments, and
can be destructive. For instance, not many people will trust
(rightfully so) running tests on their local machines.
On the other hand, Avocado is in the business of:
1) Running tests isolated (see the nrunner spanwers)
2) Checking, fulfilling and caching test requirements (see BP002)
The idea is to leverage and extend Avocado's features so that every
single new utility library in this new repository gets real test
coverage. Examples:
1. a) "avocado.utils.lv_utils" becomes "greenmatter.lv"
b) "greenmater.lv" gets a "tests/test_lv.py" module
c) "tests/test_lv.py" describes its requirements, such as:
- A software package (from the distribution): lvm2
- A specific isolation level for the spawner: vm
- An empty block device of around 10G
2. a) "avocado.utils.distro" becomes "greenmatter.distro"
b) "tests/test_distro.py:Distro.test_fedora" describe its requirements:
- Operating system: fedora:32
c) "tests/test_distro.py:Distro.test_centos" describe its requirements:
- Operating system: centos:8
For example #2, the requirement for "fedora:32" could be fulfilled by
the current environment, and the spawner selected could end up being
"process". In that case, the requirement for "centos:8" would need
to be fulfilled by another spawner, such as a container or vm based
spawner.
The resulting test coverage for each module should be generated and
presented automatically, such as it's currently being done with the
(per release) coverage for the "avocado.utils.vmimage" library:
https://avocado-framework.readthedocs.io/en/82.0/guides/writer/libs/vmimage.html#supported-images
It should be possible for a user to "shop" for a library, such as
"lv", and find out that in which environments it has been tested on.
Common API "look and feel"
~~~~~~~~~~~~~~~~~~~~~~~~~~
Getting the design of APIs right is pretty hard, but here we have a
unique change of declaring a common set of guidelines from the start
and enforcing as libraries are added.
It's expected that a "BluePrint" would be written defining, among other
things:
* how modules should be named
* how classes should be named
* how functions should be named if they return information
* how functions should be named if they check or assert some condition
* how functions should behave if running on a platform that doesn't
apply (say "distro" is found running on a BSD or Windows system)
Concurrent usage and migration plan
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It should be possible to keep using the libraries available at
Avocado-VT's "virtest.utils_*" and Avocado's "avocado.utils"
concurrently with the libraries migrated to the new repository.
Suppose your test looks like:
from avocado.utils import cpu
from avocado.utils import lv_utils
from avocado.utils import distro
from virttest import utils_net
Once "greenmatter 1.0" gets release, let's say with nothing but "lv" and
"distro" libraries. You project can add a new requirement:
greenmatter==1.0
And the next version of a test can look like:
from avocado.utils import cpu
from virttest import utils_net
from greenmatter import lv
from greenmatter import distro
Gradually, as more modules are migrated, the requirement can eventually be:
greenmatter==12.0
And the test can look like:
from greenmatter import cpu
from greenmatter import net
from greenmatter import lv
from greenmatter import distro
That's all for now. All feedback is appreciated.
Thanks!
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20200916/7dec0845/attachment.sig>
More information about the Avocado-devel
mailing list