[Avocado-devel] Post 82.0 LTS plans

[Plamen Dimitrov]

Hi Cleber,

On 2020-09-28 22:17, Cleber Rosa wrote:
> On Sat, Sep 26, 2020 at 11:06:44PM +0300, Plamen Dimitrov wrote:
>>> Avocado-VT and Avocado Compatibility
>>> ====================================
>>>
>>> Avocado-VT is Avocado's primary "customer" and the two share a common
>>> origin, and one retro feeds improvements into the other.  Having said
>>> that, a lot of energy has been spent into keeping Avocado-VT compatible
>>> with multiple Avocado of versions.  At any given moment, Avocado-VT is
>>> supposed to be compatible with:
>>>
>>> 1. Avocado 69.x LTS
>>> 2. Avocado Latest release (currently 82.0)
>>> 3. Avocado development branch
>>>
>>> With Avocado 82.0 LTS, there's an opportunity to improve this
>>> situation.  The first drastic change I'll propose is to drop as many
>>> of the compatibility requirements above, and pin Avocado-VT for the
>>> time being to require compatibility Avocado 82.0 LTS *only*.  This
>>> would greatly simplify Avocado-VT's development, as there would be
>>> one and only way to deal with Avocado.
>>
>> What doesn't remain clear to me is what is the planned versioning to
>> follow from such an arrangement? Is Avocado VT going to be tagged in
>> synchrony with Avocado releases still considering that the such tagging
>> does not imply compatibility or give any other information? Or will
>> Avocado VT follow different numbering? Or just the LTS tags? How about
>> packaging to follow from such tagging? Will we have for instance an
>> 82.0-versioned VT RPM to install alongside an 83 Avocado RPM?
>>
> 
> That's an interesting point.  So far, the releases of Avocado-VT are
> not done in a way that a roadmap is strictly followed or a number of
> specific issues have to be completed before its releases.  Avocado-VT
> releases are just a convenience tag/release events that go through
> some compatibility testing.

I see, in that case we have to make sure to never lock the versions, i.e.
make Avocado VT require the same version of Avocado which we do for our
plugin since so far we were provided such compatibility. We don't do such
version locking on the VT side though so I guess the tag will convey even
less meaning now making me wonder what is its use. For instance, what is
the convenience that remains here? Having an 83.0 VT RPM installed together
with a 83.0 Avocado RPM could now be incompatible while the tags signal
misleading information. Perhaps we could just tag the LTS versions for
which we guarantee compatibility? In this way the tags are still conveying
some information and remain meaningful. In the other case they are not
just meaningless but even misleading.

> So, even though this proposal means that Avocado-VT will be used with
> the LTS release only, the same convenience tag/release and
> compatibility testing (beyond the CI level) can continue to be done.
> 
> In short, I think Avocado-VT 83.0 (and then 84.0, 85.0, etc), can go
> through the same release process, and be made available to users in
> packaged form, even if it's meant to be used with Avocado 82.0 LTS.

In that case could the tags signal that they are just "a new deployable
package" through some suffix or similar? I understand wishing to make
new VT changes available and deployable, I just wonder if syncing with
the Avocado versions makes sense or will mislead people that don't know
these internals.

>>> On the other hand, this could cause:
>>>
>>> 1. Bitrot, where Avocado-VT would at a given point not work with
>>>    newer Avocado, say the next LTS version.
>>>
>>> 2. A "lock down", where improvements to "avocado.utils" libraries
>>>    would not be easily available to Avocado-VT tests.
>>
>> I am also concerned about the opposite direction - how easy it will be
>> to migrate fully implemented and application stable utilities from VT
>> to Avocado? In a previous conversation some of us have had but I also
>> say here for clarity to everyone in the mailing list, I think it is a
>> good idea to migrate more utilities from Avocado VT to the larger usage
>> scope of Avocado. This will (1) reduce the volume and maintenance cost
>> of Avocado VT, (2) make the possibility of compatibility with Avocado
>> even easier (as more of the code and improvements will immediately end
>> up on the Avocado side), and (3) make it easier to differentiate which
>> functionality of Avocado VT still remains irreplaceable by pure Avocado
>> like VM isolation model, etc.
>>
> 
> ACK, these are all valid concerns and noteworthy points.

I think we all agree that VT's volume should be reduced and VT's utilities
reused. I am agnostic to whether this happens in a separate repo or in
Avocado as long as the repo is maintained as promised and doesn't end up
like the `aexpect` repo where a lot of code was migrated and neglected which
made me realize how spreading well-centered projects into different
repositories could also have some really bad long term effects.

>>> Avoiding bitrot
>>> ---------------
>>>
>>> Let's assume that the next LTS release is version 95.0.  The worst
>>> possible scenario would be "waking up" and finding out that making
>>> Avocado-VT compatible Avocado 95.0 would be overwhelming and
>>> prohibitive mission.
>>>
>>> One mitigation would be to extend 82.0 LTS life time, but this would
>>> only push the problem further away and not really solve it.
>>>
>>> To avoid such a situation, I propose that a proactive effort is
>>> made every few Avocado releases, with the goal of:
>>>
>>> * evaluate the current compatibility
>>> * plan and develop compatibility tasks
>>>
>>> Those should be done not only in light of the *current* Avocado state,
>>> but also in sync with Avocado's planned features.  For example,
>>> consider the following release time line:
>>>
>>> * Avocado 83.0 released
>>> * Avocado 84.0 released
>>> * Avocado 85.0 released
>>> * Avocado-VT compatibility evaluated with Avocado 85.0
>>>  - Compatibility with newer Avocado "runner API" identified and
>>>    issue created
>>>  - Compatibility with newer Avocado "result API" identified and
>>>    issue created
>>> * Avocado 86.0 is released
>>>  - Compatibility with "runner API" resolved
>>> * Avocado 87.0 is released
>>>  - Compatibility with "result API" resolved
>>> * Avocado 88.0 released
>>> * Avocado-VT compatibility evaluated with Avocado 88.0
>>>   ...
>>> * Avocado 94.0 released (pre LTS release)
>>>  - All pending compatibility issues are addressed
>>> * Avocado 95.0 released
>>>  - Avocado-VT should be compatible with both current 95.0 LTS and 82.x
>>>    LTS
>>
>> I am not really sure how beneficial this process will end up being. On one
>> side it is more complex than simply fixing smaller (if discovered earlier)
>> incompatibilities between VT and Avocado. On the other it does reduce the
>> number of checks one has to do. I guess one of the main benefits of going
>> this way remains reduction of context switching and more clumped batch-oriented
>> fixes at certain checkpoints that won't require divided focus. At the same
>> time it remains true that with longer periods between syncs the issues will
>> pile up much more and to some extend may become harder to resolve and overcome.
>>
> 
> Besides the context switching you described, the problem with the
> current per-commit lockstep compatibility approach requires extra
> development that may not be needed at all with a coarser compatibility
> requirement.  So, changes that happen between, say, Avocado 82.0 and
> 83.0 may require compatibility changes to Avocado-VT.  Then, on
> Avocado 84.0, more work on the same area is done, and more
> compatibility related changes are required on the Avocado-VT side.
> These could really be one set of changes to ensure Avocado-VT
> compatibility with Avocado 84.0.
> 
> We could work around this on the Avocado side, by:
> 
> 1) having very long sprints;
> 
> 2) having a "next" branch that introduces changes without the promise
> of compatibility, and delay sending changes to master
> 
> But this is really a bad way of faking the real impacts of what
> happens when other software depends on another software master branch
> and would impact all other Avocado users.
> 
> I know this proposal is not a solution that makes everybody thrilled,
> but it's the best compromise I can think of.

Your proposal remains a good and efficient way to deal with constrained
maintenance resources. It is also a good way to deal with unnecessary
co-development overhead so I am sold here. It is just important to tread
these waters carefully since the there is a big gray region between how
many changes are saving time and how many will end up costing even more
time untangling. Time will be the judge where a fine line is drawn here.

>>> Integration jobs on CI can also keep running with Avocado master, but
>>> in non-gating state ("allow failures" in Travis CI terminology).  This
>>> can serve as a "temperature check" for the level of compatibility and
>>> tasks to records as issues and address in the near future.
>>
>> Another thing this section doesn't clarify much are the really long term plans
>> regarding Avocado VT. Is it planned to be supported many more LTS-es to come
>> or wished to be replaced with the newer isolation models of Avocado? Knowing
>> the end-goal can help us judge better what is a better direction in the present.
>>
> 
> I really think Avocado-VT will be around for many more LTS-es.  The
> value it has in itself, and in the form of the coverage provided by
> tp-qemu and tp-libvirt (and others) can not be replicated easily.
> 
> But, we see a trend and push to have tests being written as early as
> possible, if possible at the upstream projects by the original author
> of the features.  In this case, it's *not* easy to leverage Avocado-VT.
> 
> The win-win situation is where both the upstream developer writing a
> "bare Avocado" test, and Avocado-VT users can use as much as possible
> common code, while preserving their inherent differences at the
> framework level.  Here I'm talking specially about the the utility
> libraries, but also some Avocado test runner (and related tooling).

Agreed here, Avocado VT is the product of many years of development of
a very large group of people and I simple see no easy way one could even
consider "replacing" it at present. This being said, I am all in for the
utility migration ideas which will still reduce its size and overall
redundancy and show us if ever we could replicate its isolation model
in a replacing manner.

> The possibility of under-maintanance and overall low quality of such
> code is exactly what I hope to address here.  The proposal of a new
> repo, with *high* standards and the right tools to help achieve those
> standards, is what I suggested below.

High standards are something that I thought was always on the table, my
reference here was towards the original maintainers losing interest in
the migrated project or changing jobs as well as other external factors
that would influence the "pulse" of the project. Let's hope the new repo
(coined `avocado-utils`) will be well used and we all end up with numerous
contributions to the main projects reinventing the same utilities locally
because "it is cheaper to depend on fewer external repos" or simply because
they don't know of the existence of these utilities. This has happened
multiple times already and human energy and effort is put into compensating
for components distributed in a nontransparent manner.

>>> The most obvious solution is to move the utility libraries to its own
>>> repository.  Let's suppose this new repository/module is called
>>> "greenmatter" (my imagination ran wild, please do the same ;).  This
>>> would allow for scenarios such as:
>>>
>>> 1. a) Avocado-VT master
>>>    b) Avocado LTS (with its builtin current version of "avocado.utils")
>>>    c) new tp-qemu tests using newer "greenmatter" utility libraries
>>>
>>> 2. a) avocado-misc-tests master
>>>    b) Avocado (either LTS or latest)
>>>    c) new tests using newer "greenmatter" utility libraries
>>
>> So to get this straight, using another repository for utilities (called here
>> "utilitarianism" to add to the diversity of names) will still require a certain
>> level of compatibility between Avocado and VT, this time however indirect as it
>> reduces to compatibility between VT and "utilitarianism" and compatibility between
>> "utilitarianism" and Avocado.
>>
> 
> Right, it will require compatibility handling, but only by the project
> that uses it, and only on that direction.  What I mean is, once this
> module releases version 1.0, Avocado is free to adopt version 1.0, and
> switch from using some of its "avocado.utils" libraries to some of
> those in the new module.  It may take Avocado-VT more time to adopt
> it, and then adopt version 3.0 (or the other way around).
> 
> The end goal, however, is for all projects to move utility libraries
> to this new repo, and update their compatibility often.
> 
>> Of course, maintaining such compatibility is not exactly half the cost but even
>> less considering the typical changes such utilities undergo where breakage could
>> be mostly API-related and not due to conceptual differences. In the end the choice
>> of different repository vs using Avocado doesn't matter much for me but I wanted
>> to point out some pros and cons.
>>
> 
> Right.  The API changes are usually mechanic as you suggested and be
> commited with the new version requirements.  It's quite different than
> the current approach where Avocado-VT (and its tests) is supposed be
> compatible with all Avocado version... which leads to keeping its own
> libraries (in exchange for developer sanity :).

Exactly!

>> I digress here and focusing back on testing the utilities themselves, this could
>> be viable if we don't reach a chicken-or-egg problem. We could avoid it by using
>> k-fold self-testing where all k-1 (but the tested) utilities could be used in
>> the testing of a given utility or use some incremental self-compiling technique.
>>
> This sounds interesting.  TBH, my original, poor guy's approach, was
> to think of the tooling (say Avocado + Job definition + Spawners code)
> as a separate instalation.  So the tooling used to test the libraries
> would be using a non-development version of the libraries.  It could
> be interesting indeed to add a testing in which the tooling *also*
> uses the libraries under test, to get a sense of the compatibility,
> and maybe even use that result as a self-promotion (bumping the
> version) of the libraries used on the tooling.
> 
> Is this more or less what you had in mind?

I guess one could look for some intriguing metaprogramming automation approaches
here but for now your poor man's approach sounds much more grounded and more easily
resourceful here. It will still be interesting to observe the interaction of neighboring
versions and the possibility of an issue hiding in both the tester and test subject.

>>> Common API "look and feel"
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Getting the design of APIs right is pretty hard, but here we have a
>>> unique change of declaring a common set of guidelines from the start
>>> and enforcing as libraries are added.
>>>
>>> It's expected that a "BluePrint" would be written defining, among other
>>> things:
>>>
>>>  * how modules should be named
>>>  * how classes should be named
>>>  * how functions should be named if they return information
>>>  * how functions should be named if they check or assert some condition
>>>  * how functions should behave if running on a platform that doesn't
>>>    apply (say "distro" is found running on a BSD or Windows system)
>>
>> This sounds more like contribution guidelines than the current blueprints which
>> discuss new features or large refactoring in Avocado. I guess you mean this as
>> a venue for discussion in which sense I wonder do you mean this for the already
>> established requirements for the Avocado and VT projects or for the utility repo?
>>
> 
> You're right that the current Blue Prints on the Avocado project
> describe technical features.  But, BP000 is due to describe the
> overall usage of the Blue Prints.  Maybe Blue Prints can and should
> include guidelines too.

I think this will be confusing at least according to the current usage of blueprints.
If I am a contributor wishing to check contribution guidelines, RFCs wouldn't be the
first place to check. But it could be that my perception of blueprints is wrong and
understanding them as RFCs is wrong to begin with. I could reinterpret the BP0001 as
guidelines on how to use and write settings so this still makes sense, however I was
under the impression that the requirements resolver is a pure feature blueprint that
was described and added with the purpose of further discussion. I guess we should try
to differentiate emails looking for feedback from guidelines and choose which one of
these will end up as blueprints.

> The take away is that I'm initially interested in setting a much
> better thought out, and consequently stricter, set of contribution
> *requirements* for this new repo.

Agreed here, I guess I didn't know that such contribution guidelines are less strictly
followed in other repos in order to specify that this one needs an improvement in this
direction.
Thanks for all the answers to previous questions and follow-up on the feedback!
Plamen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20200930/8404130f/attachment.sig>