[Avocado-devel] [RFC v3]: Avocado maintainability and integration with avocado-vt

[Jeff Nelson]

On Wed, 26 Apr 2017 19:01:37 -0400
Cleber Rosa <crosa at redhat.com> wrote:

> On 04/25/2017 12:05 PM, Lukáš Doktor wrote:
> > Dne 20.4.2017 v 03:42 Cleber Rosa napsal(a):  
> >> Hi Folks.
> >>  
> > Hello Cleber,
> > 
> > thank you for the updates, in general it's good, I have few minor
> > suggestions in-line.
> >   
> >> This RFC contains proposals and clarifications regarding the
> >> maintenance and release processes of Avocado.
> >>
> >> We understand there are multiple teams currently depending on the
> >> stability of Avocado and we don't want their work to be disrupted by
> >> incompatibilities nor instabilities in new releases.
> >>
> >> This version is a minor update to RFC version 2[1], which drove the
> >> release of Avocado 36.0 LTS.  The Avocado team has plans for a new LTS
> >> release in the near future, so please consider reading and providing
> >> feedback on the proposals here.
> >>
> >> TL;DR:
> >>
> >>   We plan to keep the current approach of sprint releases every 3-4
> >>   weeks, but we're introducing "Long Term Stability" releases which
> >>   should be adopted in production environments where users can't keep
> >>   up with frequent upgrades.
> >>
> >> Changes from v2:
> >>   - Wording changes on second paragraph ("... nor instabilities...")
> >>   - Clarified on "Introduction" that change of behavior is introduced
> >>     between regular releases
> >>   - Updated distro versions for which official packages are built
> >>   - Add more clear explanation on official packages on the various
> >>     hardware platforms
> >>   - Used more recent version numbers as examples, and the planned
> >>     new LTS version too
> >>   - Explain how users can get the LTS version when using tools such as
> >>     pip
> >>   - Simplified the timeline example, with examples that will possibly
> >>     match the future versions and releases
> >>   - Documented current status of avocado-vt releases and packages
> >>
> >> Changes from v1:
> >>   - Changed "Support" to "Stability" and "supported" to "maintained"
> >>     [Jeff Nelson]
> >>   - Misc improvements and clarifications in the
> >>     supportability/stability statements [Jeff Nelson, me]
> >>   - Fixed a few typos [Jeff Nelson, me]
> >>
> >>
> >> Introduction
> >> --------------
> >>
> >> We make new releases of Avocado every 3-4 weeks on average.  In theory
> >> at least, we're very careful with backwards compatibility.  We test
> >> Avocado for regressions and we try to document any issues, so
> >> upgrading to a new version should be (again, in theory) safe.
> >>
> >> But in practice both intended and unintended changes are introduced
> >> during development, and both can be frustrating for conservative
> >> users. We also understand it's not feasible for users to upgrade
> >> Avocado very frequently in a production environment.
> >>
> >> The objective of this RFC is to clarify our maintenance practices and
> >> introduce Long Term Stability (LTS) releases, which are intended to
> >> solve, or at least mitigate, these problems.
> >>
> >>
> >> Our definition of maintained, or stable
> >> ---------------------------------------
> >>
> >> First of all, Avocado and its sub-projects are provided 'AS IS' and
> >> WITHOUT ANY WARRANTY, as described in the LICENSE file.
> >>
> >> The process described here doesn't imply any commitments or
> >> promises. It's just a set of best practices and recommendations.
> >>
> >> When something is identified as "stable" or "maintained", it means the
> >> development community makes a conscious effort to keep it working and
> >> consider reports of bugs and issues as high priorities.  Fixes
> >> submitted for these issues will also be considered high priorities,
> >> although they will be accepted only if they pass the general
> >> acceptance criteria for new contributions (design, quality,
> >> documentation, testing, etc), at the development team discretion.
> >>
> >>
> >> Maintained projects and platforms
> >> ---------------------------------
> >>
> >> The only maintained project as of today is the Avocado Test Runner,
> >> including its APIs and core plugins (the contents of the main avocado
> >> git repository).
> >>
> >> Other projects kept under the "Avocado Umbrella" in github may be
> >> maintained by different teams (e.g.: avocado-vt) or be considered
> >> experimental (e.g.: avocado-server and avocado-virt).
> >>
> >> More about avocado-vt in its own section further down.
> >>
> >> As a general rule, fixes and bug reports for Avocado when running in
> >> any modern Linux distribution are welcome.
> >>
> >> But given the limited capacity of the development team, packaged
> >> versions of Avocado will be tested and maintained only for the
> >> following Linux distributions:
> >>
> >>   * RHEL 7.x (latest)
> >>   * Fedora (stable releases from the Fedora projects)
> >>
> >> Currently all packages produced by the Avocado projects are "noarch".
> >> That means that they could be installable on any hardware platform.
> >> Still, the development team will currently attempt to provide versions
> >> that are stable for the following platforms:
> >>
> >>   * x86
> >>   * ppc64  
> > Do we still care about ppc64 BE? And how about the widely spreading
> > aarch64, do we want to include it?
> >   
> 
> Honestly, I can't properly answer that.
> 
> Jeff, can you help us here?

I'm not sure that ppc64 should stay, but rather than eliminate it
completely, I suggest replacing ppc64 (big endian) with ppc64le (little
endian) at the next LTS opportunity.

I wouldn't add another architecture or operating system to the
documentation unless there's a community member willing to support
it.

-Jeff

 
> >>   * ppc64le
> >>
> >> Contributions from the community to maintain other platforms and
> >> operating systems are very welcome.
> >>
> >> The lists above may change without prior notice.
> >>
> >>
> >> Avocado Releases
> >> ----------------
> >>
> >> The proposal is to have two different types of Avocado releases:
> >>
> >> 1. Sprint Releases:
> >>   (This is the model we currently adopt in Avocado)
> >>
> >>   They happen every 3-4 weeks (the schedule is not fixed) and
> >>   their versions are numbered serially, with decimal digits in
> >>   the format <major>.<minor>.  Examples: 47.0, 48.0, 49.0.  Minor
> >>   releases are rare, but necessary to correct some major issue
> >>   with the original release (47.1, 47.2, etc).
> >>
> >>   Only the latest Sprint Release is maintained.
> >>
> >>   In Sprint Releases we make a conscious effort to keep backwards
> >>   compatibility with the previous version (APIs and behavior) and
> >>   as a general rule and best practice, incompatible changes in
> >>   Sprint Releases should be documented in the release notes and
> >>   if possible deprecated slowly, to give users time to adapt
> >>   their environments.
> >>
> >>   But we understand changes are inevitable as the software
> >>   evolves and therefore there's no absolute promise for API and
> >>   behavioral stability.
> >>
> >> 2. Long Term Stability (LTS) Releases:
> >>
> >>   LTS releases should happen whenever the team feels the code is
> >>   stable enough to be maintained for a longer period of time,
> >> ideally once or twice per year (no fixed schedule).
> >>
> >>   They should be maintained for 18 months, receiving fixes for
> >> major bugs in the form of minor (sub-)releases. With the exception
> >> of these fixes, no API or behavior should change in a minor LTS
> >>   release.
> >>
> >>   They will be versioned just like Sprint Releases, so looking at
> >> the version number alone will not reveal the differentiate release
> >>   process and stability characteristics.
> >>
> >>   In practice each major LTS release will imply in the creation of
> >> a git branch where only serious issues affecting users will be
> >>   fixed. The code in a LTS branch is stable, frozen for new
> >> features.
> >>
> >>   Notice that although within a LTS release there's a expectation
> >>   of stability because the code is frozen, different (major) LTS
> >>   releases may include changes in behavior, API incompatibilities
> >>   and new features. The development team will make a considerable
> >>   effort to minimize and properly document these changes (changes
> >>   when comparing it to the last major LTS release).
> >>
> >>   Sprint Releases are replaced by LTS releases. I.e., in the cycle
> >>   when 52.0 (LTS) is released, that's also the version used as a
> >>   Sprint Release (there's no 52.0 -- non LTS -- in this case).
> >>
> >>   New LTS releases should be done carefully, with ample time for
> >>   announcements, testing and documentation.  It's recommended
> >>   that one or two sprints are dedicated as preparations for a LTS
> >>   release, with a Sprint Release serving as a "LTS beta" release.
> >>
> >>   Similarly, there should be announcements about the end-of-life
> >>   (EOL) of a LTS release once it approaches its 18 months of
> >>   life.
> >>
> >>
> >> Misc details  
> > How about calling this section deployment details?
> >   
> 
> Good idea.
> 
> >> ------------
> >>
> >> Sprint and LTS releases, when packaged, whenever possible, will be
> >> preferably distributed through different package channels
> >> (repositories).
> >>
> >> This is possible for repository types such as YUM/DNF repos[2].  In
> >> such cases, users can disable the regular channel, and enable the
> >> LTS version.  A request for the installation of Avocado packages
> >> will fetch the latest version available in the enabled
> >> repository.  If the LTS repository channel is enabled, the
> >> packages will receive minor updates (bugfixes only), until a new
> >> LTS version is released (roughly every 12 months).
> >>
> >> If the non-LTS channel is enabled, users will receive updates every
> >> 3-4 weeks.
> >>
> >> On other types of repos such as PyPI[3] which have no concept of
> >> "sub-repos" or "channels", users can request a version smaller than
> >> the version that succeeds the current LTS to get the latest LTS
> >> (including minor releases).  Suppose the current LTS major version
> >> is 52, but there have been minor releases 52.1 and 52.2.  By
> >> running:
> >>
> >>    $ pip install 'avocado-framework<53.0'
> >>
> >> pip provide LTS version 52.2.  If 52.3 gets released, they will be
> >> automatically deployed instead.  When a new LTS is released, users
> >> would still get the latest minor release from the 52.0 series,
> >> unless they update the version specification.
> >>
> >> The existence of LTS releases should never be used as an excuse
> >> to break a Sprint Release or to introduce gratuitous
> >> incompatibilities there. In other words, Sprint Releases should
> >> still be taken seriously, just as they are today.
> >>
> >>
> >> Timeline example
> >> ----------------
> >>
> >> Assume each sprint is taking 3 weeks.  Notice how multiple LTS
> >> releases can co-exist before EOL.
> >>  
> > It should start with:
> > 
> >     LTS release 36.0 (previous LTS release)
> >     ...
> >   
> 
> Good point.
> 
> >>     sprint release 49.0
> >>     sprint release 50.0  
> >>        --> start preparing a LTS release, so 51.0 is a beta LTS  
> >>     sprint release 51.0
> >>     LTS release 52.0 (52lts branch is created, packages go into
> >> LTS repo) sprint release 53.0  
> >>       --> bug that also affects 52.0 is found, fix gets added to  
> >>           master and 52lts branches
> >>     sprint release 54.0 + LTS 52.1  
> >>       --> bug that also affects 52.0 is found, fix gets added to  
> > it affects 52.1
> > 
> > Also I'd mention that it was a minor bug (as we did not bother
> > releasing a new version not even when new sprint was released.
> >   
> 
> Yep, good point.
> 
> Thanks for the feedback,
> - Cleber.
>