[Avocado-devel] [RFC v3]: Avocado maintainability and integration with avocado-vt

Tue Apr 25 16:05:06 UTC 2017

Dne 20.4.2017 v 03:42 Cleber Rosa napsal(a):
> Hi Folks.
>
Hello Cleber,

thank you for the updates, in general it's good, I have few minor 
suggestions in-line.

> This RFC contains proposals and clarifications regarding the
> maintenance and release processes of Avocado.
>
> We understand there are multiple teams currently depending on the
> stability of Avocado and we don't want their work to be disrupted by
> incompatibilities nor instabilities in new releases.
>
> This version is a minor update to RFC version 2[1], which drove the
> release of Avocado 36.0 LTS.  The Avocado team has plans for a new LTS
> release in the near future, so please consider reading and providing
> feedback on the proposals here.
>
> TL;DR:
>
>   We plan to keep the current approach of sprint releases every 3-4
>   weeks, but we're introducing "Long Term Stability" releases which
>   should be adopted in production environments where users can't keep
>   up with frequent upgrades.
>
> Changes from v2:
>   - Wording changes on second paragraph ("... nor instabilities...")
>   - Clarified on "Introduction" that change of behavior is introduced
>     between regular releases
>   - Updated distro versions for which official packages are built
>   - Add more clear explanation on official packages on the various
>     hardware platforms
>   - Used more recent version numbers as examples, and the planned
>     new LTS version too
>   - Explain how users can get the LTS version when using tools such as
>     pip
>   - Simplified the timeline example, with examples that will possibly
>     match the future versions and releases
>   - Documented current status of avocado-vt releases and packages
>
> Changes from v1:
>   - Changed "Support" to "Stability" and "supported" to "maintained"
>     [Jeff Nelson]
>   - Misc improvements and clarifications in the
>     supportability/stability statements [Jeff Nelson, me]
>   - Fixed a few typos [Jeff Nelson, me]
>
>
> Introduction
> --------------
>
> We make new releases of Avocado every 3-4 weeks on average.  In theory
> at least, we're very careful with backwards compatibility.  We test
> Avocado for regressions and we try to document any issues, so
> upgrading to a new version should be (again, in theory) safe.
>
> But in practice both intended and unintended changes are introduced
> during development, and both can be frustrating for conservative
> users. We also understand it's not feasible for users to upgrade
> Avocado very frequently in a production environment.
>
> The objective of this RFC is to clarify our maintenance practices and
> introduce Long Term Stability (LTS) releases, which are intended to
> solve, or at least mitigate, these problems.
>
>
> Our definition of maintained, or stable
> ---------------------------------------
>
> First of all, Avocado and its sub-projects are provided 'AS IS' and
> WITHOUT ANY WARRANTY, as described in the LICENSE file.
>
> The process described here doesn't imply any commitments or
> promises. It's just a set of best practices and recommendations.
>
> When something is identified as "stable" or "maintained", it means the
> development community makes a conscious effort to keep it working and
> consider reports of bugs and issues as high priorities.  Fixes
> submitted for these issues will also be considered high priorities,
> although they will be accepted only if they pass the general
> acceptance criteria for new contributions (design, quality,
> documentation, testing, etc), at the development team discretion.
>
>
> Maintained projects and platforms
> ---------------------------------
>
> The only maintained project as of today is the Avocado Test Runner,
> including its APIs and core plugins (the contents of the main avocado
> git repository).
>
> Other projects kept under the "Avocado Umbrella" in github may be
> maintained by different teams (e.g.: avocado-vt) or be considered
> experimental (e.g.: avocado-server and avocado-virt).
>
> More about avocado-vt in its own section further down.
>
> As a general rule, fixes and bug reports for Avocado when running in
> any modern Linux distribution are welcome.
>
> But given the limited capacity of the development team, packaged
> versions of Avocado will be tested and maintained only for the
> following Linux distributions:
>
>   * RHEL 7.x (latest)
>   * Fedora (stable releases from the Fedora projects)
>
> Currently all packages produced by the Avocado projects are "noarch".
> That means that they could be installable on any hardware platform.
> Still, the development team will currently attempt to provide versions
> that are stable for the following platforms:
>
>   * x86
>   * ppc64
Do we still care about ppc64 BE? And how about the widely spreading 
aarch64, do we want to include it?

>   * ppc64le
>
> Contributions from the community to maintain other platforms and
> operating systems are very welcome.
>
> The lists above may change without prior notice.
>
>
> Avocado Releases
> ----------------
>
> The proposal is to have two different types of Avocado releases:
>
> 1. Sprint Releases:
>   (This is the model we currently adopt in Avocado)
>
>   They happen every 3-4 weeks (the schedule is not fixed) and
>   their versions are numbered serially, with decimal digits in
>   the format <major>.<minor>.  Examples: 47.0, 48.0, 49.0.  Minor
>   releases are rare, but necessary to correct some major issue
>   with the original release (47.1, 47.2, etc).
>
>   Only the latest Sprint Release is maintained.
>
>   In Sprint Releases we make a conscious effort to keep backwards
>   compatibility with the previous version (APIs and behavior) and
>   as a general rule and best practice, incompatible changes in
>   Sprint Releases should be documented in the release notes and
>   if possible deprecated slowly, to give users time to adapt
>   their environments.
>
>   But we understand changes are inevitable as the software
>   evolves and therefore there's no absolute promise for API and
>   behavioral stability.
>
> 2. Long Term Stability (LTS) Releases:
>
>   LTS releases should happen whenever the team feels the code is
>   stable enough to be maintained for a longer period of time, ideally
>   once or twice per year (no fixed schedule).
>
>   They should be maintained for 18 months, receiving fixes for major
>   bugs in the form of minor (sub-)releases. With the exception of
>   these fixes, no API or behavior should change in a minor LTS
>   release.
>
>   They will be versioned just like Sprint Releases, so looking at the
>   version number alone will not reveal the differentiate release
>   process and stability characteristics.
>
>   In practice each major LTS release will imply in the creation of a
>   git branch where only serious issues affecting users will be
>   fixed. The code in a LTS branch is stable, frozen for new features.
>
>   Notice that although within a LTS release there's a expectation
>   of stability because the code is frozen, different (major) LTS
>   releases may include changes in behavior, API incompatibilities
>   and new features. The development team will make a considerable
>   effort to minimize and properly document these changes (changes
>   when comparing it to the last major LTS release).
>
>   Sprint Releases are replaced by LTS releases. I.e., in the cycle
>   when 52.0 (LTS) is released, that's also the version used as a
>   Sprint Release (there's no 52.0 -- non LTS -- in this case).
>
>   New LTS releases should be done carefully, with ample time for
>   announcements, testing and documentation.  It's recommended
>   that one or two sprints are dedicated as preparations for a LTS
>   release, with a Sprint Release serving as a "LTS beta" release.
>
>   Similarly, there should be announcements about the end-of-life
>   (EOL) of a LTS release once it approaches its 18 months of
>   life.
>
>
> Misc details
How about calling this section deployment details?

> ------------
>
> Sprint and LTS releases, when packaged, whenever possible, will be
> preferably distributed through different package channels
> (repositories).
>
> This is possible for repository types such as YUM/DNF repos[2].  In
> such cases, users can disable the regular channel, and enable the LTS
> version.  A request for the installation of Avocado packages will
> fetch the latest version available in the enabled repository.  If the
> LTS repository channel is enabled, the packages will receive minor
> updates (bugfixes only), until a new LTS version is released (roughly
> every 12 months).
>
> If the non-LTS channel is enabled, users will receive updates every
> 3-4 weeks.
>
> On other types of repos such as PyPI[3] which have no concept of
> "sub-repos" or "channels", users can request a version smaller than
> the version that succeeds the current LTS to get the latest LTS
> (including minor releases).  Suppose the current LTS major version is
> 52, but there have been minor releases 52.1 and 52.2.  By running:
>
>    $ pip install 'avocado-framework<53.0'
>
> pip provide LTS version 52.2.  If 52.3 gets released, they will be
> automatically deployed instead.  When a new LTS is released, users
> would still get the latest minor release from the 52.0 series, unless
> they update the version specification.
>
> The existence of LTS releases should never be used as an excuse
> to break a Sprint Release or to introduce gratuitous
> incompatibilities there. In other words, Sprint Releases should
> still be taken seriously, just as they are today.
>
>
> Timeline example
> ----------------
>
> Assume each sprint is taking 3 weeks.  Notice how multiple LTS
> releases can co-exist before EOL.
>
It should start with:

     LTS release 36.0 (previous LTS release)
     ...

>     sprint release 49.0
>     sprint release 50.0
>        --> start preparing a LTS release, so 51.0 is a beta LTS
>     sprint release 51.0
>     LTS release 52.0 (52lts branch is created, packages go into LTS repo)
>     sprint release 53.0
>       --> bug that also affects 52.0 is found, fix gets added to
>           master and 52lts branches
>     sprint release 54.0 + LTS 52.1
>       --> bug that also affects 52.0 is found, fix gets added to
it affects 52.1

Also I'd mention that it was a minor bug (as we did not bother releasing 
a new version not even when new sprint was released.

>           master and 52lts branches
>     sprint release 55.0
>       --> critical bug that affects 52.1 *only* is found, fix gets
>           added to 52lts and 52.2 LTS is immediately released
>     sprint release 56.0
>     sprint release 57.0
>     sprint release 58.0
>     sprint release 59.0
>        --> EOL for 36lts (18 months since the release of 36.0lts)
>     ...
>
> avocado-vt
> ----------
>
> avocado-vt is an Avocado plugin that allows "VT tests" to be run
> inside Avocado.  It's a third-party project maintained mostly by
> Engineers from Red Hat QE with assistance from the Avocado team
> and other community members.
>
> It's a general consensus that QE teams use avocado-vt directly
> from git, usually following the master branch, which they
> control.
>
> There's no official maintenance or stability statement for
> avocado-vt.  Even though the upstream community is quite
> friendly and open to both contributions and bug reports,
> avocado-vt is made available without any promises for
> compatibility or supportability.
>
> When packaged and versioned, avocado-vt rpms should be considered
> just snapshots, available in packaged form as a convenience to
> users outside of the avocado-vt development community.  Again,
> they are made available without any promises of compatibility or
> stability.
>
> * Which Avocado version should be used by avocado-vt?
>
>   This is up to the avocado-vt community to decide, but the
>   current consensus is that to guarantee some stability in
>   production environments, avocado-vt should stick to a specific
>   LTS release of Avocado. In other words, the Avocado team
>   recommends production users of avocado-vt not to install Avocado
>   from its master branch or upgrade it from Sprint Releases.
>
>   Given each LTS release will be maintained for 18 months, it
>   should be reasonable to expect avocado-vt to upgrade to a new
>   LTS release once a year or so. This process will be done with
>   support from the Avocado team to avoid disruptions, with proper
>   coordination via the avocado mailing lists.
>
>   In practice the Avocado development team will keep watching
>   avocado-vt to detect and document incompatibilities, so when
>   the time comes to do an upgrade in production, it's expected
>   that it should happen smoothly.
>
> * Will it be possible to use the latest Avocado and avocado-vt
>   together?
>
>   Users are welcome to *try* this combination.  The Avocado
>   development team itself will do it internally as a way to monitor
>   incompatibilities and regressions.
>
>   Whenever Avocado is released, a matching versioned snapshot of
>   avocado-vt will be made.  Packages containing those avocado-vt
>   snapshots, for convenience only, will be made available in the
>   regular Avocado repository.
>
> ---
>
> References:
>
>  [1] -
> https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html
>  [2] -
> http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#fedora-from-avocado-s-own-repo
>  [3] - https://pypi.python.org/pypi
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 502 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170425/c5f025e3/attachment.sig>