[Avocado-devel] [RFC v4]: Long Term Stability (was Avocado maintainability and integration with avocado-vt)
Amador Segundo
asegundo at redhat.com
Fri Apr 28 13:58:11 UTC 2017
On Thu, Apr 27, 2017 at 5:15 PM, Cleber Rosa <crosa at redhat.com> wrote:
>
>
> On 04/27/2017 04:04 AM, Amador Segundo wrote:
>> On Thu, Apr 27, 2017 at 1:07 AM, Cleber Rosa <crosa at redhat.com> wrote:
>>> ==========================
>>> RFC: Long Term Stability
>>> ==========================
>>>
>>> This RFC contains proposals and clarifications regarding the
>>> maintenance and release processes of Avocado.
>>>
>>> We understand there are multiple teams currently depending on the
>>> stability of Avocado and we don't want their work to be disrupted by
>>> incompatibilities nor instabilities in new releases.
>>>
>>> This version is a minor update to previous versions of the same RFC
>>> (see `Changelog`_) which drove the release of Avocado 36.0 LTS. The
>>> Avocado team has plans for a new LTS release in the near future, so
>>> please consider reading and providing feedback on the proposals here.
>>>
>>> TL;DR
>>> =====
>>>
>>> We plan to keep the current approach of sprint releases every 3-4
>>> weeks, but we're introducing "Long Term Stability" releases which
>>> should be adopted in production environments where users can't keep up
>>> with frequent upgrades.
>>>
>>> Changelog
>>> =========
>>>
>>> Changes from `Version 3`_:
>>>
>>> * Converted formatting to REStructuredText
>>> * Replaced "me" mentions on version 1 changelog with proper name
>>> (Ademar Reis)
>>> * Renamed section "Misc Details" to `Deployment Details`_
>>> * Renamed "avocado-vt" to "Avocado-VT"
>>> * Start the timeline example with version 36.0
>>> * Be explicit on timeline example that a minor bug did not generate
>>> an immediate release
>>>
>>> Changes from `Version 2`_:
>>>
>>> * Wording changes on second paragraph ("... nor instabilities...")
>>> * Clarified on "Introduction" that change of behavior is introduced
>>> between regular releases
>>> * Updated distro versions for which official packages are built
>>> * Add more clear explanation on official packages on the various
>>> hardware platforms
>>> * Used more recent version numbers as examples, and the planned
>>> new LTS version too
>>> * Explain how users can get the LTS version when using tools such as
>>> pip
>>> * Simplified the timeline example, with examples that will possibly
>>> match the future versions and releases
>>> * Documented current status of Avocado-VT releases and packages
>>>
>>> Changes from `Version 1`_:
>>>
>>> * Changed "Support" to "Stability" and "supported" to "maintained"
>>> [Jeff Nelson]
>>> * Misc improvements and clarifications in the
>>> supportability/stability statements [Jeff Nelson, Ademar Reis]
>>> * Fixed a few typos [Jeff Nelson, Ademar Reis]
>>>
>>> Introduction
>>> ============
>>>
>>> We make new releases of Avocado every 3-4 weeks on average. In theory
>>> at least, we're very careful with backwards compatibility. We test
>>> Avocado for regressions and we try to document any issues, so
>>> upgrading to a new version should be (again, in theory) safe.
>>>
>>> But in practice both intended and unintended changes are introduced
>>> during development, and both can be frustrating for conservative
>>> users. We also understand it's not feasible for users to upgrade
>>> Avocado very frequently in a production environment.
>>>
>>> The objective of this RFC is to clarify our maintenance practices and
>>> introduce Long Term Stability (LTS) releases, which are intended to
>>> solve, or at least mitigate, these problems.
>>>
>>>
>>> Our definition of maintained, or stable
>>> =======================================
>>>
>>> First of all, Avocado and its sub-projects are provided 'AS IS' and
>>> WITHOUT ANY WARRANTY, as described in the LICENSE file.
>>>
>>> The process described here doesn't imply any commitments or
>>> promises. It's just a set of best practices and recommendations.
>>>
>>> When something is identified as "stable" or "maintained", it means the
>>> development community makes a conscious effort to keep it working and
>>> consider reports of bugs and issues as high priorities. Fixes
>>> submitted for these issues will also be considered high priorities,
>>> although they will be accepted only if they pass the general
>>> acceptance criteria for new contributions (design, quality,
>>> documentation, testing, etc), at the development team discretion.
>>>
>>>
>>> Maintained projects and platforms
>>> =================================
>>>
>>> The only maintained project as of today is the Avocado Test Runner,
>>> including its APIs and core plugins (the contents of the main avocado
>>> git repository).
>>>
>>> Other projects kept under the "Avocado Umbrella" in github may be
>>> maintained by different teams (e.g.: Avocado-VT) or be considered
>>> experimental (e.g.: avocado-server and avocado-virt).
>>>
>>> More about Avocado-VT in its own section further down.
>>>
>>> As a general rule, fixes and bug reports for Avocado when running in
>>> any modern Linux distribution are welcome.
>>>
>>> But given the limited capacity of the development team, packaged
>>> versions of Avocado will be tested and maintained only for the
>>> following Linux distributions:
>>>
>>> * RHEL 7.x (latest)
>>> * Fedora (stable releases from the Fedora projects)
>>>
>>> Currently all packages produced by the Avocado projects are "noarch".
>>> That means that they could be installable on any hardware platform.
>>> Still, the development team will currently attempt to provide versions
>>> that are stable for the following platforms:
>>>
>>> * x86
>>> * ppc64
>>> * ppc64le
>>>
>>> Contributions from the community to maintain other platforms and
>>> operating systems are very welcome.
>>>
>>> The lists above may change without prior notice.
>>>
>>> Avocado Releases
>>> ================
>>>
>>> The proposal is to have two different types of Avocado releases:
>>>
>>> Sprint Releases
>>> ---------------
>>>
>>> (This is the model we currently adopt in Avocado)
>>>
>>> They happen every 3-4 weeks (the schedule is not fixed) and
>>> their versions are numbered serially, with decimal digits in
>>> the format <major>.<minor>. Examples: 47.0, 48.0, 49.0. Minor
>>> releases are rare, but necessary to correct some major issue
>>> with the original release (47.1, 47.2, etc).
>>>
>>> Only the latest Sprint Release is maintained.
>>>
>>> In Sprint Releases we make a conscious effort to keep backwards
>>> compatibility with the previous version (APIs and behavior) and
>>> as a general rule and best practice, incompatible changes in
>>> Sprint Releases should be documented in the release notes and
>>> if possible deprecated slowly, to give users time to adapt
>>> their environments.
>>>
>>> But we understand changes are inevitable as the software
>>> evolves and therefore there's no absolute promise for API and
>>> behavioral stability.
>>>
>>> Long Term Stability (LTS) Releases
>>> ----------------------------------
>>>
>>> LTS releases should happen whenever the team feels the code is
>>> stable enough to be maintained for a longer period of time, ideally
>>> once or twice per year (no fixed schedule).
>>>
>>> They should be maintained for 18 months, receiving fixes for major
>>> bugs in the form of minor (sub-)releases. With the exception of
>>> these fixes, no API or behavior should change in a minor LTS
>>> release.
>>>
>>> They will be versioned just like Sprint Releases, so looking at the
>>> version number alone will not reveal the differentiate release
>>> process and stability characteristics.
>>>
>>> In practice each major LTS release will imply in the creation of a
>>> git branch where only serious issues affecting users will be
>>> fixed. The code in a LTS branch is stable, frozen for new features.
>>
>> I'd improve the sentence "only serious issues affecting users will be
>> fixed". In fact, we "backport" (meaning upstream first) "any issues"
>> (not only the serious ones) that are possible to be fixed *and* are
>> affecting users.
>>
>
> OK, good point. My suggestion is to use "important issues", because the
> definition of a "bug" and a "serious issue" is too relative. We may
> identify a bug in LTS that is mostly harmless, and decide to not fix it
> because it could require major changes. Anyway, my proposal:
>
> "In practice each major LTS release will imply in the creation of a git
> branch where only important issues affecting users will be fixed,
> usually as a backport of a fix initially applied upstream. The code in
> a LTS branch is stable, frozen for new features."
>
> How does that sound?
Yup, looks great.
>
>> Not sure if it is also worth it to define 'backport'. Depending on the
>> code distance between the LTS and the master branch, it can be a
>> simple 'cherry-pick' or a completely different fix in terms of code,
>> as long as the bug gets fixed.
>>
>
> I'd expect readers to be familiar with the concept of backport. I'm
> reluctant about introducing the concept in this document.
>
>>>
>>> Notice that although within a LTS release there's a expectation
>>> of stability because the code is frozen, different (major) LTS
>>> releases may include changes in behavior, API incompatibilities
>>> and new features. The development team will make a considerable
>>> effort to minimize and properly document these changes (changes
>>> when comparing it to the last major LTS release).
>>>
>>> Sprint Releases are replaced by LTS releases. I.e., in the cycle
>>> when 52.0 (LTS) is released, that's also the version used as a
>>> Sprint Release (there's no 52.0 -- non LTS -- in this case).
>>>
>>> New LTS releases should be done carefully, with ample time for
>>> announcements, testing and documentation. It's recommended
>>> that one or two sprints are dedicated as preparations for a LTS
>>> release, with a Sprint Release serving as a "LTS beta" release.
>>>
>>> Similarly, there should be announcements about the end-of-life
>>> (EOL) of a LTS release once it approaches its 18 months of
>>> life.
>>>
>>>
>>> Deployment details
>>> ==================
>>>
>>> Sprint and LTS releases, when packaged, whenever possible, will be
>>> preferably distributed through different package channels
>>> (repositories).
>>>
>>> This is possible for repository types such as `YUM/DNF repos`_. In
>>> such cases, users can disable the regular channel, and enable the LTS
>>> version. A request for the installation of Avocado packages will
>>> fetch the latest version available in the enabled repository. If the
>>> LTS repository channel is enabled, the packages will receive minor
>>> updates (bugfixes only), until a new LTS version is released (roughly
>>> every 12 months).
>>>
>>> If the non-LTS channel is enabled, users will receive updates every
>>> 3-4 weeks.
>>>
>>> On other types of repos such as `PyPI`_ which have no concept of
>>> "sub-repos" or "channels", users can request a version smaller than
>>> the version that succeeds the current LTS to get the latest LTS
>>> (including minor releases). Suppose the current LTS major version is
>>> 52, but there have been minor releases 52.1 and 52.2. By running::
>>>
>>> pip install 'avocado-framework<53.0'
>>>
>>> pip provide LTS version 52.2. If 52.3 gets released, they will be
>>> automatically deployed instead. When a new LTS is released, users
>>> would still get the latest minor release from the 52.0 series, unless
>>> they update the version specification.
>>>
>>> The existence of LTS releases should never be used as an excuse
>>> to break a Sprint Release or to introduce gratuitous
>>> incompatibilities there. In other words, Sprint Releases should
>>> still be taken seriously, just as they are today.
>>>
>>>
>>> Timeline example
>>> ================
>>>
>>> Assume each sprint is taking 3 weeks. Notice how multiple LTS
>>> releases can co-exist before EOL:
>>>
>>> 36.0
>>> * **LTS** release (the only LTS release available at the time of
>>> writing)
>>>
>>> 37.0 .. 49.0
>>> * sprint releases
>>>
>>> 50.0
>>> * sprint release
>>> * start preparing a LTS release, so 51.0 will be a **beta LTS**
>>>
>>> 51.0
>>> * sprint release
>>> * **beta LTS** release
>>>
>>> 52.0
>>> * **LTS** release
>>> * 52lts branch is created
>>> * packages go into LTS repo
>>> * both **36.x LTS** and **52.x LTS** maintained from this point on
>>>
>>> 53.0
>>> * sprint release
>>> * bug that also affects 52.0 is found, fix gets added to master and
>>> 52lts branches
>>
>> Fix gets added to master and backported to supported LTS branches (at
>> this point, 52lts and 36lts).
>>
>
> Yeah, good point (assuming it also applies to 36lts, as you mentioned on
> the other reply).
Yes, I think it makes sense to mention that here to make clear that
all supported branches will receive the fix, as long as there are
affected users.
My proposal:
53.0
* sprint release
* bug that also affects users of 52.0 and 36.0 is found, fix gets
added to master and is backported to 52lts and 36lts branches.
54.0
* sprint release 54.0 **AND** LTS 52.1 release **AND** LTS 36.5 release.
* minor bug that also affects users of 52.1 is found, fix gets added
to master and is backported to 52lts branch.
55.0
* sprint release 55.0 **AND** LTS 52.2 release
* critical bug that affects 52.2 *only* is found, fix gets added to
52lts and 52.3 LTS is immediately released
>
>>>
>>> 54.0
>>> * sprint release 54.0 **AND** LTS 52.1 release
>>> * minor bug that also affects 52.0 is found, fix gets added to
>>> master and 52lts branches
>>
>> Fix gets added to master and backported to supported LTS branches (at
>> this point, 52lts and 36lts).
>>
>>>
>>> 55.0
>>> * sprint release
>>> * critical bug that affects 52.1 *only* is found, fix gets added to
>>> 52lts and 52.2 LTS is immediately released
>>>
>>> 56.0
>>> * sprint release
>>>
>>> 57.0
>>> * sprint release
>>>
>>> 58.0
>>> * sprint release
>>>
>>> 59.0
>>> * sprint release
>>> * EOL for **36.x LTS** (18 months since the release of 36.0)
>>>
>>> Avocado-VT
>>> ==========
>>>
>>> Avocado-VT is an Avocado plugin that allows "VT tests" to be run
>>> inside Avocado. It's a third-party project maintained mostly by
>>> Engineers from Red Hat QE with assistance from the Avocado team
>>> and other community members.
>>>
>>> It's a general consensus that QE teams use Avocado-VT directly
>>> from git, usually following the master branch, which they
>>> control.
>>>
>>> There's no official maintenance or stability statement for
>>> Avocado-VT. Even though the upstream community is quite
>>> friendly and open to both contributions and bug reports,
>>> Avocado-VT is made available without any promises for
>>> compatibility or supportability.
>>>
>>> When packaged and versioned, Avocado-VT rpms should be considered
>>> just snapshots, available in packaged form as a convenience to
>>> users outside of the Avocado-VT development community. Again,
>>> they are made available without any promises of compatibility or
>>> stability.
>>>
>>> * Which Avocado version should be used by Avocado-VT?
>>>
>>> This is up to the Avocado-VT community to decide, but the
>>> current consensus is that to guarantee some stability in
>>> production environments, Avocado-VT should stick to a specific
>>> LTS release of Avocado. In other words, the Avocado team
>>> recommends production users of Avocado-VT not to install Avocado
>>> from its master branch or upgrade it from Sprint Releases.
>>>
>>> Given each LTS release will be maintained for 18 months, it
>>> should be reasonable to expect Avocado-VT to upgrade to a new
>>> LTS release once a year or so. This process will be done with
>>> support from the Avocado team to avoid disruptions, with proper
>>> coordination via the avocado mailing lists.
>>>
>>> In practice the Avocado development team will keep watching
>>> Avocado-VT to detect and document incompatibilities, so when
>>> the time comes to do an upgrade in production, it's expected
>>> that it should happen smoothly.
>>>
>>> * Will it be possible to use the latest Avocado and Avocado-VT
>>> together?
>>>
>>> Users are welcome to *try* this combination. The Avocado
>>> development team itself will do it internally as a way to monitor
>>> incompatibilities and regressions.
>>>
>>> Whenever Avocado is released, a matching versioned snapshot of
>>> Avocado-VT will be made. Packages containing those Avocado-VT
>>> snapshots, for convenience only, will be made available in the
>>> regular Avocado repository.
>>>
>>> .. _Version 1:
>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00006.html
>>> .. _Version 2:
>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html
>>> .. _Version 3:
>>> https://www.redhat.com/archives/avocado-devel/2017-April/msg00032.html
>>> .. _YUM/DNF repos:
>>> http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#fedora-from-avocado-s-own-repo
>>> .. _PyPI: https://pypi.python.org/pypi
>>>
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
>>>
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
>
More information about the Avocado-devel
mailing list