[Avocado-devel] [RFC v4]: Long Term Stability (was Avocado maintainability and integration with avocado-vt)
Cleber Rosa
crosa at redhat.com
Fri Apr 28 15:31:07 UTC 2017
On 04/28/2017 11:17 AM, Jeff Nelson wrote:
> On Fri, 28 Apr 2017 15:58:11 +0200
> Amador Segundo <asegundo at redhat.com> wrote:
>
>> On Thu, Apr 27, 2017 at 5:15 PM, Cleber Rosa <crosa at redhat.com> wrote:
>>>
>>>
>>> On 04/27/2017 04:04 AM, Amador Segundo wrote:
>>>> On Thu, Apr 27, 2017 at 1:07 AM, Cleber Rosa <crosa at redhat.com> wrote:
>>>>> ==========================
>>>>> RFC: Long Term Stability
>>>>> ==========================
>>>>>
>>>>> This RFC contains proposals and clarifications regarding the
>>>>> maintenance and release processes of Avocado.
>>>>>
>>>>> We understand there are multiple teams currently depending on the
>>>>> stability of Avocado and we don't want their work to be disrupted by
>>>>> incompatibilities nor instabilities in new releases.
>>>>>
>>>>> This version is a minor update to previous versions of the same RFC
>>>>> (see `Changelog`_) which drove the release of Avocado 36.0 LTS. The
>>>>> Avocado team has plans for a new LTS release in the near future, so
>>>>> please consider reading and providing feedback on the proposals here.
>>>>>
>>>>> TL;DR
>>>>> =====
>>>>>
>>>>> We plan to keep the current approach of sprint releases every 3-4
>>>>> weeks, but we're introducing "Long Term Stability" releases which
>>>>> should be adopted in production environments where users can't keep up
>>>>> with frequent upgrades.
>>>>>
>>>>> Changelog
>>>>> =========
>>>>>
>>>>> Changes from `Version 3`_:
>>>>>
>>>>> * Converted formatting to REStructuredText
>>>>> * Replaced "me" mentions on version 1 changelog with proper name
>>>>> (Ademar Reis)
>>>>> * Renamed section "Misc Details" to `Deployment Details`_
>>>>> * Renamed "avocado-vt" to "Avocado-VT"
>>>>> * Start the timeline example with version 36.0
>>>>> * Be explicit on timeline example that a minor bug did not generate
>>>>> an immediate release
>>>>>
>>>>> Changes from `Version 2`_:
>>>>>
>>>>> * Wording changes on second paragraph ("... nor instabilities...")
>>>>> * Clarified on "Introduction" that change of behavior is introduced
>>>>> between regular releases
>>>>> * Updated distro versions for which official packages are built
>>>>> * Add more clear explanation on official packages on the various
>>>>> hardware platforms
>>>>> * Used more recent version numbers as examples, and the planned
>>>>> new LTS version too
>>>>> * Explain how users can get the LTS version when using tools such as
>>>>> pip
>>>>> * Simplified the timeline example, with examples that will possibly
>>>>> match the future versions and releases
>>>>> * Documented current status of Avocado-VT releases and packages
>>>>>
>>>>> Changes from `Version 1`_:
>>>>>
>>>>> * Changed "Support" to "Stability" and "supported" to "maintained"
>>>>> [Jeff Nelson]
>>>>> * Misc improvements and clarifications in the
>>>>> supportability/stability statements [Jeff Nelson, Ademar Reis]
>>>>> * Fixed a few typos [Jeff Nelson, Ademar Reis]
>>>>>
>>>>> Introduction
>>>>> ============
>>>>>
>>>>> We make new releases of Avocado every 3-4 weeks on average. In theory
>>>>> at least, we're very careful with backwards compatibility. We test
>>>>> Avocado for regressions and we try to document any issues, so
>>>>> upgrading to a new version should be (again, in theory) safe.
>>>>>
>>>>> But in practice both intended and unintended changes are introduced
>>>>> during development, and both can be frustrating for conservative
>>>>> users. We also understand it's not feasible for users to upgrade
>>>>> Avocado very frequently in a production environment.
>>>>>
>>>>> The objective of this RFC is to clarify our maintenance practices and
>>>>> introduce Long Term Stability (LTS) releases, which are intended to
>>>>> solve, or at least mitigate, these problems.
>>>>>
>>>>>
>>>>> Our definition of maintained, or stable
>>>>> =======================================
>>>>>
>>>>> First of all, Avocado and its sub-projects are provided 'AS IS' and
>>>>> WITHOUT ANY WARRANTY, as described in the LICENSE file.
>>>>>
>>>>> The process described here doesn't imply any commitments or
>>>>> promises. It's just a set of best practices and recommendations.
>>>>>
>>>>> When something is identified as "stable" or "maintained", it means the
>>>>> development community makes a conscious effort to keep it working and
>>>>> consider reports of bugs and issues as high priorities. Fixes
>>>>> submitted for these issues will also be considered high priorities,
>>>>> although they will be accepted only if they pass the general
>>>>> acceptance criteria for new contributions (design, quality,
>>>>> documentation, testing, etc), at the development team discretion.
>>>>>
>>>>>
>>>>> Maintained projects and platforms
>>>>> =================================
>>>>>
>>>>> The only maintained project as of today is the Avocado Test Runner,
>>>>> including its APIs and core plugins (the contents of the main avocado
>>>>> git repository).
>>>>>
>>>>> Other projects kept under the "Avocado Umbrella" in github may be
>>>>> maintained by different teams (e.g.: Avocado-VT) or be considered
>>>>> experimental (e.g.: avocado-server and avocado-virt).
>>>>>
>>>>> More about Avocado-VT in its own section further down.
>>>>>
>>>>> As a general rule, fixes and bug reports for Avocado when running in
>>>>> any modern Linux distribution are welcome.
>>>>>
>>>>> But given the limited capacity of the development team, packaged
>>>>> versions of Avocado will be tested and maintained only for the
>>>>> following Linux distributions:
>>>>>
>>>>> * RHEL 7.x (latest)
>>>>> * Fedora (stable releases from the Fedora projects)
>>>>>
>>>>> Currently all packages produced by the Avocado projects are "noarch".
>>>>> That means that they could be installable on any hardware platform.
>>>>> Still, the development team will currently attempt to provide versions
>>>>> that are stable for the following platforms:
>>>>>
>>>>> * x86
>>>>> * ppc64
>>>>> * ppc64le
>>>>>
>>>>> Contributions from the community to maintain other platforms and
>>>>> operating systems are very welcome.
>>>>>
>>>>> The lists above may change without prior notice.
>>>>>
>>>>> Avocado Releases
>>>>> ================
>>>>>
>>>>> The proposal is to have two different types of Avocado releases:
>>>>>
>>>>> Sprint Releases
>>>>> ---------------
>>>>>
>>>>> (This is the model we currently adopt in Avocado)
>>>>>
>>>>> They happen every 3-4 weeks (the schedule is not fixed) and
>>>>> their versions are numbered serially, with decimal digits in
>>>>> the format <major>.<minor>. Examples: 47.0, 48.0, 49.0. Minor
>>>>> releases are rare, but necessary to correct some major issue
>>>>> with the original release (47.1, 47.2, etc).
>>>>>
>>>>> Only the latest Sprint Release is maintained.
>>>>>
>>>>> In Sprint Releases we make a conscious effort to keep backwards
>>>>> compatibility with the previous version (APIs and behavior) and
>>>>> as a general rule and best practice, incompatible changes in
>>>>> Sprint Releases should be documented in the release notes and
>>>>> if possible deprecated slowly, to give users time to adapt
>>>>> their environments.
>>>>>
>>>>> But we understand changes are inevitable as the software
>>>>> evolves and therefore there's no absolute promise for API and
>>>>> behavioral stability.
>>>>>
>>>>> Long Term Stability (LTS) Releases
>>>>> ----------------------------------
>>>>>
>>>>> LTS releases should happen whenever the team feels the code is
>>>>> stable enough to be maintained for a longer period of time, ideally
>>>>> once or twice per year (no fixed schedule).
>>>>>
>>>>> They should be maintained for 18 months, receiving fixes for major
>>>>> bugs in the form of minor (sub-)releases. With the exception of
>>>>> these fixes, no API or behavior should change in a minor LTS
>>>>> release.
>>>>>
>>>>> They will be versioned just like Sprint Releases, so looking at the
>>>>> version number alone will not reveal the differentiate release
>>>>> process and stability characteristics.
>>>>>
>>>>> In practice each major LTS release will imply in the creation of a
>>>>> git branch where only serious issues affecting users will be
>>>>> fixed. The code in a LTS branch is stable, frozen for new features.
>>>>
>>>> I'd improve the sentence "only serious issues affecting users will be
>>>> fixed". In fact, we "backport" (meaning upstream first) "any issues"
>>>> (not only the serious ones) that are possible to be fixed *and* are
>>>> affecting users.
>>>>
>>>
>>> OK, good point. My suggestion is to use "important issues", because the
>>> definition of a "bug" and a "serious issue" is too relative. We may
>>> identify a bug in LTS that is mostly harmless, and decide to not fix it
>>> because it could require major changes. Anyway, my proposal:
>>>
>>> "In practice each major LTS release will imply in the creation of a git
>>> branch where only important issues affecting users will be fixed,
>>> usually as a backport of a fix initially applied upstream. The code in
>>> a LTS branch is stable, frozen for new features."
>>>
>>> How does that sound?
>>
>> Yup, looks great.
>>
>>>
>>>> Not sure if it is also worth it to define 'backport'. Depending on the
>>>> code distance between the LTS and the master branch, it can be a
>>>> simple 'cherry-pick' or a completely different fix in terms of code,
>>>> as long as the bug gets fixed.
>>>>
>>>
>>> I'd expect readers to be familiar with the concept of backport. I'm
>>> reluctant about introducing the concept in this document.
>>>
>>>>>
>>>>> Notice that although within a LTS release there's a expectation
>>>>> of stability because the code is frozen, different (major) LTS
>>>>> releases may include changes in behavior, API incompatibilities
>>>>> and new features. The development team will make a considerable
>>>>> effort to minimize and properly document these changes (changes
>>>>> when comparing it to the last major LTS release).
>>>>>
>>>>> Sprint Releases are replaced by LTS releases. I.e., in the cycle
>>>>> when 52.0 (LTS) is released, that's also the version used as a
>>>>> Sprint Release (there's no 52.0 -- non LTS -- in this case).
>>>>>
>>>>> New LTS releases should be done carefully, with ample time for
>>>>> announcements, testing and documentation. It's recommended
>>>>> that one or two sprints are dedicated as preparations for a LTS
>>>>> release, with a Sprint Release serving as a "LTS beta" release.
>>>>>
>>>>> Similarly, there should be announcements about the end-of-life
>>>>> (EOL) of a LTS release once it approaches its 18 months of
>>>>> life.
>>>>>
>>>>>
>>>>> Deployment details
>>>>> ==================
>>>>>
>>>>> Sprint and LTS releases, when packaged, whenever possible, will be
>>>>> preferably distributed through different package channels
>>>>> (repositories).
>>>>>
>>>>> This is possible for repository types such as `YUM/DNF repos`_. In
>>>>> such cases, users can disable the regular channel, and enable the LTS
>>>>> version. A request for the installation of Avocado packages will
>>>>> fetch the latest version available in the enabled repository. If the
>>>>> LTS repository channel is enabled, the packages will receive minor
>>>>> updates (bugfixes only), until a new LTS version is released (roughly
>>>>> every 12 months).
>>>>>
>>>>> If the non-LTS channel is enabled, users will receive updates every
>>>>> 3-4 weeks.
>>>>>
>>>>> On other types of repos such as `PyPI`_ which have no concept of
>>>>> "sub-repos" or "channels", users can request a version smaller than
>>>>> the version that succeeds the current LTS to get the latest LTS
>>>>> (including minor releases). Suppose the current LTS major version is
>>>>> 52, but there have been minor releases 52.1 and 52.2. By running::
>>>>>
>>>>> pip install 'avocado-framework<53.0'
>>>>>
>>>>> pip provide LTS version 52.2. If 52.3 gets released, they will be
>>>>> automatically deployed instead. When a new LTS is released, users
>>>>> would still get the latest minor release from the 52.0 series, unless
>>>>> they update the version specification.
>>>>>
>>>>> The existence of LTS releases should never be used as an excuse
>>>>> to break a Sprint Release or to introduce gratuitous
>>>>> incompatibilities there. In other words, Sprint Releases should
>>>>> still be taken seriously, just as they are today.
>>>>>
>>>>>
>>>>> Timeline example
>>>>> ================
>>>>>
>>>>> Assume each sprint is taking 3 weeks. Notice how multiple LTS
>>>>> releases can co-exist before EOL:
>>>>>
>>>>> 36.0
>>>>> * **LTS** release (the only LTS release available at the time of
>>>>> writing)
>>>>>
>>>>> 37.0 .. 49.0
>>>>> * sprint releases
>>>>>
>>>>> 50.0
>>>>> * sprint release
>>>>> * start preparing a LTS release, so 51.0 will be a **beta LTS**
>>>>>
>>>>> 51.0
>>>>> * sprint release
>>>>> * **beta LTS** release
>>>>>
>>>>> 52.0
>>>>> * **LTS** release
>>>>> * 52lts branch is created
>>>>> * packages go into LTS repo
>>>>> * both **36.x LTS** and **52.x LTS** maintained from this point on
>>>>>
>>>>> 53.0
>>>>> * sprint release
>>>>> * bug that also affects 52.0 is found, fix gets added to master and
>>>>> 52lts branches
>>>>
>>>> Fix gets added to master and backported to supported LTS branches (at
>>>> this point, 52lts and 36lts).
>>>>
>>>
>>> Yeah, good point (assuming it also applies to 36lts, as you mentioned on
>>> the other reply).
>>
>> Yes, I think it makes sense to mention that here to make clear that
>> all supported branches will receive the fix, as long as there are
>> affected users.
>>
>> My proposal:
>>
>> 53.0
>> * sprint release
>> * bug that also affects users of 52.0 and 36.0 is found, fix gets
>> added to master and is backported to 52lts and 36lts branches.
>>
>> 54.0
>> * sprint release 54.0 **AND** LTS 52.1 release **AND** LTS 36.5 release.
>> * minor bug that also affects users of 52.1 is found, fix gets added
>> to master and is backported to 52lts branch.
>>
>> 55.0
>> * sprint release 55.0 **AND** LTS 52.2 release
>> * critical bug that affects 52.2 *only* is found, fix gets added to
>> 52lts and 52.3 LTS is immediately released
>
> Maybe I'm being pedantic, but how about some visual separation between
> the sprint release and the things that follow? It looks to me as if
> one could interpret everything under a numbered sprint release as
> happening at the same time, when in fact we want to point out that
> (a) bug discovery can happen at any time
> (b) the bugfix occurs next
> (c) the severity of the defect determines the timing of the release
> (c.1) moderate and minor bugfixes to lts branches are held until the
> next sprint release
> (c.2) critical bugs are released asynchronously, without waiting for the
> next sprint release.
>
Those are all valid and true points. I've spent quite some time trying
to fit the timeline into a easy to read format (that is also valid
REstructuredText), so I won't promise that I'll be able to squeeze all
that in.
At the very least, I'll include your listing above as a summary of how
things work.
> 53
> * sprint release 53.0
>
> * moderate bug that affects users of master, 52lts and 36lts is found
> * fix is added to master and backported to 52lts and 36lts
>
> 54
> * sprint release 54.0 **AND** 36lts release 36.5 **AND** 52lts release 52.1
>
> * minor bug that affects users of master and 52lts is found
> * fix is added to master and backported to 52lts
>
> 55
> * sprint release 55.0 **AND** 52lts release 52.2
>
> * critical bug is found that affects users of 52lts *only*
> * fix is made to 52lts and 52lts release 52.3 is immediately announced
>
> I dunno; just something to think about.
>
> -Jeff
>
>
Yep, thinking about it! Thanks for the feedback.
- Cleber.
>>>
>>>>>
>>>>> 54.0
>>>>> * sprint release 54.0 **AND** LTS 52.1 release
>>>>> * minor bug that also affects 52.0 is found, fix gets added to
>>>>> master and 52lts branches
>>>>
>>>> Fix gets added to master and backported to supported LTS branches (at
>>>> this point, 52lts and 36lts).
>>>>
>>>>>
>>>>> 55.0
>>>>> * sprint release
>>>>> * critical bug that affects 52.1 *only* is found, fix gets added to
>>>>> 52lts and 52.2 LTS is immediately released
>>>>>
>>>>> 56.0
>>>>> * sprint release
>>>>>
>>>>> 57.0
>>>>> * sprint release
>>>>>
>>>>> 58.0
>>>>> * sprint release
>>>>>
>>>>> 59.0
>>>>> * sprint release
>>>>> * EOL for **36.x LTS** (18 months since the release of 36.0)
>>>>>
>>>>> Avocado-VT
>>>>> ==========
>>>>>
>>>>> Avocado-VT is an Avocado plugin that allows "VT tests" to be run
>>>>> inside Avocado. It's a third-party project maintained mostly by
>>>>> Engineers from Red Hat QE with assistance from the Avocado team
>>>>> and other community members.
>>>>>
>>>>> It's a general consensus that QE teams use Avocado-VT directly
>>>>> from git, usually following the master branch, which they
>>>>> control.
>>>>>
>>>>> There's no official maintenance or stability statement for
>>>>> Avocado-VT. Even though the upstream community is quite
>>>>> friendly and open to both contributions and bug reports,
>>>>> Avocado-VT is made available without any promises for
>>>>> compatibility or supportability.
>>>>>
>>>>> When packaged and versioned, Avocado-VT rpms should be considered
>>>>> just snapshots, available in packaged form as a convenience to
>>>>> users outside of the Avocado-VT development community. Again,
>>>>> they are made available without any promises of compatibility or
>>>>> stability.
>>>>>
>>>>> * Which Avocado version should be used by Avocado-VT?
>>>>>
>>>>> This is up to the Avocado-VT community to decide, but the
>>>>> current consensus is that to guarantee some stability in
>>>>> production environments, Avocado-VT should stick to a specific
>>>>> LTS release of Avocado. In other words, the Avocado team
>>>>> recommends production users of Avocado-VT not to install Avocado
>>>>> from its master branch or upgrade it from Sprint Releases.
>>>>>
>>>>> Given each LTS release will be maintained for 18 months, it
>>>>> should be reasonable to expect Avocado-VT to upgrade to a new
>>>>> LTS release once a year or so. This process will be done with
>>>>> support from the Avocado team to avoid disruptions, with proper
>>>>> coordination via the avocado mailing lists.
>>>>>
>>>>> In practice the Avocado development team will keep watching
>>>>> Avocado-VT to detect and document incompatibilities, so when
>>>>> the time comes to do an upgrade in production, it's expected
>>>>> that it should happen smoothly.
>>>>>
>>>>> * Will it be possible to use the latest Avocado and Avocado-VT
>>>>> together?
>>>>>
>>>>> Users are welcome to *try* this combination. The Avocado
>>>>> development team itself will do it internally as a way to monitor
>>>>> incompatibilities and regressions.
>>>>>
>>>>> Whenever Avocado is released, a matching versioned snapshot of
>>>>> Avocado-VT will be made. Packages containing those Avocado-VT
>>>>> snapshots, for convenience only, will be made available in the
>>>>> regular Avocado repository.
>>>>>
>>>>> .. _Version 1:
>>>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00006.html
>>>>> .. _Version 2:
>>>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html
>>>>> .. _Version 3:
>>>>> https://www.redhat.com/archives/avocado-devel/2017-April/msg00032.html
>>>>> .. _YUM/DNF repos:
>>>>> http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#fedora-from-avocado-s-own-repo
>>>>> .. _PyPI: https://pypi.python.org/pypi
>>>>>
>>>>>
>>>>> --
>>>>> Cleber Rosa
>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
>>>>>
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
>>>
>
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170428/01c1b33a/attachment.sig>
More information about the Avocado-devel
mailing list