[Avocado-devel] [RFC v3]: Avocado maintainability and integration with avocado-vt
Cleber Rosa
crosa at redhat.com
Thu Apr 20 01:42:00 UTC 2017
Hi Folks.
This RFC contains proposals and clarifications regarding the
maintenance and release processes of Avocado.
We understand there are multiple teams currently depending on the
stability of Avocado and we don't want their work to be disrupted by
incompatibilities nor instabilities in new releases.
This version is a minor update to RFC version 2[1], which drove the
release of Avocado 36.0 LTS. The Avocado team has plans for a new LTS
release in the near future, so please consider reading and providing
feedback on the proposals here.
TL;DR:
We plan to keep the current approach of sprint releases every 3-4
weeks, but we're introducing "Long Term Stability" releases which
should be adopted in production environments where users can't keep
up with frequent upgrades.
Changes from v2:
- Wording changes on second paragraph ("... nor instabilities...")
- Clarified on "Introduction" that change of behavior is introduced
between regular releases
- Updated distro versions for which official packages are built
- Add more clear explanation on official packages on the various
hardware platforms
- Used more recent version numbers as examples, and the planned
new LTS version too
- Explain how users can get the LTS version when using tools such as
pip
- Simplified the timeline example, with examples that will possibly
match the future versions and releases
- Documented current status of avocado-vt releases and packages
Changes from v1:
- Changed "Support" to "Stability" and "supported" to "maintained"
[Jeff Nelson]
- Misc improvements and clarifications in the
supportability/stability statements [Jeff Nelson, me]
- Fixed a few typos [Jeff Nelson, me]
Introduction
--------------
We make new releases of Avocado every 3-4 weeks on average. In theory
at least, we're very careful with backwards compatibility. We test
Avocado for regressions and we try to document any issues, so
upgrading to a new version should be (again, in theory) safe.
But in practice both intended and unintended changes are introduced
during development, and both can be frustrating for conservative
users. We also understand it's not feasible for users to upgrade
Avocado very frequently in a production environment.
The objective of this RFC is to clarify our maintenance practices and
introduce Long Term Stability (LTS) releases, which are intended to
solve, or at least mitigate, these problems.
Our definition of maintained, or stable
---------------------------------------
First of all, Avocado and its sub-projects are provided 'AS IS' and
WITHOUT ANY WARRANTY, as described in the LICENSE file.
The process described here doesn't imply any commitments or
promises. It's just a set of best practices and recommendations.
When something is identified as "stable" or "maintained", it means the
development community makes a conscious effort to keep it working and
consider reports of bugs and issues as high priorities. Fixes
submitted for these issues will also be considered high priorities,
although they will be accepted only if they pass the general
acceptance criteria for new contributions (design, quality,
documentation, testing, etc), at the development team discretion.
Maintained projects and platforms
---------------------------------
The only maintained project as of today is the Avocado Test Runner,
including its APIs and core plugins (the contents of the main avocado
git repository).
Other projects kept under the "Avocado Umbrella" in github may be
maintained by different teams (e.g.: avocado-vt) or be considered
experimental (e.g.: avocado-server and avocado-virt).
More about avocado-vt in its own section further down.
As a general rule, fixes and bug reports for Avocado when running in
any modern Linux distribution are welcome.
But given the limited capacity of the development team, packaged
versions of Avocado will be tested and maintained only for the
following Linux distributions:
* RHEL 7.x (latest)
* Fedora (stable releases from the Fedora projects)
Currently all packages produced by the Avocado projects are "noarch".
That means that they could be installable on any hardware platform.
Still, the development team will currently attempt to provide versions
that are stable for the following platforms:
* x86
* ppc64
* ppc64le
Contributions from the community to maintain other platforms and
operating systems are very welcome.
The lists above may change without prior notice.
Avocado Releases
----------------
The proposal is to have two different types of Avocado releases:
1. Sprint Releases:
(This is the model we currently adopt in Avocado)
They happen every 3-4 weeks (the schedule is not fixed) and
their versions are numbered serially, with decimal digits in
the format <major>.<minor>. Examples: 47.0, 48.0, 49.0. Minor
releases are rare, but necessary to correct some major issue
with the original release (47.1, 47.2, etc).
Only the latest Sprint Release is maintained.
In Sprint Releases we make a conscious effort to keep backwards
compatibility with the previous version (APIs and behavior) and
as a general rule and best practice, incompatible changes in
Sprint Releases should be documented in the release notes and
if possible deprecated slowly, to give users time to adapt
their environments.
But we understand changes are inevitable as the software
evolves and therefore there's no absolute promise for API and
behavioral stability.
2. Long Term Stability (LTS) Releases:
LTS releases should happen whenever the team feels the code is
stable enough to be maintained for a longer period of time, ideally
once or twice per year (no fixed schedule).
They should be maintained for 18 months, receiving fixes for major
bugs in the form of minor (sub-)releases. With the exception of
these fixes, no API or behavior should change in a minor LTS
release.
They will be versioned just like Sprint Releases, so looking at the
version number alone will not reveal the differentiate release
process and stability characteristics.
In practice each major LTS release will imply in the creation of a
git branch where only serious issues affecting users will be
fixed. The code in a LTS branch is stable, frozen for new features.
Notice that although within a LTS release there's a expectation
of stability because the code is frozen, different (major) LTS
releases may include changes in behavior, API incompatibilities
and new features. The development team will make a considerable
effort to minimize and properly document these changes (changes
when comparing it to the last major LTS release).
Sprint Releases are replaced by LTS releases. I.e., in the cycle
when 52.0 (LTS) is released, that's also the version used as a
Sprint Release (there's no 52.0 -- non LTS -- in this case).
New LTS releases should be done carefully, with ample time for
announcements, testing and documentation. It's recommended
that one or two sprints are dedicated as preparations for a LTS
release, with a Sprint Release serving as a "LTS beta" release.
Similarly, there should be announcements about the end-of-life
(EOL) of a LTS release once it approaches its 18 months of
life.
Misc details
------------
Sprint and LTS releases, when packaged, whenever possible, will be
preferably distributed through different package channels
(repositories).
This is possible for repository types such as YUM/DNF repos[2]. In
such cases, users can disable the regular channel, and enable the LTS
version. A request for the installation of Avocado packages will
fetch the latest version available in the enabled repository. If the
LTS repository channel is enabled, the packages will receive minor
updates (bugfixes only), until a new LTS version is released (roughly
every 12 months).
If the non-LTS channel is enabled, users will receive updates every
3-4 weeks.
On other types of repos such as PyPI[3] which have no concept of
"sub-repos" or "channels", users can request a version smaller than
the version that succeeds the current LTS to get the latest LTS
(including minor releases). Suppose the current LTS major version is
52, but there have been minor releases 52.1 and 52.2. By running:
$ pip install 'avocado-framework<53.0'
pip provide LTS version 52.2. If 52.3 gets released, they will be
automatically deployed instead. When a new LTS is released, users
would still get the latest minor release from the 52.0 series, unless
they update the version specification.
The existence of LTS releases should never be used as an excuse
to break a Sprint Release or to introduce gratuitous
incompatibilities there. In other words, Sprint Releases should
still be taken seriously, just as they are today.
Timeline example
----------------
Assume each sprint is taking 3 weeks. Notice how multiple LTS
releases can co-exist before EOL.
sprint release 49.0
sprint release 50.0
--> start preparing a LTS release, so 51.0 is a beta LTS
sprint release 51.0
LTS release 52.0 (52lts branch is created, packages go into LTS repo)
sprint release 53.0
--> bug that also affects 52.0 is found, fix gets added to
master and 52lts branches
sprint release 54.0 + LTS 52.1
--> bug that also affects 52.0 is found, fix gets added to
master and 52lts branches
sprint release 55.0
--> critical bug that affects 52.1 *only* is found, fix gets
added to 52lts and 52.2 LTS is immediately released
sprint release 56.0
sprint release 57.0
sprint release 58.0
sprint release 59.0
--> EOL for 36lts (18 months since the release of 36.0lts)
...
avocado-vt
----------
avocado-vt is an Avocado plugin that allows "VT tests" to be run
inside Avocado. It's a third-party project maintained mostly by
Engineers from Red Hat QE with assistance from the Avocado team
and other community members.
It's a general consensus that QE teams use avocado-vt directly
from git, usually following the master branch, which they
control.
There's no official maintenance or stability statement for
avocado-vt. Even though the upstream community is quite
friendly and open to both contributions and bug reports,
avocado-vt is made available without any promises for
compatibility or supportability.
When packaged and versioned, avocado-vt rpms should be considered
just snapshots, available in packaged form as a convenience to
users outside of the avocado-vt development community. Again,
they are made available without any promises of compatibility or
stability.
* Which Avocado version should be used by avocado-vt?
This is up to the avocado-vt community to decide, but the
current consensus is that to guarantee some stability in
production environments, avocado-vt should stick to a specific
LTS release of Avocado. In other words, the Avocado team
recommends production users of avocado-vt not to install Avocado
from its master branch or upgrade it from Sprint Releases.
Given each LTS release will be maintained for 18 months, it
should be reasonable to expect avocado-vt to upgrade to a new
LTS release once a year or so. This process will be done with
support from the Avocado team to avoid disruptions, with proper
coordination via the avocado mailing lists.
In practice the Avocado development team will keep watching
avocado-vt to detect and document incompatibilities, so when
the time comes to do an upgrade in production, it's expected
that it should happen smoothly.
* Will it be possible to use the latest Avocado and avocado-vt
together?
Users are welcome to *try* this combination. The Avocado
development team itself will do it internally as a way to monitor
incompatibilities and regressions.
Whenever Avocado is released, a matching versioned snapshot of
avocado-vt will be made. Packages containing those avocado-vt
snapshots, for convenience only, will be made available in the
regular Avocado repository.
---
References:
[1] -
https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html
[2] -
http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#fedora-from-avocado-s-own-repo
[3] - https://pypi.python.org/pypi
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170419/08776a07/attachment.sig>
More information about the Avocado-devel
mailing list