[Fedora-packaging] Proposal: support multiple Python minor versions

[David Malcolm]

Proposal: support multiple Python minor version, as one instance of a
generalized way of supporting variant stacks.

Currently we support a single version of Python; we pick a specific
minor-release of Python (currently "2.6", in fact 2.6.4 in rawhide), and
all modules are assumed to be for that "2.MINOR" version of Python.

We plan to extend this further in Fedora 13, with the idea that there's
additionally a specific minor-release of Python 3 (currently "3.1", in
fact 3.1.1), and all python3-foo rpms are assumed to be for that
"3.MINOR" version of Python [1]


Some people want more: they want additional minor versions: for example,
when we want to upgrade "python" to mean "python 2.7", we will rebuild
all our RPMs for 2.7, but some users may have code written to 2.6 that
they don't want to bother porting.

This makes more sense in an EPEL context: in EPEL5 we have the system
python as "2.4", but many people want a "2.6" stack, others want "2.3",
2.5", "3.1", etc etc.

There are ways of doing this within the current Fedora/EPEL packaging
framework, but it would lead to an explosion of specfiles and CVS
directories. I dislike this: I think of specfiles as human-readable
source code for driving a reproducable build; I hate having many
almost-identical copies of a source file.

I had an idea for how this could be done whilst minimizing "specfile
explosion"; it allows for the generation of parallel families of RPMs
from one specfile, moving the complexity from the individual packager
into the build system, and onto release-engineering (sorry Jesse!) - and
eating up more disk space, build time, etc. 

Caveat: I haven't looked into the details of whether Koji, package CVS,
and Makfile.common can actually cope with some of the ideas here, and
I've not tested the specifics of the rpm macros;  the proposal is
independent of but compatible with the Python3 feature (I think).

PROPOSAL:
- Introduce the following new packaging macros:
  - "pyprefix":
    - defined as "python" for the system build of python
    - defined as "pythonMN" for a build of python-M.N e.g. "python26"
for "python-2.6"
    - for use in the names of packages
  - "pysuffix":
    - defined as the empty string for the system build of python
    - defined as "-M.N" for builds of python-M.N e.g "-32" for
"python-3.2"
    - for use in /usr/bin when giving alternative versions of tools e.g.
"/usr/bin/pydoc-3.2" to be the python-3.2 version of "pydoc"

- Within python-foo module specfiles and the python module packaging
guidelines:

  - Add default definitions of the macros:
      %{!?pyprefix: %global pyprefix python}
      %{!?pysuffix: %global pysuffix}

  - Change:
      Name:  python-foo
    to:
      Name:  %{pyprefix}-foo

    and similarly for all dependencies, so that e.g.:
      BuildRequires:  python-devel python-setuptools
    becomes:
      BuildRequires:  %{pyprefix}-devel %{pyprefix}-setuptools

  - In %install, move scripts if necessary (to allow
parallel-installability), adding:
      for script in foo bar baz ; do
         mv %{buildroot}/usr/bin/$script mv
%{buildroot}/usr/bin/$script-%{pysuffix}
      done

This gives us a single specfile, which can be used with different
definitions of these macros to give multiple builds, leading to multiple
SRPMs (the package changes name for the alternative builds).

Where do these macros gets expanded out?

Say in Koji's "dist-f14" we have python = python2.7  and python3 =
python3.2, and we want to support alternative python stacks.

Then for the dist-f14 build tags, introduce multiple new tags:
  dist-f14-altbuild-python23
  dist-f14-altbuild-python24
  dist-f14-altbuild-python25
  dist-f14-altbuild-python26
  dist-f14-altbuild-python30
  dist-f14-altbuild-python31
Each of these "altbuild" tags inherit from the main dist-f14 build tag,
adding a single RPM: "rpm-altbuild-macros-pythonMN-0.1-1.fc14" e.g.
"rpm-altbuild-macros-python26-0.1-1.fc14" which contains a single file:
    /etc/rpm/altbuild-python26.macros 
(changing the version as appropriate), defining the macros:
  %global pyprefix python26
  %global pysuffix -2.6
  %global __python /usr/bin/python2.6
(need to ensure this last one actually gets used; need to move the
define from rpm to python-devel)

These Koji tags are in addition to the regular one ("dist-f14"), in
which "python" and "python3" have their default meanings.

All of these tags would deposit their built packages into dist-f14

Similarly, for EPEL5, extend "dist-5E-epel-testing-candidate",
introducing multiple child tags:
  dist-5E-epel-testing-candidate-altbuild-python23
  dist-5E-epel-testing-candidate-altbuild-python25
  dist-5E-epel-testing-candidate-altbuild-python26
  dist-5E-epel-testing-candidate-altbuild-python27
  dist-5E-epel-testing-candidate-altbuild-python30
  dist-5E-epel-testing-candidate-altbuild-python31
all of which would deposit their built packages into
dist-5E-epel-testing-candidate

For packaging CVS, add a new, optional file to a dist CVS subdirectory,
"build-variability".  So for package "python-foo",
"python-foo/EL-5/build-variability" might contain these lines:
  dist-5E-epel-testing-candidate-altbuild-python23
  dist-5E-epel-testing-candidate # 2.4 is the default in EPEL5
  dist-5E-epel-testing-candidate-altbuild-python25
  dist-5E-epel-testing-candidate-altbuild-python26
  dist-5E-epel-testing-candidate-altbuild-python27
  dist-5E-epel-testing-candidate-altbuild-python30
  dist-5E-epel-testing-candidate-altbuild-python31

If "build-variability" is present, it would change the meaning of "make
build": instead of triggering a job that does a single build into
"dist-5E-epel-testing-candidate", it would create a job that builds into
all of the build tags listed in the file.  If any of these builds fail,
the whole build fails, and no RPMs reach he destination tag.

If a particular module only works with some minor-releases (perhaps it
doesn't work with python 3 yet, or needs python>=2.4 etc), then the file
would be edited appropriately.

I have no great love for "build-variability" as a name.

Caveat: I'm handwaving here; I haven't looked into the details of what
it would take on the Koji side and to Makefile.common

We'd be going from two dimensions of variability:
  - build architecture (i386, x86_64 etc; currently expressed internally
in Koji)
  - OS release (devel, F12, EPEL5 etc)
where we have a CVS directory per OS-release (one dimension)

up to three dimensions of variability:
  - build architecture (i386, x86_64 etc)
  - OS release (devel, F12, EPEL5 etc)
  - Python minor release (2.3, 2.4, 2.5, 2.6, etc)
whilst keeping just one CVS directory per OS-release; the package
maintainer only has to edit a single specfile, and can attempt to ensure
that it builds across all minor-releases he/she cares about;
release-engineering would have a fair bit of extra complexity to deal
with though.

It may be necessary to add extra macros to the altbuild macro files so
that specfiles can somehow easily conditionalize the build for altbuild
variants matching some criteria (e.g. on altbuild for 2.6 and above,
need this extra Patch3")

I've tried to structure this so it can be reused for other language
runtimes where we might want to support multiple versions; perhaps the
Perl SIG would want a similar family of "altbuild" macros (I'm not a
perl person so maybe I'm missing something here).

The proposal only exposes a single extra dimension of variability;
higher dimensionality seems to lead to combinatorial explosion (e.g
sqlalchemy versions).  

It also assumes (perhaps overoptimistically) than these other language
runtimes are independent, so that we never have collisions where e.g.
something wants to consume both multiple python stacks and multiple perl
stacks.  This assumption fails for the case of e.g. multiple parallel
versions of say postgres or httpd, etc where you might also want
multiple versions of python, and multiple mod_pythons M * N
combinatorial explosion [2]


Thoughts?  Total insanity?

Dave

[1] https://fedoraproject.org/wiki/Features/Python3F13
[2] so for e.g. python-psycopg2 you could unroll a 2-dimensional
variability table into the file in dist-CVS, and into Koji tags,
expressing all combinations of the postgres libs and of the python minor
version; similarly for shipping multiple httpd versions versus python
versions for mod_python.  This many-dimensional approach quickly gets
too complex to deal with, I think.