[Fedora-packaging] Proposal: support multiple Python minor versions
Toshio Kuratomi
a.badger at gmail.com
Fri Nov 20 21:50:42 UTC 2009
On Fri, Nov 20, 2009 at 12:37:00PM -0500, David Malcolm wrote:
> Proposal: support multiple Python minor version, as one instance of a
> generalized way of supporting variant stacks.
>
> Currently we support a single version of Python; we pick a specific
> minor-release of Python (currently "2.6", in fact 2.6.4 in rawhide), and
> all modules are assumed to be for that "2.MINOR" version of Python.
>
> We plan to extend this further in Fedora 13, with the idea that there's
> additionally a specific minor-release of Python 3 (currently "3.1", in
> fact 3.1.1), and all python3-foo rpms are assumed to be for that
> "3.MINOR" version of Python [1]
>
>
> Some people want more: they want additional minor versions: for example,
> when we want to upgrade "python" to mean "python 2.7", we will rebuild
> all our RPMs for 2.7, but some users may have code written to 2.6 that
> they don't want to bother porting.
>
> This makes more sense in an EPEL context: in EPEL5 we have the system
> python as "2.4", but many people want a "2.6" stack, others want "2.3",
> 2.5", "3.1", etc etc.
>
> There are ways of doing this within the current Fedora/EPEL packaging
> framework, but it would lead to an explosion of specfiles and CVS
> directories. I dislike this: I think of specfiles as human-readable
> source code for driving a reproducable build; I hate having many
> almost-identical copies of a source file.
>
> I had an idea for how this could be done whilst minimizing "specfile
> explosion"; it allows for the generation of parallel families of RPMs
> from one specfile, moving the complexity from the individual packager
> into the build system, and onto release-engineering (sorry Jesse!) - and
> eating up more disk space, build time, etc.
>
> Caveat: I haven't looked into the details of whether Koji, package CVS,
> and Makfile.common can actually cope with some of the ideas here, and
> I've not tested the specifics of the rpm macros; the proposal is
> independent of but compatible with the Python3 feature (I think).
>
> PROPOSAL:
> - Introduce the following new packaging macros:
> - "pyprefix":
> - defined as "python" for the system build of python
> - defined as "pythonMN" for a build of python-M.N e.g. "python26"
> for "python-2.6"
> - for use in the names of packages
> - "pysuffix":
> - defined as the empty string for the system build of python
> - defined as "-M.N" for builds of python-M.N e.g "-32" for
> "python-3.2"
> - for use in /usr/bin when giving alternative versions of tools e.g.
> "/usr/bin/pydoc-3.2" to be the python-3.2 version of "pydoc"
>
> - Within python-foo module specfiles and the python module packaging
> guidelines:
>
> - Add default definitions of the macros:
> %{!?pyprefix: %global pyprefix python}
> %{!?pysuffix: %global pysuffix}
>
> - Change:
> Name: python-foo
> to:
> Name: %{pyprefix}-foo
>
> and similarly for all dependencies, so that e.g.:
> BuildRequires: python-devel python-setuptools
> becomes:
> BuildRequires: %{pyprefix}-devel %{pyprefix}-setuptools
>
> - In %install, move scripts if necessary (to allow
> parallel-installability), adding:
> for script in foo bar baz ; do
> mv %{buildroot}/usr/bin/$script mv
> %{buildroot}/usr/bin/$script-%{pysuffix}
> done
>
> This gives us a single specfile, which can be used with different
> definitions of these macros to give multiple builds, leading to multiple
> SRPMs (the package changes name for the alternative builds).
>
> Where do these macros gets expanded out?
>
> Say in Koji's "dist-f14" we have python = python2.7 and python3 =
> python3.2, and we want to support alternative python stacks.
>
> Then for the dist-f14 build tags, introduce multiple new tags:
> dist-f14-altbuild-python23
> dist-f14-altbuild-python24
> dist-f14-altbuild-python25
> dist-f14-altbuild-python26
> dist-f14-altbuild-python30
> dist-f14-altbuild-python31
> Each of these "altbuild" tags inherit from the main dist-f14 build tag,
> adding a single RPM: "rpm-altbuild-macros-pythonMN-0.1-1.fc14" e.g.
> "rpm-altbuild-macros-python26-0.1-1.fc14" which contains a single file:
> /etc/rpm/altbuild-python26.macros
> (changing the version as appropriate), defining the macros:
> %global pyprefix python26
> %global pysuffix -2.6
> %global __python /usr/bin/python2.6
> (need to ensure this last one actually gets used; need to move the
> define from rpm to python-devel)
>
> These Koji tags are in addition to the regular one ("dist-f14"), in
> which "python" and "python3" have their default meanings.
>
> All of these tags would deposit their built packages into dist-f14
>
> Similarly, for EPEL5, extend "dist-5E-epel-testing-candidate",
> introducing multiple child tags:
> dist-5E-epel-testing-candidate-altbuild-python23
> dist-5E-epel-testing-candidate-altbuild-python25
> dist-5E-epel-testing-candidate-altbuild-python26
> dist-5E-epel-testing-candidate-altbuild-python27
> dist-5E-epel-testing-candidate-altbuild-python30
> dist-5E-epel-testing-candidate-altbuild-python31
> all of which would deposit their built packages into
> dist-5E-epel-testing-candidate
>
> For packaging CVS, add a new, optional file to a dist CVS subdirectory,
> "build-variability". So for package "python-foo",
> "python-foo/EL-5/build-variability" might contain these lines:
> dist-5E-epel-testing-candidate-altbuild-python23
> dist-5E-epel-testing-candidate # 2.4 is the default in EPEL5
> dist-5E-epel-testing-candidate-altbuild-python25
> dist-5E-epel-testing-candidate-altbuild-python26
> dist-5E-epel-testing-candidate-altbuild-python27
> dist-5E-epel-testing-candidate-altbuild-python30
> dist-5E-epel-testing-candidate-altbuild-python31
>
> If "build-variability" is present, it would change the meaning of "make
> build": instead of triggering a job that does a single build into
> "dist-5E-epel-testing-candidate", it would create a job that builds into
> all of the build tags listed in the file. If any of these builds fail,
> the whole build fails, and no RPMs reach he destination tag.
>
> If a particular module only works with some minor-releases (perhaps it
> doesn't work with python 3 yet, or needs python>=2.4 etc), then the file
> would be edited appropriately.
>
> I have no great love for "build-variability" as a name.
>
> Caveat: I'm handwaving here; I haven't looked into the details of what
> it would take on the Koji side and to Makefile.common
>
> We'd be going from two dimensions of variability:
> - build architecture (i386, x86_64 etc; currently expressed internally
> in Koji)
> - OS release (devel, F12, EPEL5 etc)
> where we have a CVS directory per OS-release (one dimension)
>
> up to three dimensions of variability:
> - build architecture (i386, x86_64 etc)
> - OS release (devel, F12, EPEL5 etc)
> - Python minor release (2.3, 2.4, 2.5, 2.6, etc)
> whilst keeping just one CVS directory per OS-release; the package
> maintainer only has to edit a single specfile, and can attempt to ensure
> that it builds across all minor-releases he/she cares about;
> release-engineering would have a fair bit of extra complexity to deal
> with though.
>
> It may be necessary to add extra macros to the altbuild macro files so
> that specfiles can somehow easily conditionalize the build for altbuild
> variants matching some criteria (e.g. on altbuild for 2.6 and above,
> need this extra Patch3")
>
> I've tried to structure this so it can be reused for other language
> runtimes where we might want to support multiple versions; perhaps the
> Perl SIG would want a similar family of "altbuild" macros (I'm not a
> perl person so maybe I'm missing something here).
>
> The proposal only exposes a single extra dimension of variability;
> higher dimensionality seems to lead to combinatorial explosion (e.g
> sqlalchemy versions).
>
> It also assumes (perhaps overoptimistically) than these other language
> runtimes are independent, so that we never have collisions where e.g.
> something wants to consume both multiple python stacks and multiple perl
> stacks. This assumption fails for the case of e.g. multiple parallel
> versions of say postgres or httpd, etc where you might also want
> multiple versions of python, and multiple mod_pythons M * N
> combinatorial explosion [2]
>
>
> Thoughts? Total insanity?
>
Yes, insanity! :-)
But my objection is more along the implementation route taken rather than
the general idea of multiple versions of python2.
Right now, anyone can take an srpm, satisfy the BuildRequirements and a
static list of build dependencies listed in a buildsys package and generate
the builds. This proposal encodes some requirement information into the
buildsystem so that you can't just build the package on your own system.
Something that keeps all of the information in the SRPM is better.
Also, this scheme would leave us forcing maintainers to, some extent care
about multiple python versions. Even if, you only work with python3 and I
only work with python2, if there's a bug preventing builds on one python
version then the other person is partially blocked while that's being
figured out. If one python version is requiring rebuilds to fix runtime
bugs, users of the package on all python versions receive new packages
because the build triggers package rebuilding for all versions.
keeping binary rpms and srpms might be more difficult in this plan too. If
python24-foo was built from python-foo-1.0-1 but python-foo-1.0-2 only
builds python25-foo, do we reap the python24-foo and python-foo-1.0-1 SRPM
from the repositories or do we leave them?
I think if we do go for multiple python interpreter versions, having
separate srpms is going to provide a better experience for managing the
packages and for the end user.
-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/fedora-packaging/attachments/20091120/4b9c8f55/attachment.sig>
More information about the Fedora-packaging
mailing list