<div class="gmail_quote">On Thu, Oct 1, 2009 at 7:15 PM, David Malcolm <<a href="mailto:dmalcolm@redhat.com">dmalcolm@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Proposal: Python 3 in Fedora 13 "Evolutionary, not revolutionary": build a python 3 stack parallel-installable with the python 2 stack. = High-level summary = - Python 3.0 was released almost 10 months ago, on 2008-12-03, and the latest release of the 3.* branch is 3.1.1, released on 2009-08-17. - Other distros have python 3, though not necessarily with anything "on top" resembling the full python 2 stack. - We have a working, valuable python 2 stack, which is used by critical system components (yum and anaconda): we must not destabilize the python 2 stack. - Python 3 is sufficiently different from python 2 that we need them to be independent software stacks. - I plan to spend a large chunk of my $DAYJOB over the next few months trying to build a useful Python 3 stack for Fedora 13, for some definition of "useful" (help will be appreciated!) - I don't want to add extra work for package maintainers: if you maintain an SRPM of a python 2 module that's working for you, you shouldn't feel obligated to own a separate SRPM for python 3. If someone has a need for the module on python 3, they can take on that work. = Background = Python 3 is intended by upstream to be the future of Python, but we have many critical components that use Python 2. Python 2 and Python 3 are sufficiently different that we need both (try writing "print" in each). Python 2 will be around for a long time. An interesting summary of Python 3 adoption can be seen here: <a href="http://renesd.blogspot.com/2009/09/py3kpython3-more-than-one-year-on-096.html" target="_blank">http://renesd.blogspot.com/2009/09/py3kpython3-more-than-one-year-on-096.html</a> An earlier proposal about python 3 in Fedora is here: <a href="https://www.redhat.com/archives/fedora-devel-list/2008-May/msg02417.html" target="_blank">https://www.redhat.com/archives/fedora-devel-list/2008-May/msg02417.html</a> Going forward, I will have plenty of time to spend, as part of my dayjob, on Python in Fedora [1] = Proposal = I want to get python 3 into Fedora, but I don't want to break yum or anaconda (or anything else, for that matter!) How to do this? I propose that Fedora shall have separate, parallel-installable Python 2 and Python 3 stacks. I believe we can get things to the point where on a Fedora box you'd be able to install both stacks, and have some processes running python 2 code, and some running python 3, simultaneously. Where I would draw the line is on having both python 2 and python 3 running within the same _process_: the two libraries share most of their symbol names, but with differing implementations, and the result of trying to dynamically link the two into the same address space would be highly unstable. As an example, you'd be able to install both mod_python and mod_python3 rpms, but you wouldn't be able to (sanely) configure httpd to have both running simultaneously (I guess we should add a run-time warning for this case) Scoping: - this work would target Fedora 13. I'd avoid pushing it into F12 until it's proven safe to do so - the proposal is for python 2 vs python 3. It could be extended to support having multiple minor-versions of Python as well, but that's a big extension of the work involved and not something I'm planning to work on myself. = Details = We should split python 2 and python 3 at the source RPM level, where possible. The easy case is when upstream release separate tarballs for the python 2 and python 3 versions of code. For example, given package "python-foo" in packaging CVS, there would be a separate "python3-foo" for the python 3 version. There would be no expectation that the two would need to upgrade in lock-step. (The two SRPMS could have different maintainers within Fedora: the packager of a python 2 module might not yet have any interest in python 3) The more difficult case is when the python module is emitted as part of the build of a larger module. Some examples: - the build of "rpm" itself emits an "rpm-python" subpackage. - Another example is the "postgres" srpm, which emits a "postgresql-python" subpackage. In a quick attempt at seeing other examples, on my laptop (F11), here are the packages installed that provide python modules where the package name differs from the srpm name: $ rpm -qf /usr/lib/python2.*/site-packages/* | grep -v "is not owned" | </blockquote><div>SIA, this is off of topic , i am sure. BUT, it is very strange that could be exists, perhaps, some file or directory not owned by someone. Isn't it ? </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> sort | uniq | xargs rpm -q --qf "%{NAME}-%{VERSION}-%{RELEASE} %{SOURCERPM}\n" | sed -e"s/.src.rpm//" | squeal -f table col0, col1 from - where col0 != col1 col0| col1| ----------------------------------------+----------------------------------+ at-spi-python-1.26.0-1.fc11| at-spi-1.26.0-1.fc11| audit-libs-python-1.7.13-1.fc11| audit-1.7.13-1.fc11| cracklib-python-2.8.13-4| cracklib-2.8.13-4| gamin-python-0.1.10-4.fc11| gamin-0.1.10-4.fc11| hplip-libs-3.9.8-12.fc11| hplip-3.9.8-12.fc11| libproxy-python-0.2.3-10.fc11| libproxy-0.2.3-10.fc11| libselinux-python-2.0.80-1.fc11| libselinux-2.0.80-1.fc11| libsemanage-python-2.0.31-4.fc11| libsemanage-2.0.31-4.fc11| libuser-python-0.56.9-3| libuser-0.56.9-3| libxml2-python-2.7.3-3.fc11| libxml2-2.7.3-3.fc11| newt-python-0.52.10-4.fc11| newt-0.52.10-4.fc11| plague-common-0.4.5.7-5.20090612cvs.fc11| plague-0.4.5.7-5.20090612cvs.fc11| policycoreutils-python-2.0.62-12.12.fc11| policycoreutils-2.0.62-12.12.fc11| python-magic-5.03-2.fc11| file-5.03-2.fc11| python-slip-dbus-0.1.15-3.fc11| python-slip-0.1.15-3.fc11| python-slip-gtk-0.1.15-3.fc11| python-slip-0.1.15-3.fc11| rpm-python-4.7.1-1.fc11| rpm-4.7.1-1.fc11| setroubleshoot-server-2.1.14-3.fc11| setroubleshoot-2.1.14-3.fc11| system-config-printer-libs-1.1.8-6.fc11|system-config-printer-1.1.8-6.fc11| Such SRPMS have a: BuildRequires: python-devel which is a subpackage from the python build (2.6) My plan here is that we should create a python3-devel subpackage as a subpackage of the python3 build, and have it parallel-installable with the python-devel package. We could then %prep the rpm build for each of the above so that the python 3 support is built as a parallel component of the build, independently of the python 2 support e.g. by copying the python2 support into a separate dir, then applying a patch as necessary (and somehow wiring up the configuration/make so it builds both...) The caveat here is that I haven't tried actually doing this yet for any of these packages. Issues with this approach: - I don't yet know if autoconfiguration will work well with both -devel packages installed - It will probably involve actually doing the porting work for each package (yay, we get to be leaders!) - Whoever does this for a package needs to work closely with the upstream for that package I'll have a go at doing this for rpm when I get back from vacation. Arguably the existing python 2 binding of rpm isn't great, but it's probably best to go for close compatibility between python 2 and python 3 rather than try to overhaul the bindings as part of the port to python 3: one thing at a time! "Naming convention" proposal: How does this sound: - an rpm with a "python-" prefix means a python 2 rpm, of the "default" python 2 minor version (for Fedora this will be the most recent stable upstream minor release, for EPEL it will be the minor release of 2 that came with the distro, so 2.4 for EPEL5) - an rpm with a "python3-" prefix means a python 3 rpm, of the "default" python 3 minor version (for Fedora this will be the most recent stable upstream release) (we could extend this to have specific minor-releases (e.g. use "python24-" for a python 2.4 stack, in case some brave soul wants to get zope/plone running. But may be better to try to fix the zope/2.6 incompatibility at that point (caveat: haven't looked at the details of the issue). I don't intend to work on such versions)) What about packages without a "python-" prefix? Proposal: If upstream has a naming convention for python2 vs python3, use it. Otherwise, add a "python3-" prefix to make things clear. I'm not sure about the details here. Examples? There have been various discussions upstream about what to call the python 3 binary. My favorite quote on the subject is from Guido, <a href="http://mail.python.org/pipermail/python-3000/2008-March/012421.html" target="_blank">http://mail.python.org/pipermail/python-3000/2008-March/012421.html</a> : > During the next 3 years or so, installing Py3k as the default "python" > will be a deed of utter irresponsibility and is likely to break your > system in subtle ways (both OSX and Linux these days use Python for > certain system tasks). If you *really* want to shoot yourself in the > foot this way, go ahead and explicitly use "make altinstall > bininstall" or link it yourself. I propose that for Fedora we have "/usr/bin/python3" for the system/default version of python 3 and "/usr/bin/python" for the system/default version of python 2. Both would be symlinks to a binary with the minor-release embedded in the name ("/usr/bin/python3.1" and "/usr/bin/python/2.6"). As I understand things, this should make us broadly in agreement with upstream; see e.g.: <a href="http://mail.python.org/pipermail/python-dev/2009-April/088862.html" target="_blank">http://mail.python.org/pipermail/python-dev/2009-April/088862.html</a> <a href="http://mail.python.org/pipermail/python-dev/2009-April/088884.html" target="_blank">http://mail.python.org/pipermail/python-dev/2009-April/088884.html</a> A rough plan for Fedora 13 might be: - get python3 packaged in a manner compatible with the above - (persuade /usr/lib/rpm/brp-python-bytecompile to use the correct python when building rpms containing .py files) - get rpm bindings working with python3 - get some useful components working e.g. a web stack: Django, TurboGears etc (though e.g. Django's py3k support is a long way off IIRC); ideas? - solidify packaging guidelines for python 2 vs python 3 once we've got some experience with the above and hopefully proven the techniques - look at porting major components over to python3 (but probably don't actually do this for F13; leave python 2 as the critical component, I suspect): - yum (rpm) - anaconda However "no plan survives contact with the enemy", we'll see how things turn out in reality when trying to get a full integrated stack working. Future work (F14?) could involve cutting over the major components, so that the base install would bring in "python3", and "python" would become optional. Obviously there's a _lot_ to be done before that can be done sanely. = Progress so far = I've put together a somewhat-working python3 srpm, based on the python srpm (with lots of FIXMEs added...) It's not yet ready for a formal package review (I'm working through the various patches, and it's not yet fully installable in parallel with the python 2 stack), but you can see the specfile here: <a href="http://people.redhat.com/dmalcolm/python3/python3.spec" target="_blank">http://people.redhat.com/dmalcolm/python3/python3.spec</a> and an SRPM here: <a href="http://people.redhat.com/dmalcolm/python3/python3-3.1.1-1.fc13.src.rpm" target="_blank">http://people.redhat.com/dmalcolm/python3/python3-3.1.1-1.fc13.src.rpm</a> After I did this work, I saw that a couple of other people have written Python 3 srpms for Fedora: <a href="http://ivazquez.fedorapeople.org/packages/python3000/" target="_blank">http://ivazquez.fedorapeople.org/packages/python3000/</a> and <a href="https://bugzilla.redhat.com/show_bug.cgi?id=526126" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=526126</a> and there was this proposal for doing Python 3 in F10: <a href="https://www.redhat.com/archives/fedora-devel-list/2008-May/msg02417.html" target="_blank">https://www.redhat.com/archives/fedora-devel-list/2008-May/msg02417.html</a> similar to this one. Obviously I want to work with those people to come up with a working python 3 rpm in Fedora. There's also the merge-review for python: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=226342" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=226342</a> ; many of the comments there would thus also apply to my srpm, given that I used the former as my starting point. On a tangent: we currently have 2.6.2 in F12/rawhide. Upstream is preparing a 2.6.3 release, with many bugfixes. This seems to me like a candidate for F13. (the release notes for the 2.6.3 rc are long; I'd want it to have a _lot_ of testing before pushing it to F12)) Thoughts? Dave [1] I'm being transferred at work, and will be able to spend a lot of time on Python within Fedora. Previously, my job involved keeping various internal RH systems working (with any Fedora work done afterhours or as a side project). Having said that, I'm about to go on vacation with no access to a computer for the period October 3rd-10th -- fedora-devel-list mailing list <a href="mailto:fedora-devel-list@redhat.com">fedora-devel-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/fedora-devel-list" target="_blank">https://www.redhat.com/mailman/listinfo/fedora-devel-list</a> </blockquote></div>