[scl.org] rh-python36 SPEC file question (circular dependency)

Joshua Kramer joskra42.list at gmail.com
Sun Jun 10 17:44:41 UTC 2018 

With the packages installed, I did a strace while running RPM to see
what I could find.  Here is an excerpt of the results:

open("/etc/rpm/macros.python3.rh-python36", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1879, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x76efb000
read(3, "# define these macros in a way w"..., 4096) = 1879
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x76efa000
read(4, "# Locale name alias data base.\n#"..., 4096) = 2502
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x76efa000, 4096)                = 0
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/rpm.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/rpm.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/rpm.mo", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/rpm.mo", O_RDONLY) = -1 ENOENT
(No such file or directory)
write(2, "error: ", 7error: )                  = 7
write(2, "Macro % has illegal name (%defin"..., 35Macro % has illegal
name (%define)

It appears that once RPM reads the line "# define these amcros in a
way..." it runs into something it doesn't like, then it attempts to
generate an error message using the correct locale and LC_MESSAGES.
Failing that it spits out the default message.

I went to the offending file (macros.python3.rh-python36) and removed
the comment, with the same result.

Where should I look next?


On Sat, Jun 9, 2018 at 2:33 PM, Joshua Kramer <joskra42.list at gmail.com> wrote:
> I had actually attempted to do just that, and ran into another
> interesting problem.  When I install rh-python36-python-devel package,
> it breaks the RPM database in some way.  With that package installed,
> every time I do anything with RPM, I get several instances of this
> error message:  "error: Macro % has illegal name (%define)"
>
> This error does not interfere with the ability of RPM to install or
> remove packages.  However, as part of the rpmbuild process,
> dependencies of RPM packages are checked.  When this occurs, some
> program spits out several instances of that error message, and the
> program returns with an error.  RPMBUILD then aborts the build when it
> detects the error.
>
> I know the error is a bash shell error, but I have not yet determined
> what in the -devel package is broken in such a way to cause that error
> to occur whenever RPM is run.
>
> On Sat, Jun 9, 2018 at 12:03 PM, Charalampos Stratakis
> <cstratak at redhat.com> wrote:
>>
>>
>> ----- Original Message -----
>>> From: "Joshua Kramer" <joskra42.list at gmail.com>
>>> To: "Charalampos Stratakis" <cstratak at redhat.com>, sclorg at redhat.com
>>> Sent: Saturday, June 9, 2018 4:47:43 PM
>>> Subject: Re: [scl.org] rh-python36 SPEC file question (circular dependency)
>>>
>>> > Long story short:
>>> > 1. Build python with rehweel set to 0.
>>> .....
>>>
>>> I got further with these steps, but there are some omissions.  I
>>> attempted to build python.spec as the first step (after setting
>>> with_rewheel to 0), but it complained about a missing dependency of
>>> rh-python36.  So I then built the rh-python36 metapackage first, and I
>>> had to install that as root.  The first pass of python.spec then built
>>> successfully.
>>>
>>> I then modified python-setuptools.spec to disable build_wheel, and
>>> attempted to build.  It failed because it requires rh-python36-devel.
>>> So I attempt to install that package as root, and there are additional
>>> dependency issues:
>>>         /opt/rh/rh-python36/root/usr/bin/python3.6m is needed by
>>> rh-python36-python-devel-3.6.3-3.el7.centos.armv7hl
>>>         libpython3.6m.so.rh-python36-1.0 is needed by
>>> rh-python36-python-devel-3.6.3-3.el7.centos.armv7hl
>>>         python(abi) = 3.6 is needed by
>>> rh-python36-python-devel-3.6.3-3.el7.centos.armv7hl
>>>         rh-python36-python(armv7hnl-32) = 3.6.3-3.el7.centos is needed
>>> by rh-python36-python-devel-3.6.3-3.el7.centos.armv7hl
>>>         rh-python36-python-libs(armv7hnl-32) = 3.6.3-3.el7.centos is
>>> needed by rh-python36-python-devel-3.6.3-3.el7.centos.armv7hl
>>>
>>> I've attempted various combinations of force-installing the packages I
>>> have, with various other issues.  So what is the correct next step in
>>> getting the setuptools package to build?
>>>
>>>
>>
>> Without something which contains a dependency solver you will have to manually install the packages at your system.
>>
>> At this point it seems that you will have to install the python-devel, python and python-libs.
>>
>> You can check the dependencies of each rpm by invoking 'rpm -qp --requires <rpm>', or by examining the %package
>> directives at the SPEC file of python. Although without doing it in some sort of build system (e.g. centos/fedora builds system),
>> you will have to manually inspect what the dependencies are each time.
>>
>> Please let me know if that worked for you.
>>
>>>
>>>
>>> On Fri, Jun 8, 2018 at 12:10 PM, Charalampos Stratakis
>>> <cstratak at redhat.com> wrote:
>>> >
>>> >
>>> > ----- Original Message -----
>>> >> From: "Joshua Kramer" <joskra42.list at gmail.com>
>>> >> To: "Neal Gompa" <ngompa13 at gmail.com>
>>> >> Cc: sclorg at redhat.com
>>> >> Sent: Thursday, June 7, 2018 2:22:56 PM
>>> >> Subject: Re: [scl.org] rh-python36 SPEC file question (circular
>>> >> dependency)
>>> >>
>>> >> Thanks, Neal!  That got me a little farther.  Per the comments at the
>>> >> top of they python.spec file, I set with_rewheel to 0, then I built
>>> >> the Python RPM.  That seems to have worked.
>>> >>
>>> >> Now, when I attempt to build setuptools, using this command:
>>> >> rpmbuild -ba ./python-setuptools.spec --define 'scl rh-python36' >
>>> >> python-setuptools-build.log 2>&1
>>> >>
>>> >> ...the build command fails with these errors:
>>> >> error: Failed build dependencies:
>>> >>         rh-python36-python-devel is needed by
>>> >> rh-python36-python-setuptools-36.5.0-1.el7.centos.noarch
>>> >>         rh-python36-python-pip is needed by
>>> >> rh-python36-python-setuptools-36.5.0-1.el7.centos.noarch
>>> >>         rh-python36-python-wheel is needed by
>>> >> rh-python36-python-setuptools-36.5.0-1.el7.centos.noarch
>>> >>
>>> >> Attempting to build the python-pip package results in similar errors.
>>> >>
>>> >> I have the following packages installed:
>>> >> rh-python36-2.0-1.el7.src.rpm
>>> >> rh-python36-python-3.6.3-3.el7.src.rpm
>>> >> rh-python36-python-pip-9.0.1-2.el7.src.rpm
>>> >> rh-python36-python-psycopg2-2.7.3-1.el7.src.rpm
>>> >> rh-python36-python-setuptools-36.5.0-1.el7.src.rpm
>>> >> rh-python36-python-virtualenv-15.1.0-2.el7.src.rpm
>>> >> rh-python36-python-wheel-0.30.0a0-1.el7.src.rpm
>>> >> rh-python36-scipy-0.19.1-2.el7.src.rpm
>>> >>
>>> >> What should I look for next?
>>> >>
>>> >> Thanks!
>>> >>
>>> >
>>> > Hello. I am one of the maintainers for that collection so feel free to ping
>>> > me
>>> > directly, either on IRC or mail.
>>> >
>>> > Sorry for the complications you are facing, bootstrapping the interpreter
>>> > to an rpm
>>> > is indeed a quite complex procedure and it could be documented better.
>>> >
>>> > I will try to outline the steps with some explanations.
>>> >
>>> > Firstly, python bundles pip and setuptools, so ideally we'd like python to
>>> > utilize the system
>>> > installed packages instead of the bundled ones. What happens is that we
>>> > first compile python
>>> > including the bundled wheels (that's when we set the 'rewheel' macro to 0).
>>> >
>>> > Then we'd like to build pip and setuptools in the wheel [0] format so they
>>> > can be utilized by python.
>>> > However we still do not have the 'wheel' package built in order to do that,
>>> > so we'll have to (byte)compile
>>> > them the usual way, then compile wheel, and then build again python and
>>> > setuptools as wheels.
>>> >
>>> > So you should set the build_wheel macro in python-pip's and
>>> > python-setuptools' SPEC file to 0, build them and then
>>> > build the python-wheel package.
>>> >
>>> > Then after you have built python-wheel, rebuild python-pip and
>>> > python-setuptools with the build_wheel macro macro set to 1,
>>> > as we can create the wheel format now since we have the wheel package on
>>> > our system.
>>> >
>>> > Finally rebuild python with rewheel set to 1 in order for python to strip
>>> > the bundled wheels and utilize the system installed
>>> > ones (with a little bit of magic [1].
>>> >
>>> > Long story short:
>>> > 1. Build python with rehweel set to 0.
>>> > 2. Build python-setuptools and python-pip with build_wheel set to 0.
>>> > 3. Build python-wheel.
>>> > 4. Build python-setuptools and python-pip with build_wheel set to 1.
>>> > 5. Build python with rehweel set to 1.
>>> >
>>> > I hope that helps (and I hope I didn't forget any steps :) ).
>>> >
>>> > [0] https://pythonwheels.com/
>>> > [1]
>>> > https://github.com/fedora-python/rewheel/blob/master/rewheel/__init__.py
>>> >
>>> >> On Mon, Jun 4, 2018 at 10:23 PM, Neal Gompa <ngompa13 at gmail.com> wrote:
>>> >> > On Mon, Jun 4, 2018 at 9:38 PM Joshua Kramer <joskra42.list at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hello,
>>> >> >>
>>> >> >> I am trying to compile rh-python36 to use with CentOS on a Raspberry
>>> >> >> Pi.
>>> >> >>
>>> >> >> On line 192 of the SPEC file are these lines:
>>> >> >> %if 0%{?with_rewheel}
>>> >> >> BuildRequires: %{?scl_prefix}python-setuptools
>>> >> >> BuildRequires: %{?scl_prefix}python-pip
>>> >> >> %endif
>>> >> >>
>>> >> >> What is this supposed to do?  It prevents the package from building,
>>> >> >> because there is a circular dependency- if I try to build python36, it
>>> >> >> complains that python36-setuptools and python36-pip are not
>>> >> >> installed... but I can't install those without having Python installed
>>> >> >> first.  If I comment those lines out I get no error and the build
>>> >> >> almost completes.  (It then fails the "multiprocessing" test, but
>>> >> >> that's beside the point.)  However, I don't want to break something by
>>> >> >> commenting them out... so is there a better or "right" way to get rid
>>> >> >> of the circular dependency?
>>> >> >>
>>> >> >
>>> >> > Python is a two-cycle build. You build the first time with rewheel
>>> >> > disabled (that removes those BRs), then after that's successfully
>>> >> > built, you use those artifacts to build again with rewheel enabled.
>>> >> >
>>> >> >
>>> >> > --
>>> >> > 真実はいつも一つ！/ Always, there's only one truth!
>>> >>
>>> >> _______________________________________________
>>> >> SCLorg mailing list
>>> >> SCLorg at redhat.com
>>> >> https://www.redhat.com/mailman/listinfo/sclorg
>>> >>
>>> >
>>> > --
>>> > Regards,
>>> >
>>> > Charalampos Stratakis
>>> > Software Engineer
>>> > Python Maintenance Team, Red Hat
>>>
>>
>> --
>> Regards,
>>
>> Charalampos Stratakis
>> Software Engineer
>> Python Maintenance Team, Red Hat

More information about the SCLorg mailing list