compat-python, Zope...

[Hans de Goede]

Panu Matilainen wrote:
> On Wed, 15 Oct 2008, Hans de Goede wrote:
> 
>> James Antill wrote:
>>> On Wed, 2008-10-15 at 09:08 +0200, Oliver Falk wrote:
>>>> James Antill wrote:
>>>> [ ... ]
>>>>>> I can think of a Python 2.4 package that lives within the Zope 
>>>>>> tree to make it extra hard for others to use it by accident - but 
>>>>>> I don't think that this would be neat, seen from a FHS point of view.
>>>>>  In some ways this might be doable, at least it has less pain points
>>>>> than packaging it "properly".
>>>> I'm not sure what you mean here? Do you mean with properly an 
>>>> /usr/%{_lib}/python%{major}.{minor}/ installation? Well, I'd like to 
>>>> invite everybody to have a look at the livna packages. Those are 
>>>> fine and don't hurt the main python...
>>>
>>> % repoquery --repoid=livna --provides compat-python24-imaging 
>>> _imaging.so()(64bit)
>>> _imagingft.so()(64bit)
>>> _imagingmath.so()(64bit)
>>> compat-python24-imaging = 1.1.6-1.lvn9
>>> % repoquery --provides python-imaging               _imaging.so()(64bit)
>>> _imagingft.so()(64bit)
>>> _imagingmath.so()(64bit)
>>
>> Yes rpmbuild is very broken in that it generates provides for .so 
>> files which are not under libdir, so plugins for applications / libs / 
>> languages can have conflicting Provides, luckily nothing should ever 
>> Require these plugins.
> 
> If something requires / can require them, how can they be "wrong"?

Its all about namespaces, normal libraries all share a common namespace, so all 
sonames (or filenames when the lib lacks a soname) in the standard library 
path, or in additional library paths added through /etc/ld.so.conf.d must be 
unique as they share one namespace.

Plugins otoh which should be and usually are installed outside of the regular 
library path, do not have to have a unique soname (or filename) and often 
don't, basicly every applications which uses plugings has its own plugin 
namespace usually under %{_libdir}/%{name}. Having these plugins have provides 
like foo_plugin(soname) is fine having them provide just soname is wrong, as 
these provides provide stuff in a *completely different namespace*

> Not that I disagree

Glad you agree :) So if we agree that doing soname provides for things like 
plugins is wrong as it polutes the library provide namespace with non library 
provides. Lets start looking at a solution.

> The above are silly and completely useless for all 
> I can tell. But how is rpmbuild going to know what's a real library and 
> what's not? %{_libdir} vs outside it is not going to work

I disagree, as your numbers and explanation below shows using a having 
DT_SONAME or not check does not work. Only doing soname provides generation for 
so files under /%{_lib} and %{_libdir} clearly is the right thing to do. Sure 
some stuff will break, and I understand that you are afraid of that, but those 
things are exceptions, lets first get the generic case right, which clearly is 
to only do soname provides for files directly under /%{_lib} and %{_libdir} and 
then start worrying about exceptions.

 > as it'd cause
> perfectly legal uses of /etc/ld.so.conf.d

/etc/ld.so.conf.d is special and most uses of it can be handled automatically 
by rpmbuild. In most (all?) cases where /etc/ld.so.conf.d gets used the 
libraries installed in the path listed in /etc/ld.so.conf.d/foo, are part of 
the same (sub)package as /etc/ld.so.conf.d/foo, so we can patch (hack) the 
rpmbuild dependency generator(s both intern and extern) to see if there are 
files under /etc/ld.so.conf.d in the subpackage and if there are get the paths 
from them and add those to the path to search for soname provides.

Thinking about this more, there might be exceptions here, so we may not even 
want to do this hack, as we will need a mechanism for special cases anyway, so 
I would like to suggest to solve all special cases like /etc/ld.so.conf.d using 
the following:

1) Add a soname_provides_search_path %define, which defaults to:
    "/%{_lib}:%{libdir}"
2) Allow spec files to redefine this (to add additional paths to search) and
    make sure we use the specfile defined version if present when generating
    Provides

This mechanism can be used to handle any special cases no matter how exotic, 
while keeping rpmbuild all nice and clean.

We could the do a full install see what is under /etc/ld.so.conf.d and file 
bugs against all packages putting files there to warn them that they should 
start redefining soname_provides_search_path because rpmbuild will no longer 
search the entire world for soname provides.

 > and LD_LIBRARY_PATH (both of
> which are runtime, not buildtime configuration items) to break.

Using LD_LIBRARY_PATH in packages is usually evil, and where it isn't the 
user/developer apparently does not want the soname's in that dir to always be 
part of the global library namespace, in which case the sonames in that dir 
certainly do not belong in the global library Provides namespace.

Again this is al about namespaces (a concept which rpm Provides sorely lack, 
but thats a different discussion).

> The question to me is: is there a way to *realiably* detect dlopen()'ed 
> modules from real libraries? There are a number of possible hints that 
> can be looked at, such as having DT_SONAME or not but AFAICT, it's all 
> just guesswork. Since the elf linker doesn't require DT_SONAME for 
> shared libraries, we'd break the dependency resolution of valid uses if 
> we filtered those out from provides generation.
> 
> As there doesn't seem to be hard rules (feel free to prove me wrong)

There is one hard rule: libraries are to be installed in the default library 
path searched by /lib[64]/ld-linux.so.2

Everything which does not do that (like qt3) is an ugly hack (as qt4 has 
proven), an exception, and one should never optimize for the exceptions.

> we 
> can detect at build-time, then heuristics + manual overrides is the best 
> we can get.
 >
 > One possibility is a build configuration flag that allows
> switching off generation of provides for DSO's without DT_SONAME, flip 
> that on in redhat-rpm-config and rebuild what's necessary. That'd be 
> essentially a one-liner patch to rpm.
> 
> Just to get an idea of what that would "gain": On my F9 + selected bits 
> from rawhide system, the number of DSO's in /%{_lib}, %{_libdir} and 
> their subdirectories is 3385. Roughly one third of them don't have 
> DT_SONAME set. Looking at the remaining ~2000, there's a lot of 
> "unnecessary" stuff left still.
> But then, libtool seems to be putting soname into things built with 
> -module which makes the DT_SONAME test kinda moot already.
> 

The whole quote directly above indicates this is the wrong solution, as its a 
half baked solution at best.

The proper solution is really easy we need to only generate soname provides for 
files in the default library path, and add a mechanism (as proposed above) for 
corner cases to modify this behaviour from the specfile.

Let me repeat: I understand that you are afraid of breaking things, but I don't 
think it will be that bad. As always I'm willing to put my time where my mouth 
is, so if we do this for F11 (and pretty please do) I'm more then willing to 
help fixing up any fall out, hell you may asign any bugs you get filed about 
this change to me and I'll handle them (anything to get this mess fixed up once 
and for all).

Regards,

Hans