comps discussion at fudcon and the future

[Florian Festi]

Hi!

I fully agree that this current package handling situation is less then
perfect. We have all been overrun by the amazing success and growth of
Fedora. A lot of mechanisms that have been put into place years ago do no
longer fit - the comps file being one.

seth vidal wrote:
> Problems we see:
> - browsing packages is a difficult problem when you have 10000+ pkgs.
> - comps has a lot of meanings and it is not obvious what they all are/do
> - conditional pkgs (which provide a way of saying "install pkgX only 
>   if pkgY is installed) are several colors of doom b/c they aren't a 
>   dependency relationship and creates clutter on systems.
> - users expect groups to be more persistent on their systems and to act
>   more like pkgs (ie: yum update should update groups, too)
> - optional/default/mandatory pkgs are confusing and not useful to most
>   people - the types are only useful when browsing, not when installing.

While I agree with the list above I believe it only is a small part of the
problem and the suggested solution does not even cover them completely.

But there are other problems, challenges and developments that came up
within the packaging domain:

Handling of language dependent content:
This is related to the "conditionals" mentioned above. With the increasing
number of languages supported and packages being properly translated we ship
more and more language dependent content the users are not interested in. We
are currently missing both a way to package these contents properly and a
mechanism the control which should be actually installed.

Multilib:
The situation has changed a lot since we introduced multilib. The challenge
who is going to support 64bit processors first has been decided long ago and
the age of 32bit processors is ending. There are several annoying features
of the solution that was chosen at the time (overwriting 32 bit binaries,
install lots of files twice) and we should get rid of them at some point in
the (not too near) future. But this may require some other changes first...

Packaging noarch content:
We have just started
http://fedoraproject.org/wiki/Features/NoarchSubpackages and hope it will
make its way into F11. A long with all it's benefits I also expect it to
increase the number of packages and to change the ration of arch dependent
and noarch packages a lot.

Doing things per packages gets too expensive:
Even a small amount of work or possibility of error gets expensive and
annoying if it has to be done in a lot of packages. The example we are
looking at right now are scripts that are handling special types of files
like: ldconfig, install-info, gtk-update-icon-cache, ... We hope to come up
with a solution soon.
Another idea under that topic is the (semi) automatic creation of sub
packages. This could be used to address several other problems like multilib
or language files. But right now it is only a vague idea...

Poor implementations:
Several part of our tool chain are poorly implemented and though do not
scale linearly. Some parts of RPM already got fixed (I hope the code can hit
rawhide soon) but I am quite sure there are enough places left to keep us
busy (e.g. transferring the repo data of a updates repo takes quadratic
(O(#packages*#repodate_updated) bandwidth over its life time). The larger
the number will get the smaller the visible sins will get.

Ratio between installed and available packages:
We are dealing with an increasing size of meta data that is not actually
used (by the system dealing with them). This is not yet a problem but will
require deep architectural changes if it gets one.


I believe this list (including the issues brought up in the original post)
will require deep changes to the way we package Fedora during the next few
releases. Changes to the comps file should better fit into our overall plan.



Nevertheless I'd like to comment the suggested solution:

Some kind of conditionals will need to stay. As soon as we have a better
solution we can easily drop them but I cannot see yet when this is going to
happen. The solution will surely be part of yum and might even require
changes to rpm (No soft requires discussion now and here!).

The details of the implementation feel very fishy. I am not sure if this
change is really going to ease the pain or make it harder.

I think the discussion of the implementation details misses the point: Who
is going to define and apply the keywords for the packages and who is going
to define the groups in the new comps file. Comps was introduced to take
control, responsibility and burden of the decision what should be installed
and offered to the user by default away from the single packagers (with the
additional benefit of being able to make changes without rebuilding the
packages). And this was a "good idea"(tm). I now have doubt that a central
instance will be capable to keep track of the 15000 or soon may be even
50000 packages. Comming up with a set of keywords and applying them to all
packages will be even much more work and source of much more trouble that
the current comps file. Especially as (more) domain specific knowledge will 
be needed.
So IMHO we need a mechanism that still keeps the single packager out of the
process but allow much more people and groups to participate in it. I could
image that SIGs or may be even Spins get their own comps/groups/keyword
files to deal with a given application domain. May be users could even
decide what comps file to install on their system. So everybody could
package their view of the Fedora world. I am still very undecided how this 
should look like but I don't think we are ready to discuss the 
implementation details yet.

Florian