[Pulp-dev] To integrate Fus or not to....

Milan Kovacik mkovacik at redhat.com
Mon Oct 15 07:52:09 UTC 2018


On Fri, Oct 12, 2018 at 7:11 PM Jeff Ortel <jortel at redhat.com> wrote:

>
>
> On 10/12/2018 11:37 AM, Milan Kovacik wrote:
>
>
>
> On Fri, Oct 12, 2018 at 5:17 PM Jeff Ortel <jortel at redhat.com> wrote:
>
>>
>>
>> On 10/12/2018 09:53 AM, Milan Kovacik wrote:
>>
>>
>>
>> On Fri, Oct 12, 2018 at 3:59 PM Jeff Ortel <jortel at redhat.com> wrote:
>>
>>>
>>>
>>> On 10/10/2018 08:59 AM, Milan Kovacik wrote:
>>>
>>> ...that might be the question we should ask ourselves once again when it
>>> comes to recursive copying of units between repositories.
>>>
>>> I'd like to poll folks opinions about the possibilities that we may have
>>> when it comes to integrating third party solvers in Pulp. My yesterday's
>>> chat with the #fedora-modularity folks about us integrating the Fus[1]
>>> solver in order to reuse the Fus algorithm ran into a couple of bumps:
>>>
>>> * it would be laborous to create a programmatic Python API between Fus
>>> and Pulp because we can't directly use the libsolv thingies (pools,
>>> solvables and friends) in such an API because Fus is written utilizing
>>> GObject, which is incompatible with Swig, which in turn is used in libsolv
>>> to expose the python bindings. One would have to either re-wrap libsolv
>>> code in Fus to work with pygobject or submit PRs against libsolv to support
>>> GObject introspection. I dunno the details of either approach (yet) but
>>> from the sad faces on the IRC and the Fus PR[1] it seemed like a lot of
>>> work but it's still an option
>>>
>>> * we still should be able to integrate thru a pipe into Fus, that would
>>> make it possible to dump modular and ursine metadata into Fus to perform
>>> the dependency solving in a separate subprocess. We should probably
>>> re-check the reasons behind our previous decision not to do the same with
>>> DNF[2].
>>>
>>>
>>> How is Integration with Fus via pipe (CLI) easier than with gobject?
>>> Either way, you "can't directly use the libsolv thingies (pools, solvables
>>> and friends)".  Right?  What am I missing?
>>>
>>>
>> Right, a publish-like operation would be required every time, for all
>> repositories involved in the copy to dump the metadata to the pipe(s);
>> sample of this interface is can be found in Pungi:
>> https://pagure.io/pungi/blob/master/f/pungi/wrappers/fus.py the "query"
>> is passed thru command line.
>> I just learnt Fedora will keep modules and their ursine deps in separate
>> repos, so the source repo won't necessarily be closed on dependencies thus
>> multiple source repos would be needed.
>>
>>
>> This be done using the Fus gobject interface as well?
>>
>
> we'd just dump the XML (and YAML) metadata and run: fus --repo
> source1,1,/path/to/pipe1 --repo source2,2,/path/to/pipe2 --repo
> target,system,/path/to/target_pipe  "module(walrus)" "penguin:1-2.3" etc
> then parse the textual output of fus such as:
>
>
> Can't this ^ be done with Fus through gobject as well and instead of
> parsing textual output, inspect the objects returned?
>
>
To avoid having to search for the resolved units (in Mongo), we have to
maintain a mapping[3] between the solver namespace and Pulp to be able to
refer back to the Pulp units.
Optimizing just the parsing wouldn't have much benefit;  we'd best benefit
from an API that would allow Pulp to have a reference to the solver unit
representation.
Currently we do this thru the libsolv.i[4] bindings therefore the Fus
(Py)GObject interface would have to expose a similar (hopefully smaller)
interface.
This still seemed a little bit too much work last time I checked with Fus
folks on #fedora-modularity.
Note please also that the GObject way won't allow for flexibility in the
solving algorithm[5] itself should Pulp require some adjustments in the
future.


[3]
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/pulp_solv.py#L243,#L244
[4] https://github.com/openSUSE/libsolv/blob/master/bindings/solv.i
[5] https://github.com/fedora-modularity/fus/blob/master/fus.c#L955

>
> # ---%>---------
>   - nothing provides policycoreutils-python-utils needed by
> container-selinux-2:2.69-3.git452b90d.module_2040+0e96cf1b.noarch
>
> Problem 1 / 1:
>   - conflicting requests
>   - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides /bin/bash needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides /usr/bin/python3 needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides python3-dateutil needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides dbus needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
> # ---->%----------
> (fus:8524): fus-WARNING **: 15:13:09.350: Can't resolve all solvables
> module:docker:2017.0:20180816194539:3ff668f0.x86_64 at f29
> module:container-tools:2017.0:20180816194450:80bd9113.x86_64 at f29
> *docker-devel-2:1.13.1-61.git9cb56fd.module_2109+7c83ead1.noarch at f29
> *containers-common-0.1.31-14.dev.gitb0b750d.module_2040+0e96cf1b.x86_64 at f29
>
>
>
>
>>
>>>
>>> * we should be able to extend current libsolv solver in Pulp,
>>> reimplementing the algorithm from Fus. This might be as laborous as the
>>> first option. It would probably give us more flexibility as well as more
>>> room for screwing things up but the responsibility would be ours alone.
>>>
>>> Please let me know what option seems more appealing to you; other option
>>> suggestion are welcome  too.
>>>
>>> Cheers,
>>> milan
>>>
>>> [1] https://github.com/fedora-modularity/fus/pull/46
>>> [2] https://pulp.plan.io/issues/3528#note-7
>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing listPulp-dev at redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20181015/b4dfb946/attachment.htm>


More information about the Pulp-dev mailing list