[Pulp-dev] Fwd: Re: Changesets Challenges
jortel at redhat.com
Tue Apr 10 14:43:54 UTC 2018
On 04/06/2018 09:15 AM, Brian Bouterse wrote:
> Several plugins have started using the Changesets including
> pulp_ansible, pulp_python, pulp_file, and perhaps others. The
> Changesets provide several distinct points of value which are great,
> but there are two challenges I want to bring up. I want to focus only
> on the problem statements first.
> 1. There is redundant "differencing" code in all plugins. The
> Changeset interface requires the plugin writer to determine what units
> need to be added and those to be removed. This requires all plugin
> writers to write the same non-trivial differencing code over and over.
> For example, you can see the same non-trivial differencing code
> present in pulp_ansible
> and pulp_python
> Line-wise, this "differencing" code makes up a large portion (maybe
> 50%) of the sync code itself in each plugin.
Ten lines of trivial set logic hardly seems like a big deal but any
duplication is worth exploring.
> 2. Plugins can't do end-to-end stream processing. The Changesets
> themselves do stream processing, but when you call into
> changeset.apply_and_drain() you have to have fully parsed the metadata
> already. Currently when fetching all metadata from Galaxy,
> pulp_ansible takes about 380 seconds (6+ min). This means that the
> actual Changeset content downloading starts 380 seconds later than it
> could. At the heart of the problem, the fetching+parsing of the
> metadata is not part of the stream processing.
The additions/removals can be any interable (like generator) and by
using ChangeSet.apply() and iterating the returned object, the pluign
can "turn the crank" while downloading and processing the metadata. The
ChangeSet.apply_and_drain() is just a convenience method. I don't see
how this is a limitation of the ChangeSet.
> Do you see the same challenges I do? Are these the right problem
> statements? I think with clear problem statements a solution will be
> easy to see and agree on.
I'm not convinced that these are actual problems/challenges that need to
be addressed in the near term.
> Pulp-dev mailing list
> Pulp-dev at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Pulp-dev