[Pulp-dev] Changesets Challenges

[Milan Kovacik]

Brian,

thanks for the proposal!

On Fri, Apr 6, 2018 at 4:15 PM, Brian Bouterse <bbouters at redhat.com> wrote:
> Several plugins have started using the Changesets including pulp_ansible,
> pulp_python, pulp_file, and perhaps others. The Changesets provide several
> distinct points of value which are great, but there are two challenges I
> want to bring up. I want to focus only on the problem statements first.
>
> 1. There is redundant "differencing" code in all plugins. The Changeset
> interface requires the plugin writer to determine what units need to be
> added and those to be removed. This requires all plugin writers to write the
> same non-trivial differencing code over and over. For example, you can see
> the same non-trivial differencing code present in pulp_ansible, pulp_file,
> and pulp_python. Line-wise, this "differencing" code makes up a large
> portion (maybe 50%) of the sync code itself in each plugin.

+1; DRY is always better

>
> 2. Plugins can't do end-to-end stream processing. The Changesets themselves
> do stream processing, but when you call into changeset.apply_and_drain() you
> have to have fully parsed the metadata already. Currently when fetching all
> metadata from Galaxy, pulp_ansible takes about 380 seconds (6+ min). This
> means that the actual Changeset content downloading starts 380 seconds later
> than it could. At the heart of the problem, the fetching+parsing of the
> metadata is not part of the stream processing.
>
> Do you see the same challenges I do? Are these the right problem statements?
> I think with clear problem statements a solution will be easy to see and
> agree on.

Totally, esp. on these, easier-to-see-the-value-of ones.

Cheers,
milan

>
> Thanks!
> Brian
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>