[Pulp-dev] Deferred downloading (Lazy) catalog management in pulp3

[Jeff Ortel]

BACKGROUND:

We have learned a lot about deferred (lazy) download catalog management in pulp2.  Currently, this is only
supported by the RPM plugin importers.  The importer flow made adding support cumbersome. The importer(s) will
generate catalog entries *only* when the download-policy is not "immediate" as follows:

- For each unit not in the repository that it would have downloaded.
- For each unit already associated.

The reason for #2 is to support cases where the download policy has changed from "immediate" to one of the
deferred policies.  After switching policies, the user has to do a sync to generate catalog entries.  The
downside is the importer regenerates all of the entries when only a few are necessary.

PROPOSAL:

In pulp3, deferred download catalog management is provided by the proposed ChangeSet.  With this change in
tooling, I want to propose a change in the strategy for managing catalog entries.  I propose the ChangeSet
add/remove entries for content regardless of download-policy.

This has the following advantages:

- Changing download policies would not require a sync to generate catalog entries.

- Better supports a use case we are starting to hear about.  "As a user, I want to reclaim disk space by
switch to deferred download policy and then delete stored content."  Deleting stored content would be an
entirely separate story.

- The catalog could be managed by deltas but always be complete.  The overhead of managing the catalog is
proportional to the number of units being added/removed from the repository.  That is, on initial sync, all
the catalog entries are added.  Subsequent syncs will only add/remove entries for content being added/removed
to the repository.

- The downside is that users not using deferred downloading would incur the overhead of managing the catalog
but we probably need benchmarks using postgres to evaluate this.


Thoughts?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170314/99d0c682/attachment.sig>