[Pulp-list] Associating content units between repos using different download policies

[Simon Baatz]

Hi Dennis, Justin,

On Mon, Dec 18, 2017 at 04:37:23PM -0500, Dennis Kliban wrote:
>    On Mon, Dec 18, 2017 at 2:49 PM, Justin Sherrill
>    <[1]jsherril at redhat.com> wrote:
> 
>    On 12/18/2017 02:31 PM, Dennis Kliban wrote:
> 
>    The use case you are describing is not one that Pulp 2 currently
>    supports. However, I can picture this feature working in 2 different
>    ways:
>    1) A publisher for a repository with an 'immediate' download policy on
>    an importer should make sure that all the content it is publishing is
>    actually downloaded.
>    2) A copy of content from an 'on_demand' repo to an 'immediate' repo
>    should schedule the copied units to be downloaded.
> 
>    Could you issue a 'download repo' task to one of the secondary repos to
>    force those units to download?
>    pulp-admin repo download
>    Note that this is *different* than 'pulp-admin rpm repo sync'
>    It wouldn't be automatic, but could be scheduled.
> 
>    Justin, thank you for reminding me of this feature. The REST API is
>    documented here[0]. Make this request against your 'destination'
>    repository and this will spawn a task to download all content units
>    that have not been downloaded yet in that repository.

Thank you for pointing this out!  I did not know 'pulp-admin repo'
and somehow overlooked the download API call when I searched for a
way to trigger downloading in the documentation.

It is not automatic, but doing an additional API request is no
problem at all for us.  I did a quick test and it seems to do what I
was looking for: after the call, all necessary content units are
available locally and publishing works including the rsync
distributors.


- Simon


>    [0]
>    [2]https://docs.pulpproject.org/dev-guide/integration/rest-api/repo/syn
>    c.html#download-a-repository
>    -Dennis
> 
>    -Justin
> 
>    What do others think? Is there another way to solve this problem?
>    -Dennis
> 
>    On Mon, Dec 18, 2017 at 8:09 AM, Simon Baatz <[3]gmbnomis at gmail.com>
>    wrote:
> 
>      We are trying to optimize one of our use cases using Pulp's
>      'on_demand' download policy, but it does not work as expected (I
>      reckon that's because the deferred download feature was designed for
>      a
>      different use case).
>      We have one or even multiple large 'source' RPM repos from which we
>      associate a much smaller number of RPMs into a dedicated
>      'destination'
>      repo. We then publish (locally and using rsync distributors) the
>      'destination' repo, which is used for installation on our servers.
>      Up to now, the source repos sync from their feeds using 'immediate'
>      as
>      download policy. As we only need a small subset of the RPMs, we
>      tested
>      to set the download policy to 'on_demand' on the source repos in
>      order
>      to download the needed RPMs only. We kept 'immediate' for the
>      'destination' repo we are associating RPMs into.
>      However, associating an RPM into that repo does not trigger a
>      download
>      (neither immediately nor by the 'deferred_download' task).
>      Publishing
>      the 'destination' repo works and content can be downloaded even
>      though
>      the download policy is set to 'immediate' (i.e. squid/pulp_streamer
>      will deliver the content unit as long as it is not present). The
>      rsync
>      publisher fails however, because most of the symlinks are pointing
>      to non-existent destinations.
>      As a workaround we tried to use another repo whose feed points to
>      the
>      'destination' repo and sync it. This will cause Pulp to download and
>      store the needed RPMs.
>      Basically, I have two questions:
>      - Is this how association between repos with different download
>        policies is supposed to work? (I would have expected this to not
>        work at all or to trigger downloading)
>      - If so, is there a more direct way to ensure or trigger the
>      download of
>        all RPMs in our destination repo?
>      Btw., we tested this on Pulp 2.14.3.
>      - Simon
>      _______________________________________________
>      Pulp-list mailing list
>      [4]Pulp-list at redhat.com
>      [5]https://www.redhat.com/mailman/listinfo/pulp-list
> 
> _______________________________________________
> Pulp-list mailing list
> [6]Pulp-list at redhat.com
> [7]https://www.redhat.com/mailman/listinfo/pulp-list
> 
>      _______________________________________________
>      Pulp-list mailing list
>      [8]Pulp-list at redhat.com
>      [9]https://www.redhat.com/mailman/listinfo/pulp-list
> 
> References
> 
>    1. mailto:jsherril at redhat.com
>    2. https://docs.pulpproject.org/dev-guide/integration/rest-api/repo/sync.html#download-a-repository
>    3. mailto:gmbnomis at gmail.com
>    4. mailto:Pulp-list at redhat.com
>    5. https://www.redhat.com/mailman/listinfo/pulp-list
>    6. mailto:Pulp-list at redhat.com
>    7. https://www.redhat.com/mailman/listinfo/pulp-list
>    8. mailto:Pulp-list at redhat.com
>    9. https://www.redhat.com/mailman/listinfo/pulp-list

> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list