[Pulp-list] Pulp3 Automation

[Maarten Beeckmans]

On 2020-09-29 18:22, Dennis Kliban wrote:
> On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans <maartenb at inuits.eu 
> <mailto:maartenb at inuits.eu>> wrote:
>
>     We are trying to automate our Pulp3 setup the same way as our
>     Pulp2 setup.
>
>     We have the following use cases in our Pulp2 setup:
>
>     a) We create mirrors and repositories automatically for every
>     environment (using puppet and the API). The repositories are
>     defined in a yaml configuration file (Puppet Hiera).
>       - We have several problems for doing this in Pulp3. It's very
>     complex to automatically create mirrors (ex. centos-8-appstream)
>     for the development environment (that are in sync with the
>     upstream repositories). A possible solution for this problem would
>     be to add a 'latest' flag to publications and distributions so
>     that they are always pointing to the latest version of the repository.
>       - It is possible to create the repositories with scripts, but
>     that's specific and quite hacky. It's also not possible to
>     automatically update the publications and distributions this way.
>     We would need to keep a local copy of what distribution maps to
>     what repository, but that's not a good idea. Why keep a local copy
>     of the mapping if that's present in pulp.
>
>
> We are already planning to improve this workflow. A single API call 
> will create a task that will sync a repo, create a new publication, 
> and update the distribution. We are first going to introduce the 
> change in pulp_file and then other plugins will follow. 
> https://pulp.plan.io/issues/7469
This issue seems to be solving my issue.
>
>     b) We upload packages from one repository (that is not served to
>     clients) to another repository based on lists in yaml (cherry
>     picking), or straight from a (jenkins) Pipeline. This is done by
>     the pulp-admin commands.
>       - cherry picking could be done by copying content from one
>     repository to another using the copy API call, but we want to
>     automatically expose those changes to the development environment
>     (promotion can be done as in c).
>       - Uploading content isn't an issue because this is possible by
>     converting the pulp-admin commands to api calls.
>
>
> You want to be able to upload a package and have that operation create 
> a new publication and update the distribution. This would be similar 
> to the story that I linked above for syncing - except for the upload 
> use case.  Am I understanding correctly?
Yes, that's what we want. If we create an rpm itself, we want that's it 
directly available in the (development) repository so we don't have to 
create a new publication and update the distribution. For adding it to 
production, we would promote the repository as described in c. Same when 
copying packages between repositories.
>
>     c) We want to be able to 'promote' a set of repositories from one
>     environment to another. We are using a script that is generated by
>     puppet when creating the repositories in a).
>       For 'promoting' a set of repositories to one environment to
>     another environment, this is explained
>     https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
>
>
> I don't understand what challenge you are describing in C.
Now we have different repositories for development and production. We 
want to be able to easily promote repositories from development to 
production (with archiving, the production repository keeps X older 
versions of packages), this may be done with one command in the cli 
interface or different commands that can be scripted. This is somewhat 
explained in the docs, but it's very high level. Maybe it's a good idea 
to add an example in the rpm docs (with the promoting of an CentOS 
repository)?
>
>     Thanks for all the work
>
>     Maarten
>     _______________________________________________
>     Pulp-list mailing list
>     Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>     https://www.redhat.com/mailman/listinfo/pulp-list
>
-- 
(o-    Maarten Beeckmans
//\     Open Source Consultant
V_/_  Inuits -https://www.inuits.eu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200929/e4ced0dd/attachment.htm>