[Pulp-list] Pulp3 Automation

Tue Sep 29 19:44:58 UTC 2020

On Tue, Sep 29, 2020 at 2:26 PM Dennis Kliban <dkliban at redhat.com> wrote:

> On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans <maartenb at inuits.eu>
> wrote:
>
>> On 2020-09-29 18:22, Dennis Kliban wrote:
>>
>> On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans <maartenb at inuits.eu>
>> wrote:
>>
>>> We are trying to automate our Pulp3 setup the same way as our Pulp2
>>> setup.
>>>
>>> We have the following use cases in our Pulp2 setup:
>>>
>>> a) We create mirrors and repositories automatically for every
>>> environment (using puppet and the API). The repositories are defined in a
>>> yaml configuration file (Puppet Hiera).
>>>   - We have several problems for doing this in Pulp3. It's very complex
>>> to automatically create mirrors (ex. centos-8-appstream) for the
>>> development environment (that are in sync with the upstream repositories).
>>> A possible solution for this problem would be to add a 'latest' flag to
>>> publications and distributions so that they are always pointing to the
>>> latest version of the repository.
>>>   - It is possible to create the repositories with scripts, but that's
>>> specific and quite hacky. It's also not possible to automatically update
>>> the publications and distributions this way. We would need to keep a local
>>> copy of what distribution maps to what repository, but that's not a good
>>> idea. Why keep a local copy of the mapping if that's present in pulp.
>>>
>>
>> We are already planning to improve this workflow. A single API call will
>> create a task that will sync a repo, create a new publication, and update
>> the distribution. We are first going to introduce the change in pulp_file
>> and then other plugins will follow. https://pulp.plan.io/issues/7469
>>
>> This issue seems to be solving my issue.
>>
>>
>>
>>> b) We upload packages from one repository (that is not served to
>>> clients) to another repository based on lists in yaml (cherry picking), or
>>> straight from a (jenkins) Pipeline. This is done by the pulp-admin commands.
>>>   - cherry picking could be done by copying content from one repository
>>> to another using the copy API call, but we want to automatically expose
>>> those changes to the development environment (promotion can be done as in
>>> c).
>>>   - Uploading content isn't an issue because this is possible by
>>> converting the pulp-admin commands to api calls.
>>>
>>
>> You want to be able to upload a package and have that operation create a
>> new publication and update the distribution. This would be similar to the
>> story that I linked above for syncing - except for the upload use case.  Am
>> I understanding correctly?
>>
>> Yes, that's what we want. If we create an rpm itself, we want that's it
>> directly available in the (development) repository so we don't have to
>> create a new publication and update the distribution. For adding it to
>> production, we would promote the repository as described in c. Same when
>> copying packages between repositories.
>>
>
> I definitely see the utility in making the content available to the users
> right away. Please file a story at https://pulp.plan.io/issues/new/ so we
> have this use captured. This use case should be considered also when
> implementing #7469.
>
>
>>
>>
>>> c) We want to be able to 'promote' a set of repositories from one
>>> environment to another. We are using a script that is generated by puppet
>>> when creating the repositories in a).
>>>   For 'promoting' a set of repositories to one environment to another
>>> environment, this is explained
>>> https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
>>>
>>>
>> I don't understand what challenge you are describing in C.
>>
>> Now we have different repositories for development and production. We
>> want to be able to easily promote repositories from development to
>> production (with archiving, the production repository keeps X older
>> versions of packages), this may be done with one command in the cli
>> interface or different commands that can be scripted. This is somewhat
>> explained in the docs, but it's very high level. Maybe it's a good idea to
>> add an example in the rpm docs (with the promoting of an CentOS repository)?
>>
>
> I agree that we need more docs around this. Can you please file an issue
> about this also?
>
> There are two ways to accomplish this.
>
> 1) Create a distribution for each environment. In this case whenever you
> are ready to promote a publication from the Dev distribution to the
> Production distribution you just simply look at which publication is
> currently assigned to Dev and assign it to Production.
>
> When you want to roll back to a previous publication you need to query for
> a publication that is associated with a specific repository version (or
> publication date of creation) that you are interested in. Then you need to
> update the distribution with that publication.
>
> This workflow gets tricky when you need to release a hotfix to production
> and skip all the other environments. Since the repository version history
> is probably much further along than Production is right now, you need to
> find the repository version that is associated with the current publication
> that is associated with the Production distribution. Then you need to use
> the 'modify' API[0] with the Production repository version as the
> base_version and add/remove any packages that you need for Production. Then
> you need to create a new publication and associate it with the Production
> distribution. Then you will need to create another repository version using
> the previous repository version as the base_version. This will get you back
> to where you were before in your Development environment.
>
> 2) Create a repository and distribution for each environment. Each time
> you want to promote a repository to the Production environment you need to
> use the repository 'modify' API[0] to create a new Production repository
> version. The base_version would be the Dev repository version which you
> want to promote. You would then create a publication for that new
> Production repository version and associate that publication with the
> Production distribution.
>
> The rollback process is similar to the above, but the advantage of this
> approach is that you have separate repository version history for each
> environment. When you need to skip environments and go straight to
> Production, you can simply add/remove packages to the latest Production
> repository version without having to figure out which version your
> Production repository is at.
>
>
>
I just read through our docs and now I am not sure if my explanation is any
better than what we already have there. Please reply with any questions
about the workflow. I am hoping this dialogue will help us improve the docs.

> [0]
> https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify
>
>
>>
>>
>>> Thanks for all the work
>>>
>>> Maarten
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>> --
>> (o-    Maarten Beeckmans
>> //\     Open Source Consultant
>> V_/_  Inuits - https://www.inuits.eu
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200929/6cd4d730/attachment.htm>