[Pulp-list] Pulp3 Automation
Maarten Beeckmans
maartenb at inuits.eu
Wed Sep 30 06:39:32 UTC 2020
On 2020-09-29 21:44, Dennis Kliban wrote:
> On Tue, Sep 29, 2020 at 2:26 PM Dennis Kliban <dkliban at redhat.com
> <mailto:dkliban at redhat.com>> wrote:
>
> On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans
> <maartenb at inuits.eu <mailto:maartenb at inuits.eu>> wrote:
>
> On 2020-09-29 18:22, Dennis Kliban wrote:
>> On Tue, Sep 29, 2020 at 10:30 AM Maarten Beeckmans
>> <maartenb at inuits.eu <mailto:maartenb at inuits.eu>> wrote:
>>
>> We are trying to automate our Pulp3 setup the same way as
>> our Pulp2 setup.
>>
>> We have the following use cases in our Pulp2 setup:
>>
>> a) We create mirrors and repositories automatically for
>> every environment (using puppet and the API). The
>> repositories are defined in a yaml configuration file
>> (Puppet Hiera).
>> - We have several problems for doing this in Pulp3.
>> It's very complex to automatically create mirrors (ex.
>> centos-8-appstream) for the development environment (that
>> are in sync with the upstream repositories). A possible
>> solution for this problem would be to add a 'latest' flag
>> to publications and distributions so that they are always
>> pointing to the latest version of the repository.
>> - It is possible to create the repositories with
>> scripts, but that's specific and quite hacky. It's also
>> not possible to automatically update the publications and
>> distributions this way. We would need to keep a local
>> copy of what distribution maps to what repository, but
>> that's not a good idea. Why keep a local copy of the
>> mapping if that's present in pulp.
>>
>>
>> We are already planning to improve this workflow. A single
>> API call will create a task that will sync a repo, create a
>> new publication, and update the distribution. We are first
>> going to introduce the change in pulp_file and then other
>> plugins will follow. https://pulp.plan.io/issues/7469
> This issue seems to be solving my issue.
>>
>> b) We upload packages from one repository (that is not
>> served to clients) to another repository based on lists
>> in yaml (cherry picking), or straight from a (jenkins)
>> Pipeline. This is done by the pulp-admin commands.
>> - cherry picking could be done by copying content from
>> one repository to another using the copy API call, but we
>> want to automatically expose those changes to the
>> development environment (promotion can be done as in c).
>> - Uploading content isn't an issue because this is
>> possible by converting the pulp-admin commands to api calls.
>>
>>
>> You want to be able to upload a package and have that
>> operation create a new publication and update the
>> distribution. This would be similar to the story that I
>> linked above for syncing - except for the upload use case.
>> Am I understanding correctly?
> Yes, that's what we want. If we create an rpm itself, we want
> that's it directly available in the (development) repository
> so we don't have to create a new publication and update the
> distribution. For adding it to production, we would promote
> the repository as described in c. Same when copying packages
> between repositories.
>
>
> I definitely see the utility in making the content available to
> the users right away. Please file a story at
> https://pulp.plan.io/issues/new/ so we have this use captured.
> This use case should be considered also when implementing #7469.
>
I've described this problem in the following issue:
https://pulp.plan.io/issues/7626
>> c) We want to be able to 'promote' a set of repositories
>> from one environment to another. We are using a script
>> that is generated by puppet when creating the
>> repositories in a).
>> For 'promoting' a set of repositories to one
>> environment to another environment, this is explained
>> https://docs.pulpproject.org/pulpcore/workflows/promotion.html.
>>
>>
>> I don't understand what challenge you are describing in C.
> Now we have different repositories for development and
> production. We want to be able to easily promote repositories
> from development to production (with archiving, the production
> repository keeps X older versions of packages), this may be
> done with one command in the cli interface or different
> commands that can be scripted. This is somewhat explained in
> the docs, but it's very high level. Maybe it's a good idea to
> add an example in the rpm docs (with the promoting of an
> CentOS repository)?
>
>
> I agree that we need more docs around this. Can you please file an
> issue about this also?
>
> There are two ways to accomplish this.
>
> 1) Create a distribution for each environment. In this case
> whenever you are ready to promote a publication from the Dev
> distribution to the Production distribution you just simply look
> at which publication is currently assigned to Dev and assign it to
> Production.
>
> When you want to roll back to a previous publication you need to
> query for a publication that is associated with a specific
> repository version (or publication date of creation) that you are
> interested in. Then you need to update the distribution with that
> publication.
>
> This workflow gets tricky when you need to release a hotfix to
> production and skip all the other environments. Since the
> repository version history is probably much further along than
> Production is right now, you need to find the repository version
> that is associated with the current publication that is associated
> with the Production distribution. Then you need to use the
> 'modify' API[0] with the Production repository version as the
> base_version and add/remove any packages that you need for
> Production. Then you need to create a new publication and
> associate it with the Production distribution. Then you will need
> to create another repository version using the previous repository
> version as the base_version. This will get you back to where you
> were before in your Development environment.
>
> 2) Create a repository and distribution for each environment. Each
> time you want to promote a repository to the Production
> environment you need to use the repository 'modify' API[0] to
> create a new Production repository version. The base_version would
> be the Dev repository version which you want to promote. You would
> then create a publication for that new Production repository
> version and associate that publication with the Production
> distribution.
>
> The rollback process is similar to the above, but the advantage of
> this approach is that you have separate repository version history
> for each environment. When you need to skip environments and go
> straight to Production, you can simply add/remove packages to the
> latest Production repository version without having to figure out
> which version your Production repository is at.
>
>
>
> I just read through our docs and now I am not sure if my explanation
> is any better than what we already have there. Please reply with any
> questions about the workflow. I am hoping this dialogue will help us
> improve the docs.
Your explanation here is much more clear to me than how it's described
in the docs. I think it would be a better idea to add the advantages and
disadvantages to the documentation and maybe add some examples as in the
other workflows.
> [0]
> https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify
>
>> Thanks for all the work
>>
>> Maarten
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20200930/e0fb1ac9/attachment.htm>
More information about the Pulp-list
mailing list