<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at 2:26 PM Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at 1:37 PM Maarten Beeckmans <<a href="mailto:maartenb@inuits.eu" target="_blank">maartenb@inuits.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p> </p>
<div>
<div>On 2020-09-29 18:22, Dennis Kliban
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at
10:30 AM Maarten Beeckmans <<a href="mailto:maartenb@inuits.eu" target="_blank">maartenb@inuits.eu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>We are trying to automate our Pulp3 setup the
same way as our Pulp2 setup.<br>
<br>
We have the following use cases in our Pulp2 setup:<br>
<br>
a) We create mirrors and repositories automatically
for every environment (using puppet and the API).
The repositories are defined in a yaml configuration
file (Puppet Hiera).<br>
- We have several problems for doing this in
Pulp3. It's very complex to automatically create
mirrors (ex. centos-8-appstream) for the development
environment (that are in sync with the upstream
repositories). A possible solution for this problem
would be to add a 'latest' flag to publications and
distributions so that they are always pointing to
the latest version of the repository.<br>
- It is possible to create the repositories with
scripts, but that's specific and quite hacky. It's
also not possible to automatically update the
publications and distributions this way. We would
need to keep a local copy of what distribution maps
to what repository, but that's not a good idea. Why
keep a local copy of the mapping if that's present
in pulp.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>We are already planning to improve this workflow. A
single API call will create a task that will sync a repo,
create a new publication, and update the distribution. We
are first going to introduce the change in pulp_file and
then other plugins will follow. <a href="https://pulp.plan.io/issues/7469" target="_blank">https://pulp.plan.io/issues/7469</a></div>
</div>
</div>
</blockquote>
This issue seems to be solving my issue.<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>b) We upload packages from one repository (that
is not served to clients) to another repository
based on lists in yaml (cherry picking), or straight
from a (jenkins) Pipeline. This is done by the
pulp-admin commands.<br>
- cherry picking could be done by copying content
from one repository to another using the copy API
call, but we want to automatically expose those
changes to the development environment (promotion
can be done as in c).<br>
- Uploading content isn't an issue because this is
possible by converting the pulp-admin commands to
api calls.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>You want to be able to upload a package and have that
operation create a new publication and update the
distribution. This would be similar to the story that I
linked above for syncing - except for the upload use
case. Am I understanding correctly?<br>
</div>
</div>
</div>
</blockquote>
Yes, that's what we want. If we create an rpm itself, we want
that's it directly available in the (development) repository so we
don't have to create a new publication and update the
distribution. For adding it to production, we would promote the
repository as described in c. Same when copying packages between
repositories.<br></div></div></blockquote><div><br></div><div>I definitely see the utility in making the content available to the users right away. Please file a story at <a href="https://pulp.plan.io/issues/new/" target="_blank">https://pulp.plan.io/issues/new/</a> so we have this use captured. This use case should be considered also when implementing #7469. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>c) We want to be able to 'promote' a set of
repositories from one environment to another. We are
using a script that is generated by puppet when
creating the repositories in a).<br>
For 'promoting' a set of repositories to one
environment to another environment, this is
explained <a href="https://docs.pulpproject.org/pulpcore/workflows/promotion.html" target="_blank">https://docs.pulpproject.org/pulpcore/workflows/promotion.html</a>.<br>
<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I don't understand what challenge you are describing in
C.<br>
</div>
</div>
</div>
</blockquote>
Now we have different repositories for development and production.
We want to be able to easily promote repositories from development
to production (with archiving, the production repository keeps X
older versions of packages), this may be done with one command in
the cli interface or different commands that can be scripted. This
is somewhat explained in the docs, but it's very high level. Maybe
it's a good idea to add an example in the rpm docs (with the
promoting of an CentOS repository)?<br></div></div></blockquote><div><br></div><div>I agree that we need more docs around this. Can you please file an issue about this also?<br></div><div><br></div><div>There are two ways to accomplish this. <br></div><div><br></div><div>1) Create a distribution for each environment. In this case whenever you are ready to promote a publication from the Dev distribution to the Production distribution you just simply look at which publication is currently assigned to Dev and assign it to Production. <br></div><div><br></div><div>When you want to roll back to a previous publication you need to query for a publication that is associated with a specific repository version (or publication date of creation) that you are interested in. Then you need to update the distribution with that publication. <br></div><div><br></div><div>This workflow gets tricky when you need to release a hotfix to production and skip all the other environments. Since the repository version history is probably much further along than Production is right now, you need to find the repository version that is associated with the current publication that is associated with the Production distribution. Then you need to use the 'modify' API[0] with the Production repository version as the base_version and add/remove any packages that you need for Production. Then you need to create a new publication and associate it with the Production distribution. Then you will need to create another repository version using the previous repository version as the base_version. This will get you back to where you were before in your Development environment. <br></div><div><br></div><div>2) Create a repository and distribution for each environment. Each time you want to promote a repository to the Production environment you need to use the repository 'modify' API[0] to create a new Production repository version. The base_version would be the Dev repository version which you want to promote. You would then create a publication for that new Production repository version and associate that publication with the Production distribution. <br></div><div><br></div><div>The rollback process is similar to the above, but the advantage of this approach is that you have separate repository version history for each environment. When you need to skip environments and go straight to Production, you can simply add/remove packages to the latest Production repository version without having to figure out which version your Production repository is at. <br></div><div><br></div><div><br></div></div></div></div></div></blockquote><div><br></div><div>I just read through our docs and now I am not sure if my explanation is any better than what we already have there. Please reply with any questions about the workflow. I am hoping this dialogue will help us improve the docs.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div></div><div></div><div>[0] <a href="https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify" target="_blank">https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>Thanks for all the work<br>
<br>
Maarten</div>
</div>
</div>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote>
</div>
</div>
</blockquote>
<pre cols="72">--
(o- Maarten Beeckmans
//\ Open Source Consultant
V_/_ Inuits - <a href="https://www.inuits.eu" target="_blank">https://www.inuits.eu</a></pre>
</div>
</div>
_______________________________________________<br>
Pulp-list mailing list<br>
<a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></div></div></div>
</blockquote></div></div>