<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 2020-09-29 21:44, Dennis Kliban
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAPmNiuoEPymSd_LS2+GQWqf02U4CjaSVySuzj5GCwtJ7QqMV7g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at 2:26
PM Dennis Kliban <<a href="mailto:dkliban@redhat.com"
target="_blank" moz-do-not-send="true">dkliban@redhat.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 29,
2020 at 1:37 PM Maarten Beeckmans <<a
href="mailto:maartenb@inuits.eu" target="_blank"
moz-do-not-send="true">maartenb@inuits.eu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p> </p>
<div>
<div>On 2020-09-29 18:22, Dennis Kliban wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Tue, Sep 29, 2020 at 10:30 AM Maarten
Beeckmans <<a
href="mailto:maartenb@inuits.eu"
target="_blank"
moz-do-not-send="true">maartenb@inuits.eu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div
style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>We are trying to automate our
Pulp3 setup the same way as our
Pulp2 setup.<br>
<br>
We have the following use cases
in our Pulp2 setup:<br>
<br>
a) We create mirrors and
repositories automatically for
every environment (using puppet
and the API). The repositories
are defined in a yaml
configuration file (Puppet
Hiera).<br>
- We have several problems for
doing this in Pulp3. It's very
complex to automatically create
mirrors (ex. centos-8-appstream)
for the development environment
(that are in sync with the
upstream repositories). A
possible solution for this
problem would be to add a
'latest' flag to publications
and distributions so that they
are always pointing to the
latest version of the
repository.<br>
- It is possible to create the
repositories with scripts, but
that's specific and quite hacky.
It's also not possible to
automatically update the
publications and distributions
this way. We would need to keep
a local copy of what
distribution maps to what
repository, but that's not a
good idea. Why keep a local copy
of the mapping if that's present
in pulp.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>We are already planning to improve
this workflow. A single API call will
create a task that will sync a repo,
create a new publication, and update
the distribution. We are first going
to introduce the change in pulp_file
and then other plugins will follow. <a
href="https://pulp.plan.io/issues/7469" target="_blank"
moz-do-not-send="true">https://pulp.plan.io/issues/7469</a></div>
</div>
</div>
</blockquote>
This issue seems to be solving my issue.<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div
style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>b) We upload packages from
one repository (that is not
served to clients) to another
repository based on lists in
yaml (cherry picking), or
straight from a (jenkins)
Pipeline. This is done by the
pulp-admin commands.<br>
- cherry picking could be done
by copying content from one
repository to another using the
copy API call, but we want to
automatically expose those
changes to the development
environment (promotion can be
done as in c).<br>
- Uploading content isn't an
issue because this is possible
by converting the pulp-admin
commands to api calls.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>You want to be able to upload a
package and have that operation create
a new publication and update the
distribution. This would be similar to
the story that I linked above for
syncing - except for the upload use
case. Am I understanding correctly?<br>
</div>
</div>
</div>
</blockquote>
Yes, that's what we want. If we create an rpm
itself, we want that's it directly available
in the (development) repository so we don't
have to create a new publication and update
the distribution. For adding it to production,
we would promote the repository as described
in c. Same when copying packages between
repositories.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I definitely see the utility in making the
content available to the users right away. Please
file a story at <a
href="https://pulp.plan.io/issues/new/"
target="_blank" moz-do-not-send="true">https://pulp.plan.io/issues/new/</a>
so we have this use captured. This use case should
be considered also when implementing #7469. <br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p>I've described this problem in the following issue:
<a class="moz-txt-link-freetext" href="https://pulp.plan.io/issues/7626">https://pulp.plan.io/issues/7626</a></p>
<blockquote type="cite"
cite="mid:CAPmNiuoEPymSd_LS2+GQWqf02U4CjaSVySuzj5GCwtJ7QqMV7g@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div
style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>c) We want to be able to
'promote' a set of repositories
from one environment to another.
We are using a script that is
generated by puppet when
creating the repositories in a).<br>
For 'promoting' a set of
repositories to one environment
to another environment, this is
explained <a
href="https://docs.pulpproject.org/pulpcore/workflows/promotion.html"
target="_blank"
moz-do-not-send="true">https://docs.pulpproject.org/pulpcore/workflows/promotion.html</a>.<br>
<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I don't understand what challenge
you are describing in C.<br>
</div>
</div>
</div>
</blockquote>
Now we have different repositories for
development and production. We want to be able
to easily promote repositories from
development to production (with archiving, the
production repository keeps X older versions
of packages), this may be done with one
command in the cli interface or different
commands that can be scripted. This is
somewhat explained in the docs, but it's very
high level. Maybe it's a good idea to add an
example in the rpm docs (with the promoting of
an CentOS repository)?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I agree that we need more docs around this. Can
you please file an issue about this also?<br>
</div>
<div><br>
</div>
<div>There are two ways to accomplish this. <br>
</div>
<div><br>
</div>
<div>1) Create a distribution for each environment.
In this case whenever you are ready to promote a
publication from the Dev distribution to the
Production distribution you just simply look at
which publication is currently assigned to Dev and
assign it to Production. <br>
</div>
<div><br>
</div>
<div>When you want to roll back to a previous
publication you need to query for a publication
that is associated with a specific repository
version (or publication date of creation) that you
are interested in. Then you need to update the
distribution with that publication. <br>
</div>
<div><br>
</div>
<div>This workflow gets tricky when you need to
release a hotfix to production and skip all the
other environments. Since the repository version
history is probably much further along than
Production is right now, you need to find the
repository version that is associated with the
current publication that is associated with the
Production distribution. Then you need to use the
'modify' API[0] with the Production repository
version as the base_version and add/remove any
packages that you need for Production. Then you
need to create a new publication and associate it
with the Production distribution. Then you will
need to create another repository version using
the previous repository version as the
base_version. This will get you back to where you
were before in your Development environment. <br>
</div>
<div><br>
</div>
<div>2) Create a repository and distribution for
each environment. Each time you want to promote a
repository to the Production environment you need
to use the repository 'modify' API[0] to create a
new Production repository version. The
base_version would be the Dev repository version
which you want to promote. You would then create a
publication for that new Production repository
version and associate that publication with the
Production distribution. <br>
</div>
<div><br>
</div>
<div>The rollback process is similar to the above,
but the advantage of this approach is that you
have separate repository version history for each
environment. When you need to skip environments
and go straight to Production, you can simply
add/remove packages to the latest Production
repository version without having to figure out
which version your Production repository is at. <br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I just read through our docs and now I am not sure if my
explanation is any better than what we already have there.
Please reply with any questions about the workflow. I am
hoping this dialogue will help us improve the docs.<br>
</div>
</div>
</div>
</blockquote>
<p>Your explanation here is much more clear to me than how it's
described in the docs. I think it would be a better idea to add
the advantages and disadvantages to the documentation and maybe
add some examples as in the other workflows.<br>
</p>
<blockquote type="cite"
cite="mid:CAPmNiuoEPymSd_LS2+GQWqf02U4CjaSVySuzj5GCwtJ7QqMV7g@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div>[0] <a
href="https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify"
target="_blank" moz-do-not-send="true">https://pulp-rpm.readthedocs.io/en/latest/restapi.html#operation/repositories_rpm_rpm_modify</a></div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div
style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div>Thanks for all the work<br>
<br>
Maarten</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>