<div dir="ltr">I am late to the thread, so I apologize if I repeat things that have been discussed already.<div><br></div><div>Is it a meaningful use case to publish an older version of the repo? Once published, do you keep track of which version got published, and how do you decide which version to push next? This seems like a complication to me.</div><div><br></div><div>As a user / content developer, it seems more useful to me to always publish the latest (i.e. don't have an optional version for publishing), but have the ability to copy from a specific version of a repo into another repo (or the same repo, effectively reverting the content of latest).</div><div><br></div><div>So I would shift the discussion away from the REST API (for now), and more into the expected behavior for manipulating content within pulp. The operations I am aware of are: syncing units, importing units, copying/deleting units, and I am seeking clarification on how versioning will work for each.</div><div><br></div><div>Syncing is probably the easiest, because it can handle all the changes internally and create a new version at the end.</div><div><br></div><div>For importing, if you don't want to create unnecessary intermediate versions that are meaningless, I would want the ability to upload more than one unit and associate it to the repo, and then create a version. In other words, a transactional multi-upload.</div><div><br></div><div>For copying, as suggested above, I want to optionally specify the version.</div><div><br></div><div>Deleting by itself is not hard, it does what it needs to do and then creates a version.</div><div><br></div><div>The more complicated use case would be: what if I wanted to change the contents of repoA:</div><div>* add 3 packages from repo1 version 1</div><div>* add 4 packages from repo2 (latest)</div><div>* delete 5 packages</div><div><br></div><div>and at the end have a single version change for repoA.</div><div><br></div><div>Or, for the same repoA:</div><div>* delete all units of type "rpm" and name "glibc"</div><div>* copy unit type "rpm" and name "glibc" from two versions ago</div><div><br></div><div><br></div><div>If you wanted this use case, then you need a new resource type, somewhat similar to a Task, let's call it Transaction. It is tied to the repository it operates on (repoA in the example above), and locks it from further changes until the transaction is committed or aborted. It could be implemented internally as a repository. You start with the current contents of repoA, and you perform whatever operations you need to do (including changing repo metadata). When you "commit" the Transaction, it becomes *the* new version of the repository and unlocks repoA.</div><div><br></div><div>Whether a Version is a full copy of the repo or a delta is an implementation detail. I would argue for full copy, otherwise you run into the inefficiencies of cvs which had to apply patches in reverse order just to get to a version in the past. I would find it more useful to have a repo diff resource (diff version 1 with version 3, or repo1 version 1 with repo2 latest).</div><div><br></div><div>Unfortunately, it is a rather large paradigm shift, and not one that you can push in a 3.0 -> 3.1 transition. Parts of it will need to land in 3.0 proper, determining what can be left out is an exercise to the reader who managed to keep up with my long emails.</div><div><br></div><div>Hey, a man can dream.</div><div><br></div><div>Mihai</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 30, 2017 at 11:00 AM, David Davis <span dir="ltr"><<a href="mailto:daviddavis@redhat.com" target="_blank">daviddavis@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Brian,<div><br></div><div>The issue is not adding new paths in the API to support versioned repositories. The issue is having to remove or change paths which we can’t do if we want to adhere to semantic versioning. See the example from my email about /repositorycontents/. Also, see @mhrivnak’s writeup here:</div><div><br></div><div><a href="https://pulp.plan.io/projects/pulp/wiki/Repository_Versions#REST-API" target="_blank">https://pulp.plan.io/projects/<wbr>pulp/wiki/Repository_Versions#<wbr>REST-API</a><br></div><div><br></div><div>I think we can probably make tweaks to our API to support the adding of versioned repos in 3.1+ but we’re not set up right now to do that. It also might be worth considering how much time it’ll take to make our APIs compatible with versioned repos in 3.1+ versus just adopting @mhrivnak’s versioned repo changes.</div></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-6618908000343350467gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br></div><div>David<br></div></div></div></div></div></div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">On Thu, Nov 30, 2017 at 10:32 AM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I agree if we couldn't add versioned repos later, then we should add it now. I believe the current API is already well setup to add it later. Specifically, here is a non-nested example of the versioned repository resource to be added in 3.1+<br></div><div><br></div><div>/api/v3/repo_version/</div><div><br></div><div>The user would POST to that to cause a new repo version to be created. What are the contents of that repo version? Well it depends on the POST data. That repo version could be the result of a sync being performed, or perhaps it could also include mass associate/unassociate operations right there. In other words, I see a clear path to adding such a resource, and I see endless flexibility in terms of the users intention (via POST data) of what should be contained in that repo version.<br></div><div><br></div><div>Can some feedback be given on this design? Why won't this work in 3.1+?</div><div><br></div><div>To give my own answer to the question. The main impact I see when adding this in 3.1+ is on plugin writers not users. Specifically the plugin writer API would change because a plugin writer would no longer be associating content with a repo as it's primary activity. Instead their code would be to associate content with a repo version. We can roll out that change in a clear, coordinated way with the plugin community. It's also appropriate because the plugin API is still < 1.0. We would bump the plugin api from 0.1 to 0.2 and plugins can easily declare their compatibility as pulpcore-plugin < 0.2. Relative to the effort of creating a plugin, porting plugin code for a change like this I think would be low effort.<br></div><div><br></div></div><div class="m_-6618908000343350467HOEnZb"><div class="m_-6618908000343350467h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 29, 2017 at 10:19 AM, Jeff Ortel <span dir="ltr"><<a href="mailto:jortel@redhat.com" target="_blank">jortel@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 11/29/2017 07:22 AM, David Davis wrote:<br>
> I think we could design an API in 3.0 that would support versioned repos in 3.1+. However, our current API<br>
> does not. For example, the /repositorycontents/ endpoint doesn't make sense with versioned repos as no one<br>
> would want to add/remove content units one-by-one when doing so would generate a new repo version each time.<br>
> Imagine that we end up with an endpoint in 3.0 that’s not compatible with versioned repos. What would we do? I<br>
> think this is a strong argument for adding versioned repos now.<br>
<br>
</span>agreed.<br>
<span><br>
><br>
> Of course the main drawback is that it might delay the beta. But I wonder by how much. It might be good to<br>
> groom the versioned repo user stories so that (a) we can see how much value they provide to end users and (b)<br>
> how closely they align with the work @mhrivnak has done.<br>
<br>
</span>agreed.<br>
<br>
><br>
><br>
> David<br>
<span>><br>
> On Tue, Nov 28, 2017 at 4:00 PM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a> <mailto:<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>>> wrote:<br>
><br>
> In reading back over the last email thread in May, it ended with us looking at URL options to ensure we<br>
> could release 3.0 and add in repo versions in 3.1+. We definitely want repo versions in the 3.y line, so<br>
> we wanted to make sure that was possible. If it wasn't, then we may have to add it into 3.0.<br>
><br>
> That question is a lot easier now given how firm the API is. I think we can add in versioned repos in<br>
> 3.1+, in a natural way. Just like a user creates a Publication which triggers a publish, a user would<br>
> create a RepoVersion which would trigger a sync to produce that new RepoVersion. The repo versions work<br>
> needs to continue, but first I hope we prioritize getting to Beta 1 for core. There are a lot of use cases<br>
> in black on the MVP which are not implemented or written in Redmine. I believe closing that gap would be a<br>
> better use of time given that we can add this later.<br>
><br>
> What do others think?<br>
><br>
><br>
</span><span>> On Tue, Nov 28, 2017 at 2:24 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a> <mailto:<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>>> wrote:<br>
><br>
> I have a hard objection to including versioned repositories in 3.0. We agreed to make sure that our<br>
> current design would not prevent us from adding versioned repositories in the future. We did NOT agree<br>
> to including versioned repositories in 3.0 release. This is a big code change that did not go through<br>
> our regular planning process. I greatly appreciate your effort in driving this feature forward, but we<br>
> should take a step back and go through our regular process. I am also concerned that adding such a big<br>
> change at this time will delay the beta.<br>
><br>
> -Dennis<br>
><br>
><br>
</span>> On Tue, Nov 28, 2017 at 10:10 AM, Michael Hrivnak <<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a> <mailto:<a href="mailto:mhrivnak@redhat.com" target="_blank">mhrivnak@redhat.com</a>>><br>
<div><div class="m_-6618908000343350467m_1454868006967578345h5">> wrote:<br>
><br>
> Following up on previous discussions, I did an analysis of how repository versioning would impact<br>
> Pulp 3's current REST API and plugin API. A lot has changed since we last discussed the topic (in<br>
> May 2017), such as how we handle publications, and how the REST API is laid out. You can read the<br>
> analysis here:<br>
><br>
> <a href="https://pulp.plan.io/projects/pulp/wiki/Repository_Versions" rel="noreferrer" target="_blank">https://pulp.plan.io/projects<wbr>/pulp/wiki/Repository_Versions</a><br>
> <<a href="https://pulp.plan.io/projects/pulp/wiki/Repository_Versions" rel="noreferrer" target="_blank">https://pulp.plan.io/project<wbr>s/pulp/wiki/Repository_Version<wbr>s</a>><br>
><br>
> We previously discussed and vetted the mechanics at great length. While there was broad agreement<br>
> on the value to Pulp 3, there was uncertainty about the details of how it would impact REST<br>
> clients and plugin writers, and also uncertainty about how long it would take to fully implement.<br>
><br>
> In the course of my recent analysis, two things became clear. 1) both current APIs are not<br>
> compatible and would have to change. Details are on the wiki page above. 2) the PoC from earlier<br>
> this year indeed covers the hard parts, leaving mostly DRF details to sort out.<br>
><br>
><br>
> I don't agree with your assessment that the current REST API is not compatible with adding repository<br>
> versions. A repository version is it's own resource that can be added<br>
><br>
><br>
><br>
> I started rebasing the PoC onto current 3.0-dev, and within an hour I had it working with the<br>
> updated REST endpoints. With that having been so easy, I threw caution to the wind, and within a<br>
> few hours I had a fully functional branch that covered all the key use cases.<br>
><br>
> - sync creates a new version<br>
> - versions and their content sets are visible through the REST API<br>
> - each version shows what content was added and removed<br>
> - versions can be deleted, which queues a task that squashes changes as previously discussed<br>
> - the ChangeSet and pulp_file were updated to work with versions<br>
> - publish defaults to using the latest version<br>
><br>
> I also created a set of tests to help prove that it behaves correctly:<br>
><br>
> <a href="https://gist.github.com/mhrivnak/69af54063dff7465212914094dff34c2" rel="noreferrer" target="_blank">https://gist.github.com/mhriv<wbr>nak/69af54063dff7465212914094d<wbr>ff34c2</a><br>
> <<a href="https://gist.github.com/mhrivnak/69af54063dff7465212914094dff34c2" rel="noreferrer" target="_blank">https://gist.github.com/mhri<wbr>vnak/69af54063dff7465212914094<wbr>dff34c2</a>><br>
><br>
> I have just about 12 hours of recent work into it, and the code is PR-ready. It's just missing doc<br>
> updates and release notes. It's been difficult to keep discussion moving toward a full plan due to<br>
> the uncertainties mentioned above, so hopefully this can alleviate those concerns and give<br>
> everyone something concrete to look at.<br>
><br>
</div></div>> <a href="https://github.com/pulp/pulp/pull/3228" rel="noreferrer" target="_blank">https://github.com/pulp/pulp/<wbr>pull/3228</a> <<a href="https://github.com/pulp/pulp/pull/3228" rel="noreferrer" target="_blank">https://github.com/pulp/pulp/<wbr>pull/3228</a>><br>
> <a href="https://github.com/pulp/pulp_file/pull/20" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_<wbr>file/pull/20</a> <<a href="https://github.com/pulp/pulp_file/pull/20" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_<wbr>file/pull/20</a>><br>
<span>><br>
> Two notable items are missing. One is that there is no way to arbitrarily add and remove content<br>
> from a repo now, since this removes the "repositorycontent" endpoint. But we need to solve that<br>
> with a more formal and bulk add/remove API anyway. I also found that the "repositorycontent"<br>
> endpoint was not using tasks, and thus there was no repo locking, so it needed additional work<br>
> anyway. Based on this overall effort, I think it will be very easy to add if we just agree on what<br>
> the endpoints should look like.<br>
><br>
> The other is that publish does not in this PR accept a reference to a version. It always uses the<br>
> latest. That would also be a very easy enhancement to make.<br>
><br>
> I am happy to support getting this merged as I transition to being a more passive community<br>
> member, assuming there are no objections. I am also of course happy to help support this into the<br>
> future, as I believe strongly in its value and importance (see previous thread).<br>
><br>
> Please provide feedback and questions. If a live meeting this week would help expedite evaluation<br>
> of this effort, I'm happy to schedule that. And assuming there are no hard objections, I'm happy<br>
> to proceed with documentation updates.<br>
><br>
> Thanks!<br>
><br>
> --<br>
><br>
> Michael Hrivnak<br>
><br>
> Principal Software Engineer, RHCE<br>
><br>
> Red Hat<br>
><br>
><br>
> _____________________________<wbr>__________________<br>
> Pulp-dev mailing list<br>
</span>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a>><br>
><br>
><br>
><br>
> _____________________________<wbr>__________________<br>
> Pulp-dev mailing list<br>
> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a>><br>
><br>
><br>
><br>
> _____________________________<wbr>__________________<br>
> Pulp-dev mailing list<br>
> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a>><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailma<wbr>n/listinfo/pulp-dev</a>><br>
<div class="m_-6618908000343350467m_1454868006967578345HOEnZb"><div class="m_-6618908000343350467m_1454868006967578345h5">><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Pulp-dev mailing list<br>
> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
><br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div></div></div>
<br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/<wbr>mailman/listinfo/pulp-dev</a><br>
<br></blockquote></div><br></div>