<div dir="ltr"><div>I agree if we couldn't add versioned repos later, then we should add it now. I believe the current API is already well setup to add it later. Specifically, here is a non-nested example of the versioned repository resource to be added in 3.1+ </div><div> </div><div>/api/v3/repo_version/</div><div> </div><div>The user would POST to that to cause a new repo version to be created. What are the contents of that repo version? Well it depends on the POST data. That repo version could be the result of a sync being performed, or perhaps it could also include mass associate/unassociate operations right there. In other words, I see a clear path to adding such a resource, and I see endless flexibility in terms of the users intention (via POST data) of what should be contained in that repo version. </div><div> </div><div>Can some feedback be given on this design? Why won't this work in 3.1+?</div><div> </div><div>To give my own answer to the question. The main impact I see when adding this in 3.1+ is on plugin writers not users. Specifically the plugin writer API would change because a plugin writer would no longer be associating content with a repo as it's primary activity. Instead their code would be to associate content with a repo version. We can roll out that change in a clear, coordinated way with the plugin community. It's also appropriate because the plugin API is still < 1.0. We would bump the plugin api from 0.1 to 0.2 and plugins can easily declare their compatibility as pulpcore-plugin < 0.2. Relative to the effort of creating a plugin, porting plugin code for a change like this I think would be low effort. </div><div> </div></div><div class="gmail_extra"> <div class="gmail_quote">On Wed, Nov 29, 2017 at 10:19 AM, Jeff Ortel <<a href="mailto:jortel@redhat.com" target="_blank">jortel@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On 11/29/2017 07:22 AM, David Davis wrote: > I think we could design an API in 3.0 that would support versioned repos in 3.1+. However, our current API > does not. For example, the /repositorycontents/ endpoint doesn't make sense with versioned repos as no one > would want to add/remove content units one-by-one when doing so would generate a new repo version each time. > Imagine that we end up with an endpoint in 3.0 that’s not compatible with versioned repos. What would we do? I > think this is a strong argument for adding versioned repos now. agreed. > > Of course the main drawback is that it might delay the beta. But I wonder by how much. It might be good to > groom the versioned repo user stories so that (a) we can see how much value they provide to end users and (b) > how closely they align with the work @mhrivnak has done. agreed. > > > David > > On Tue, Nov 28, 2017 at 4:00 PM, Brian Bouterse <<a href="mailto:bbouters@redhat.com">bbouters@redhat.com</a> <mailto:<a href="mailto:bbouters@redhat.com">bbouters@redhat.com</a>>> wrote: > > In reading back over the last email thread in May, it ended with us looking at URL options to ensure we > could release 3.0 and add in repo versions in 3.1+. We definitely want repo versions in the 3.y line, so > we wanted to make sure that was possible. If it wasn't, then we may have to add it into 3.0. > > That question is a lot easier now given how firm the API is. I think we can add in versioned repos in > 3.1+, in a natural way. Just like a user creates a Publication which triggers a publish, a user would > create a RepoVersion which would trigger a sync to produce that new RepoVersion. The repo versions work > needs to continue, but first I hope we prioritize getting to Beta 1 for core. There are a lot of use cases > in black on the MVP which are not implemented or written in Redmine. I believe closing that gap would be a > better use of time given that we can add this later. > > What do others think? > > > On Tue, Nov 28, 2017 at 2:24 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com">dkliban@redhat.com</a> <mailto:<a href="mailto:dkliban@redhat.com">dkliban@redhat.com</a>>> wrote: > > I have a hard objection to including versioned repositories in 3.0. We agreed to make sure that our > current design would not prevent us from adding versioned repositories in the future. We did NOT agree > to including versioned repositories in 3.0 release. This is a big code change that did not go through > our regular planning process. I greatly appreciate your effort in driving this feature forward, but we > should take a step back and go through our regular process. I am also concerned that adding such a big > change at this time will delay the beta. > > -Dennis > > > On Tue, Nov 28, 2017 at 10:10 AM, Michael Hrivnak <<a href="mailto:mhrivnak@redhat.com">mhrivnak@redhat.com</a> <mailto:<a href="mailto:mhrivnak@redhat.com">mhrivnak@redhat.com</a>>> <div><div class="h5">> wrote: > > Following up on previous discussions, I did an analysis of how repository versioning would impact > Pulp 3's current REST API and plugin API. A lot has changed since we last discussed the topic (in > May 2017), such as how we handle publications, and how the REST API is laid out. You can read the > analysis here: > > <a href="https://pulp.plan.io/projects/pulp/wiki/Repository_Versions" rel="noreferrer" target="_blank">https://pulp.plan.io/projects/pulp/wiki/Repository_Versions</a> > <<a href="https://pulp.plan.io/projects/pulp/wiki/Repository_Versions" rel="noreferrer" target="_blank">https://pulp.plan.io/projects/pulp/wiki/Repository_Versions</a>> > > We previously discussed and vetted the mechanics at great length. While there was broad agreement > on the value to Pulp 3, there was uncertainty about the details of how it would impact REST > clients and plugin writers, and also uncertainty about how long it would take to fully implement. > > In the course of my recent analysis, two things became clear. 1) both current APIs are not > compatible and would have to change. Details are on the wiki page above. 2) the PoC from earlier > this year indeed covers the hard parts, leaving mostly DRF details to sort out. > > > I don't agree with your assessment that the current REST API is not compatible with adding repository > versions. A repository version is it's own resource that can be added > > > > I started rebasing the PoC onto current 3.0-dev, and within an hour I had it working with the > updated REST endpoints. With that having been so easy, I threw caution to the wind, and within a > few hours I had a fully functional branch that covered all the key use cases. > > - sync creates a new version > - versions and their content sets are visible through the REST API > - each version shows what content was added and removed > - versions can be deleted, which queues a task that squashes changes as previously discussed > - the ChangeSet and pulp_file were updated to work with versions > - publish defaults to using the latest version > > I also created a set of tests to help prove that it behaves correctly: > > <a href="https://gist.github.com/mhrivnak/69af54063dff7465212914094dff34c2" rel="noreferrer" target="_blank">https://gist.github.com/mhrivnak/69af54063dff7465212914094dff34c2</a> > <<a href="https://gist.github.com/mhrivnak/69af54063dff7465212914094dff34c2" rel="noreferrer" target="_blank">https://gist.github.com/mhrivnak/69af54063dff7465212914094dff34c2</a>> > > I have just about 12 hours of recent work into it, and the code is PR-ready. It's just missing doc > updates and release notes. It's been difficult to keep discussion moving toward a full plan due to > the uncertainties mentioned above, so hopefully this can alleviate those concerns and give > everyone something concrete to look at. > </div></div>> <a href="https://github.com/pulp/pulp/pull/3228" rel="noreferrer" target="_blank">https://github.com/pulp/pulp/pull/3228</a> <<a href="https://github.com/pulp/pulp/pull/3228" rel="noreferrer" target="_blank">https://github.com/pulp/pulp/pull/3228</a>> > <a href="https://github.com/pulp/pulp_file/pull/20" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_file/pull/20</a> <<a href="https://github.com/pulp/pulp_file/pull/20" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_file/pull/20</a>> > > Two notable items are missing. One is that there is no way to arbitrarily add and remove content > from a repo now, since this removes the "repositorycontent" endpoint. But we need to solve that > with a more formal and bulk add/remove API anyway. I also found that the "repositorycontent" > endpoint was not using tasks, and thus there was no repo locking, so it needed additional work > anyway. Based on this overall effort, I think it will be very easy to add if we just agree on what > the endpoints should look like. > > The other is that publish does not in this PR accept a reference to a version. It always uses the > latest. That would also be a very easy enhancement to make. > > I am happy to support getting this merged as I transition to being a more passive community > member, assuming there are no objections. I am also of course happy to help support this into the > future, as I believe strongly in its value and importance (see previous thread). > > Please provide feedback and questions. If a live meeting this week would help expedite evaluation > of this effort, I'm happy to schedule that. And assuming there are no hard objections, I'm happy > to proceed with documentation updates. > > Thanks! > > -- > > Michael Hrivnak > > Principal Software Engineer, RHCE > > Red Hat > > > _______________________________________________ > Pulp-dev mailing list > <a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a>> > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a>> > > > > _______________________________________________ > Pulp-dev mailing list > <a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a>> > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a>> > > > > _______________________________________________ > Pulp-dev mailing list > <a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a> <mailto:<a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a>> > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> <<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a>> <div class="HOEnZb"><div class="h5">> > > > > _______________________________________________ > Pulp-dev mailing list > <a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> > </div></div> _______________________________________________ Pulp-dev mailing list <a href="mailto:Pulp-dev@redhat.com">Pulp-dev@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> </blockquote></div> </div>