[Pulp-dev] versioned repositories

Wed May 24 19:38:51 UTC 2017

Versioned repos are super important to the future of Pulp. I need to state
that because I'm going to make a case for us deferring this
discussion/design/prototype until after Pulp 3.0 is GA. My major concern is
getting Pulp3 to a release sooner instead of later.

* To retell what I heard @jsherrill state on the use cases call yesterday.
Implementing a feature internally in an MVP and not fully exposing it to
the user does not make sense. Internals we can change/rework all day, but
the thing we need to get right is the API because of semver.

* If ^ is true, I think all we need right now is to understand what the API
looks like with 3.0 that will allow us to bring the versioned repo
amazingness with 3.1+ without breaking semver. Some of that is in this
thread, but most of the thread is answering questions about use cases for
versioned repos usage which would be in 3.1 at the earliest.

We could spend time and energy on sorting use cases and implementation
options now, but I think instead we should get to work on wrapping up core
and getting into the plugin conversion which will get us closer to release.

If we adopt ^, then the next use cases call would be the regularly
scheduled 'Status and Consumer Applicability' call. Also we would need to
get answers to: What does the API look like for repositories in 3.0 and
what does it look like in 3.1+ when versioned repos are introduced. We
should answer those questions not on a use case call because API layout
isn't a use case concern.

-Brian

On Wed, May 24, 2017 at 1:59 PM, Michael Hrivnak <mhrivnak at redhat.com>
wrote:

> On Wed, May 24, 2017 at 11:26 AM, Dennis Kliban <dkliban at redhat.com>
> wrote:
>
>> I noticed that the REST API examples don't mention anything about
>> deleting a particular version of a repository. This is a use case that we
>> need to support.
>>
>> -Dennis
>>
>
>
> Great point. I was hoping we could avoid that need in the short term, but
> in IRC yesterday, Justin Sherrill brought up an important, but hopefully
> rare, use case. If a user accidentally adds something secret to a repo and
> needs to remove it entirely from Pulp, we need to provide a way to
> accomplish that. In Pulp 2 that is cumbersome to do, but at least possible.
> It requires you to remove it from every repo it's in, re-publish them all,
> and do an orphan removal.
>
> Versions' content being immutable, I think that *any* time you change a
> repo, it creates a new version with the next new-and-never-before-used
> integer.
>
> Consider a repo that has versions 1-16, and I create 17 while accidentally
> adding a secret. The right next step for a user is to make whatever changes
> are necessary to the repo to remove their secret. This is so far the same
> as you would do with Pulp 2. No particular version awareness is necessary
> yet. By making that change, which presumably removes your secret content,
> Pulp creates version 18.
>
> Version 17 is now thankfully the only version in history that contains
> your secret. Removing that version will be like tearing a page out of a
> history book. You can understand the history before, and you can understand
> the history after. You can even understand what changed about the world
> since the page before to the one after. But you also plainly see that there
> is a gap in records, so exactly what happened on that page will never again
> be known. But that's ok.
>
> Data Details
> -----
>
> With this model, removing a version just squashes its history into the
> next version. It's quite simple, so let's dig into the associations for a
> moment.
>
> Any association that was added in 17 and removed in 18 just gets deleted.
>
> Any other association added or removed in version 17 has the number 17
> replaced with 18.
>
> This approach can squash arbitrary ranges of versions whether a user wants
> to just trim history, or deliberately remove something sensitive.
>
> REST API
> -----------
>
> Ok, finally to your question! Three options come to mind. There are likely
> more, so please speak up if you'd like to add a favorite. I assume that in
> any approach we take, the actual version deletion would happen in a task
> that locks the repo.
>
> 1) Allow a DELETE call on a version. It's simple and intuitive. The only
> downside is that you can't specify a range to remove in one operation.
> Maybe that doesn't matter too much; it'll be quick. I think this would be a
> good starting point.
>
> DELETE /api/v3/repositories/foo/versions/17/
>
> 2) Add a squash endpoint for the collection. The endpoint would in some
> way take a range of versions to squash. Maybe we use filtering syntax along
> these lines:
>
> DELETE /api/v3/repositories/foo/versions/?num__gte=5&num__lte=10
>
> or an action endpoint on the repo
>
> POST /api/v3/repositories/foo/squash_versions/
>   {'start': 5, 'end': 10}
>
> 3) Add an action endpoint on the surviving version:
>
> POST /api/v3/repositories/foo/versions/10/squash_since/
>   {'since': 5}
>
> Of these, I lean toward starting with the first option. It's simple and
> intuitive, and it can accomplish all known use cases, even if it may be
> inefficient for deleting a range of versions. We could add something
> resembling options 2 or 3 in a later 3.y release if necessary.
>
> But, I could also see an argument for making just one endpoint that
> facilitates a range, and it accommodates all used cases efficiently out of
> the gate. Then we'd never need option 1. It's the "There should be one--
> and preferably only one --obvious way to do it." approach.
>
> Other Considerations
> --------------
>
> One other factor to consider is publications. In an ideal world, Pulp
> would also delete any publications associated with a version that's being
> deleted. That's definitely not something Pulp 2 does (you can remove
> content without re-publishing, potentially leaving broken links if content
> is orphan-removed). If we don't tackle this in 3.0, we'll probably want to
> tackle Publications as a first-class thing some time in 3.y. In any case,
> keeping this in mind as a possible side-effect of version deletion is
> worthwhile.
>
> So what do you all think? Preferences among these? Other ideas? More
> questions?
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170524/0e4b9fb5/attachment.htm>