[Pulp-dev] [pulp 3] proposed change to publishing REST api
Brian Bouterse
bbouters at redhat.com
Wed Nov 1 17:52:07 UTC 2017
+1 exactly to what @daviddavis said about 202 because normal
I also think we can support one-time publish params w/ their call to POST
/.../publication/. Assuming we want to store those options to recalled (but
not used) later, submitting that data to the publication endpoint is
probably the best option. It won't work if we try to store the one-time
options from many publishes on a single publisher object, but we can store
each one-time set of options on their respective publication objects. I
think we do want to store them because there is likely a use case where 'as
a user I can recall the one-time options used for any given publication.'
-Brian
On Wed, Nov 1, 2017 at 1:20 PM, David Davis <daviddavis at redhat.com> wrote:
> Calling a create endpoint and getting back a 202 with a way to monitor the
> status of the request is pretty common practice in REST APIs[0]. I think it
> would be intuitive to users familiar with REST APIs that have async
> operations.
>
> I think we could support allowing users to submit one-time publish params
> with their call to POST /../publications. It’s not great option but it’s an
> option.
>
> [0] http://jsonapi.org/format/#crud-creating-responses-202
>
>
> David
>
> On Wed, Nov 1, 2017 at 10:58 AM, Jeff Ortel <jortel at redhat.com> wrote:
>
>>
>>
>> On 11/01/2017 09:16 AM, Brian Bouterse wrote:
>> > Thanks for the response. Let's not move forward until we have more
>> agreement in this area. I've written some
>> > responses inline.
>> >
>> > On Wed, Nov 1, 2017 at 9:05 AM, Jeff Ortel <jortel at redhat.com <mailto:
>> jortel at redhat.com>> wrote:
>> >
>> > I'm not yet convinced about the proposed URL change for
>> publishing. Can you help me understand why a POST to
>> > the publications collection is more appropriate than the a POST to
>> a publisher?
>> >
>> >
>> > I believe the thinking is: REST suggests that POSTing to a resource is
>> expected to create a new resource of
>> > that type. So assume a users knows REST and they know they want to get
>> a Publication created in Pulp, they
>> > know exactly how to do that just by knowing REST. In the case of a POST
>> to a publisher url with a special
>> > 'publish' keyword on the end of it (the controller endpoint), they only
>> way a user could know to do that is by
>> > reading our docs. Both approaches would work, but I believe the former
>> is more aligned with REST which means
>> > users can do more without having to read Pulp docs.
>>
>> I believe that if the user is experienced with REST, they will be
>> expecting a 201 and an href to the created
>> resource to be returned instead of a 202 and a task href. Further, they
>> will expect the resource to be
>> created as defined in the POST body. I think the side-effect of running
>> the publisher to actually create the
>> resource will be unexpected. The user would still need to read the docs
>> to understand what's happening. Not
>> saying this is all together wrong, just challenging the assertion that
>> this would be more intuitive to the user.
>>
>> >
>> >
>> >
>> > A POST to the publications/ collection means the POST body should
>> define the publication to be created.
>> > Right? What about options that need to be passed to the publisher?
>> >
>> >
>> > Yes, if we look at the fields of the publisher (link below), there are
>> only two fields: 'created' and
>> > 'publisher'. Since 'created' is set on the server automatically, the
>> user would specify only the href to the
>> > publisher in the POST body. For the MVP, we don't accept one-time
>> options, and all other options are
>> > configured on the publisher which is a different url call from both the
>> publish controller and the publication
>> > resource. So for the MVP this approach would work well. The future case
>> also is better with this approach (I
>> > think). When we do introduce one-time options, where will we store them?
>>
>> Options are not stored. That's the point of them being one-time options
>> vs. publisher attributes. My concern
>> is that making this choice means that we will never support one-time
>> publishing options. What do others think
>> of that?
>>
>> We will probably be store them on the
>> > publication too, and that makes sense, because we can't store N,
>> one-time publish options on 1 publisher
>> > instance, but we can store N, one-time publish options on N
>> publications.
>> >
>> > https://github.com/pulp/pulp/blob/15857fb0831c0998219a32e8d6
>> ba52abdba20888/platform/pulpcore/app/models/publication.py#L6
>> >
>> >
>> >
>> > On 10/31/2017 03:13 PM, Brian Bouterse wrote:
>> > > @dkliban, I'm +1 on that.
>> > >
>> > > @all, Please jump in if this is not the best direction for us to
>> go.
>> > >
>> > > On Tue, Oct 31, 2017 at 3:55 PM, Dennis Kliban <
>> dkliban at redhat.com <mailto:dkliban at redhat.com>
>> > <mailto:dkliban at redhat.com <mailto:dkliban at redhat.com>>> wrote:
>> > >
>> > > On Tue, Oct 31, 2017 at 3:52 PM, Brian Bouterse <
>> bbouters at redhat.com <mailto:bbouters at redhat.com> <mailto:
>> bbouters at redhat.com
>> > <mailto:bbouters at redhat.com>>> wrote:
>> > >
>> > > Would that return the 202 w/ a link to the task because
>> the publication hasn't been created yet? Then
>> > > using the created_resources they can see what was
>> created, and in the event of failure the task fails
>> > > and there are no created_resources.
>> > >
>> > > @dkliban is ^ the idea?
>> > >
>> > >
>> > > Yes, the response would the same as it for the /publish URL
>> right now. This is just a change in the URL
>> > > that is used to make the request.
>> > >
>> > >
>> > >
>> > > On Tue, Oct 31, 2017 at 3:48 PM, Dennis Kliban <
>> dkliban at redhat.com <mailto:dkliban at redhat.com>
>> > <mailto:dkliban at redhat.com <mailto:dkliban at redhat.com>>> wrote:
>> > >
>> > >
>> > >
>> > > On Tue, Oct 31, 2017 at 3:40 PM, Brian Bouterse <
>> bbouters at redhat.com
>> > <mailto:bbouters at redhat.com> <mailto:bbouters at redhat.com <mailto:
>> bbouters at redhat.com>>>
>> > > wrote:
>> > >
>> > > +1 to updating #3033 to have a created_resources
>> attribute which would be a list of
>> > > GenericForeignKeys. It also needs docs, but I'm
>> not entirely sure where.
>> > >
>> > > If we're going to introduce the above attribute,
>> I think having the controller endpoint as-is
>> > > would be the most usable. @dkliban do you see
>> value in changing the URL structure if the
>> > > created_resources attribute is introduced?
>> > >
>> > >
>> > > This API call creates a publication resource. A POST
>> to publishers/<id>/publications/ seems most
>> > > appropriate for creating new publication resources.
>> > >
>> > > I can help review/groom these if that is helpful.
>> > >
>> > > -Brian
>> > >
>> > >
>> > > On Tue, Oct 31, 2017 at 1:39 PM, David Davis <
>> daviddavis at redhat.com <mailto:daviddavis at redhat.com>
>> > > <mailto:daviddavis at redhat.com <mailto:
>> daviddavis at redhat.com>>> wrote:
>> > >
>> > > Personally I am not opposed to the url
>> endpoint you suggest.
>> > >
>> > > It also seems like there is some consensus
>> around adding a ‘created resources’
>> > > relationship to Task or at least prototyping
>> that out to see what it would look like.
>> > >
>> > > If no one disagrees, should I update issue
>> #3033 with those two items?
>> > >
>> > >
>> > > David
>> > >
>> > > On Wed, Oct 25, 2017 at 1:23 PM, Dennis
>> Kliban <dkliban at redhat.com <mailto:dkliban at redhat.com>
>> > > <mailto:dkliban at redhat.com <mailto:
>> dkliban at redhat.com>>> wrote:
>> > >
>> > > On Wed, Oct 25, 2017 at 11:24 AM, David
>> Davis <daviddavis at redhat.com <mailto:daviddavis at redhat.com>
>> > > <mailto:daviddavis at redhat.com <mailto:
>> daviddavis at redhat.com>>> wrote:
>> > >
>> > > I don’t know that the ambiguity
>> around whether a task has a publication or not is
>> > > a big deal. If I call the publication
>> endpoint, I’d expect a publication task
>> > > which either has 1 publication or 0
>> (if the publication failed) attached to it.
>> > >
>> > > In terms of ambiguity, I see a worse
>> problem around adding a task_id field to
>> > > publications. As a user, I don’t know
>> if a publication failed or not when I get
>> > > back a publication object. Instead, I
>> have to look up the task to see if it is a
>> > > real (or successful) publication.
>> Moreover, since we allow users to remove/clean
>> > > up tasks, that task may not even
>> exist anymore.
>> > >
>> > >
>> > > I agree that the ephemeral nature of
>> tasks makes the originally proposed solution
>> > > non-deterministic. I am open to
>> associating 'resources created' with a task instead.
>> > >
>> > > However, I still think there is value in
>> changing the rest API endpoint for starting a
>> > > publish task to POST
>> > > /api/v3/repositories/<repo-id
>> >/publishers/<type>/<name>/publications/. However, I will
>> > > start a separate thread for that
>> discussion.
>> > >
>> > > - Dennis
>> > >
>> > >
>> > >
>> > > David
>> > >
>> > > On Wed, Oct 25, 2017 at 11:03 AM,
>> Brian Bouterse <bbouters at redhat.com <mailto:bbouters at redhat.com>
>> > > <mailto:bbouters at redhat.com <mailto:
>> bbouters at redhat.com>>> wrote:
>> > >
>> > >
>> > >
>> > > On Tue, Oct 24, 2017 at 10:00 PM,
>> Michael Hrivnak <mhrivnak at redhat.com <mailto:mhrivnak at redhat.com>
>> > > <mailto:mhrivnak at redhat.com
>> <mailto:mhrivnak at redhat.com>>> wrote:
>> > >
>> > >
>> > >
>> > > On Tue, Oct 24, 2017 at 2:11
>> PM, Brian Bouterse <bbouters at redhat.com <mailto:bbouters at redhat.com>
>> > > <mailto:bbouters at redhat.com
>> <mailto:bbouters at redhat.com>>> wrote:
>> > >
>> > > Thanks everyone for all
>> the discussion! I'll try to recap the
>> > problem
>> > > and some of the solutions
>> I've heard. I'll also share some of my
>> > > perspective on them too.
>> > >
>> > > What problem are we
>> solving?
>> > > When a user calls
>> "publish" (the action API endpoint) they get a 202
>> > > w/ a link to the task.
>> That task will produce a publication. How can
>> > > the user find the
>> publication that was produced by the task? How can
>> > > the user be sure the
>> publication is fully complete?
>> > >
>> > >
>> > > What are our options?
>> > > 1) Start linking to
>> created objects from task status. I believe its
>> > > been clearly stated about
>> why we can't do this. If it's not
>> > clear, or
>> > > if there are other things
>> we should consider, let's talk about it.
>> > > Acknowledging or
>> establishing agreement on this is crucial because a
>> > > change like this would
>> bring back a lot of the user pain from
>> > pulp2. I
>> > > believe the HAL
>> suggestion falls into this area.
>> > >
>> > >
>> > > I may have missed something,
>> but I do not think this is clear. I
>> > know that
>> > > Pulp 2's API included a lot
>> of unstructured data, but that is not at all
>> > > what I'm suggesting here.
>> > >
>> > > It is standard and
>> recommended practice for REST API responses to
>> > include
>> > > links to resources along with
>> information about what type of
>> > resource each
>> > > link references. We could
>> include a reference to the created
>> > resource and
>> > > an identifier for what type
>> of resource it is, and that would be well
>> > > within the bounds of good
>> REST API design. HAL is just one of
>> > several ways
>> > > to accomplish that, and I'm
>> not pitching any particular solution
>> > there. In
>> > > any case, I'm not sure what
>> the problem would be with this approach.
>> > >
>> > >
>> > > I agree it is a standard practice
>> for a resource to include links to other
>> > > resources, but the proposal is to
>> include "generic" links is different and
>> > > creates a different user
>> experience. I believe referencing the task from the
>> > > publication will be easier for
>> users and clients. When a user looks up a
>> > > publication, they will always
>> know they'll get between 0 and 1 links to a
>> > > task. You can use that to check
>> the state of the publication. If we link to
>> > > "generic" resources (like a
>> publication) from a task, then if I ask a
>> > user "do
>> > > you expect task
>> ede3af3e-d5cf-4e18-8c57-69ac4d4e4de6 to contain a link to a
>> > > publication or not?" you can't
>> know until you query it. I think that
>> > ambiguity
>> > > was a pain point in Pulp2. I
>> don't totally reject this solution, but this is
>> > > an undesirable property (I think).
>> > >
>> > >
>> > >
>> > >
>> > > 2) Have the user find the
>> publication via query that sorts on
>> > time and
>> > > filters only for a
>> specific publisher. This could be fragile because
>> > > with a multi-user system
>> and no hard references between publications
>> > > and tasks, answering the
>> question "which is the publication for
>> > me" is
>> > > hard because another user
>> could have submitted a publish too. While
>> > > not totally perfect, this
>> could work.
>> > >
>> > >
>> > > In theory if a user queried
>> for a publication from a specific publisher
>> > > that was created between the
>> start and end times of the task, that
>> > should
>> > > unambiguously identify the
>> correct publication. But depending on
>> > > timestamps is not a
>> particularly robust nor confidence-inspiring way to
>> > > reference a resource.
>> > >
>> > > Agreed and Agreed
>> > >
>> > >
>> > >
>> > >
>> > > 3) Have the user create a
>> publication directly like any other REST
>> > > resource, and help the
>> user understand the state of that
>> > resource over
>> > > time. I believe the
>> proposal at the start of this thread is
>> > > recommending this
>> solution. I'm also +1 on this solution.
>> > >
>> > >
>> > > I think the problem with this
>> is that a user cannot create a
>> > publication.
>> > > A user can only ask a plugin
>> to create a publication. Until the plugin
>> > > creates the publication,
>> there is no publication.
>> > >
>> > >
>> > > Note a publication is an object,
>> but really we mean a publication and it's
>> > > related PublishedArtifact,
>> PublishedMetadat, etc objects. It would be
>> > > straightforward for a user to
>> create a publication using the viewset and
>> > have
>> > > the task associated with it call
>> the publisher to build out the associated
>> > > PublishedArtifact,
>> PublishedContent, PublishedMetadata, etc. We should
>> > explore
>> > > if this is good or not, but it is
>> possible.
>> > >
>> > > As an aside, this is related to a
>> problem everyone should be aware of: the
>> > > existence of a publication does
>> not guarantee that publication is finished
>> > > publishing. Even with option 1,
>> where the task creates the publisher and
>> > links
>> > > to it in the task status, while
>> the publisher is running it must save the
>> > > Publication so that the
>> PublishedArtifact, etc can link to it. So for any
>> > > given publication, in order to
>> know if it's "fully finished and consistent"
>> > > you must be able to check the
>> status of the associated task that
>> > produced it.
>> > >
>> > >
>> > >
>> > > As an aside, I don't
>> think considering versioned repos as a possible
>> > > solution is helping us
>> with this problem. The scope of the current
>> > > problem is relatively
>> small and the scope of planning for versioned
>> > > repos is large.
>> > >
>> > >
>> > > Versioned repos is a
>> potential solution. In that scenario, a user would
>> > > request publication of a
>> specific repo version (perhaps defaulting
>> > to the
>> > > latest), the publication
>> would be linked to that version, and that is an
>> > > easy mechanism for the user
>> to find the publication they want.
>> > Ultimately
>> > > the user is interested in
>> working with a specific content set
>> > anyway. They
>> > > get a repo to a state where
>> it has the content they want, and then they
>> > > publish that content set. No
>> matter what we do with publications, users
>> > > will think of them in terms
>> of related content sets. A repo version is
>> > > that immutable content set
>> they can work with confidently.
>> > >
>> > >
>> > > It's neat to me that that
>> versions are snapshots of content and publications
>> > > are snapshots of content.
>> Publications already create much of the value
>> > > propostion of versioned repos
>> with publications. They allow you to work with
>> > > specific content sets like you
>> describe. Also they allow for rollback.
>> > So that
>> > > is all great for our users. For
>> this thread, I want to bring the
>> > conversation
>> > > back to where it started, solving
>> a small problem about linking two
>> > resources
>> > > that already exist.
>> > >
>> > >
>> > > It helps the rollback
>> scenario a lot as well. Versioning repos allows a
>> > > user to see what the
>> differences are between two content sets, and thus
>> > > two different publications,
>> which informs them about when and how
>> > far back
>> > > they should roll back a
>> distribution.
>> > >
>> > >
>> > > - user discovers a horrible
>> flaw in a piece of content
>> > > - user queries for which
>> version of the repo introduced that piece
>> > of content
>> > > - user updates the
>> distribution to serve the publication that came
>> > before
>> > > the one which introduced the
>> piece of content, optionally re-publishing
>> > > that version in case its
>> publication was deleted or had never been
>> > made in
>> > > the first place.
>> > >
>> > > --
>> > >
>> > > Michael Hrivnak
>> > >
>> > > Principal Software Engineer,
>> RHCE
>> > >
>> > > Red Hat
>> > >
>> > >
>> > >
>> > > _____________________________
>> __________________
>> > > Pulp-dev mailing list
>> > > Pulp-dev at redhat.com <mailto:
>> Pulp-dev at redhat.com>
>> > <mailto:Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>>
>> > > https://www.redhat.com/mailma
>> n/listinfo/pulp-dev
>> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
>> > > <https://www.redhat.com/mailm
>> an/listinfo/pulp-dev <https://www.redhat.com/mailman/listinfo/pulp-dev>>
>> > >
>> > >
>> > >
>> > > _____________________________
>> __________________
>> > > Pulp-dev mailing list
>> > > Pulp-dev at redhat.com <mailto:
>> Pulp-dev at redhat.com> <mailto:Pulp-dev at redhat.com
>> > <mailto:Pulp-dev at redhat.com>>
>> > > https://www.redhat.com/mailma
>> n/listinfo/pulp-dev
>> > <https://www.redhat.com/mailman/listinfo/pulp-dev>
>> > > <https://www.redhat.com/mailm
>> an/listinfo/pulp-dev
>> > <https://www.redhat.com/mailman/listinfo/pulp-dev>>
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > Pulp-dev mailing list
>> > > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>> > > https://www.redhat.com/mailman/listinfo/pulp-dev <
>> https://www.redhat.com/mailman/listinfo/pulp-dev>
>> > >
>> >
>> >
>> > _______________________________________________
>> > Pulp-dev mailing list
>> > Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>> > https://www.redhat.com/mailman/listinfo/pulp-dev <
>> https://www.redhat.com/mailman/listinfo/pulp-dev>
>> >
>> >
>>
>>
>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20171101/69892cce/attachment.htm>
More information about the Pulp-dev
mailing list