[Pulp-dev] Proposal: merge the content-app & streamer

Dennis Kliban dkliban at redhat.com
Wed Dec 5 21:07:06 UTC 2018


Since no one is objecting, I'd like to see the PR[0] from @bmbouter
finished up and merged soon. I am updating the pass-through cache story[1]
to get it ready for implementation. I would like to remove all mention of
separate content app and streamer from the description.


[0] https://github.com/pulp/pulp/pull/3779
[1] https://pulp.plan.io/issues/3894

On Tue, Dec 4, 2018 at 1:54 PM Tatiana Tereshchenko <ttereshc at redhat.com>
wrote:

> +1 to merge
> +1 to have clear docs for plugin writers how to create their own content
> app
>
> On Mon, Dec 3, 2018 at 11:25 PM Dennis Kliban <dkliban at redhat.com> wrote:
>
>> It was pointed out on IRC that plugins that have to supply their own
>> content app (such as docker) currently need to supply 2 implementations of
>> it in order to support on-demand use cases. One using django and another
>> using aiohttp.
>>
>> We should not burden plugin writers in such a way. We really have to make
>> the proposed change.
>>
>> On Mon, Dec 3, 2018 at 3:24 PM Dana Walker <dawalker at redhat.com> wrote:
>>
>>> In light of the efficiency gains and lack of significant drawbacks, I'm
>>> +1 on this proposal.
>>>
>>> --Dana
>>>
>>>
>>> Dana Walker
>>>
>>> Associate Software Engineer
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com>
>>> <https://red.ht/sig>
>>>
>>>
>>> On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban <dkliban at redhat.com> wrote:
>>>
>>>> I like the idea of combining the two applications for all the reasons
>>>> already outlined on this thread. The user experience is going to be
>>>> simplified by this change. However, I want to point out that it will also
>>>> alter the plugin writer experience. Plugin writers that want to have their
>>>> own content app will now need to provide it as a plugin for the content app
>>>> (which is not a Django project). We should be able to clearly document this
>>>> for plugin writers. pulp_docker plugin will need to adopt this change. For
>>>> that reason I'd like us to make a decision on this soon.
>>>>
>>>> On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel <jortel at redhat.com> wrote:
>>>>
>>>>> *BACKGROUND*
>>>>>
>>>>> The pulp3 content app and the streamer (in-progress) currently have a
>>>>> lot of duplicate code and functionality.  At the very least, I think there
>>>>> is a opportunity to refactor both and share code.  But, this would leave us
>>>>> with two components with significant overlap in functionality.
>>>>>
>>>>> The functionality exclusive to the content-app:
>>>>>   - Optionally delegate file serving to a web server. (Eg:
>>>>> mod_xsendfile).
>>>>>   - Optional redirect to the streamer.
>>>>>
>>>>> The functionality exclusive to the streamer:
>>>>>   - Using the Remote & RemoteArtifact to download the file and stream
>>>>> on demand.
>>>>>
>>>>> Not much difference which raises the question: "Why do we have both?"
>>>>> I think the answer may be that we don't.
>>>>>
>>>>> *PROPOSAL*
>>>>>
>>>>> Let's pull the content-app out and merge it with the streamer.  The
>>>>> new content (app) would have *streamer* architecture &
>>>>> functionality.  When a requested artifact has not been downloaded, it would
>>>>> download/streamed instead of REDIRECT.  This does mean that deployments and
>>>>> development environments would need to run an additional service to serve
>>>>> content.  The /pulp/content endpoint would be on a different port than the
>>>>> API.  I see this separation as a healthy thing.  There is significant
>>>>> efficiency to be gained as well.  Let's start with eliminating the
>>>>> REDIRECTs.  Cutting the GET requests in half is a win for both the client,
>>>>> the network and the Pulp web stack.  Next is database queries.  Since both
>>>>> applications needed to perform many of the same queries, combining the
>>>>> applications will roughly cut them in half as well.  Since the streamer is
>>>>> based on asyncio and so would the merged app.
>>>>>
>>>>> There are probably lots of other pros/cons I have not considered but
>>>>> it seems relatively straight forward.
>>>>>
>>>>> I'm thinking the new content app/service would be named:
>>>>> *pulp-content*.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> _______________________________________________
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> _______________________________________________
>>>> Pulp-dev mailing list
>>>> Pulp-dev at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>> _______________________________________________
>> Pulp-dev mailing list
>> Pulp-dev at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20181205/1eb1eb01/attachment.htm>


More information about the Pulp-dev mailing list