[Pulp-dev] Basic Lazy Streaming for Pulp3 Ready for Rough Testing

[Brian Bouterse]

I've been developing the streamer functionality, and it's correctly working
(in my testing) as driven from the Remote.policy attribute. It correctly
works with 'immediate', 'on_demand', and 'cache_only'. Read more about the
expected behaviors in the epic [0].

# Try it out!
Here is the core commit needed:  https://github.com/pulp/pulp/pull/3738
Here is the streamer you should pip install from master:
https://github.com/bmbouter/pulp_streamer
Here is what it looks like to port a plugin using DeclarativeVersion, e.g.
pulp_file to support lazy:  https://github.com/pulp/pulp_file/pull/132

You'll need to configure Pulp's webserver for streaming. I did this by
exporting an environment var to dynaconf in the same bash environment as my
django run server. Specifically I configured Pulp to redirect to port
localhost:8080/streamer/ with this command:

export PULP_CONTENT='@json {"HOST": null, "WEB_SERVER": "django",
"REDIRECT": {"ENABLED": true, "PORT": 8080, "HOST": "localhost",
"PATH_PREFIX": "/streamer/"}}'

Then I run the streamer (after pip installed) with gunicorn which you also
need to pip install. Run it with:

gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class
aiohttp.GunicornWebWorker -w 2

Then sync a pulp_file repo with policy='on_demand' or policy='cache_only'
and see how Pulp behaves.

Feedback, ideas, concerns are welcome in any form. Note this is still
rough, and the following are known things to be done:

* fix tests to get Travis passing
* docs for the streamer and for pulpcore
* an installer role to install the streamer
* integration with squid to cache lots of data at the streamer
* transfer the pulp_streamer to the Pulp org on github
* publish an initial release to PyPI for users to use it
* write a blog post about porting to it and using it
* make a demo

[0]: https://pulp.plan.io/issues/3693

Thanks!
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20181120/00655964/attachment.htm>