[Pulp-dev] Asyncio I/O Downloaders

[Jeff Ortel]

On 08/24/2017 09:51 AM, Brian Bouterse wrote:
> The next step in considering the asyncio downloaders is done and ready to be looked at. The PR made [0]
> replaces the existing downloaders and is "merge-ready". The best way to read about them is through their docs
> [1]. Here are some of its highlights:
> 
> # Code size and docs
> * huge code savings by switching to asyncio. removes 2427 lines and adds 558 lines.

Replacing concurrent.futures with asyncio would remove a fair amount of code in batch.py but the remainder of
lines of code removed discarded features.  Each can be correlated to use cases [2].

- Custom error recovery.
    Supported recoverable error recovery.
    Supported unavailable mirrors.

- Event notifications.
    Supported streamer forwarding of HTTP headers.
    Supported ChangeSet knowing when an artifact was actually downloaded.
    Supported calculating digests and size needed ONLY for creating Artifacts.
    Supported (future) progress reporting when desired.
    Supported (future) mirror lists.

- Custom Writers - handling of downloaded bits.
    Supported writing files to disk.
    Supported the streamer streaming downloaded bit stream in the Twisted response instead of writing
    the file to disk.

- File URLs.
    supported: "file:///myfile"

- Custom validations without having to subclass the downloader.
    Included: digest and size validations but plugin writer could implement additional validations.

- Standardized settings for SSL and Auth.
    Supported consistent settings across downloader types.
    Supported validation of settings.

- Shared Contexts - collaborative sharing of resources between downloaders.
    Supported shared: sessions (connection pools), auth tokens and resolved mirror lists.
    Supported the sharing of resources without requiring the user of the downloader to know
    which resources are shared or how they are shared.  Resource sharing needs to happen in the
    streamer too.

- Support for HEAD requests (HttpDownload).
    Just fetches the HTTP headers.

- Standard exceptions.
    Supported catching DownloadError, NotFound and NotAuthorized.
    Encapsulated client lib specific exceptions and error behaviors.

- Downloader Attachments - attach any user defined object to the downloader.
    Supported easy correlation between a downloader and an artifact being downloaded.
	

> * ^ is surprising considering the majority of lines added are docs
> * Read the compiled docs here [1]
> 
> # Features
> * It has 100% feature parity with the existing http/https downloaders. 

See gaps above.

Are authenticated proxies supported?

Ftp can be added, but purposely left
> out for now. The requirements being fulfilled are written here [2].
> * It has more features than the existing downloaders including additional ssl config, proxy_auth, additional
> connection options, and more.

Can you enumerate them?

> 
> # Easier for plugin writers
> * An easier download customization experience by having to subclass fewer objects than the existing
> downloaders. 

Can you be specific?

> This design has one downloader for both synchronous or asynchronous downloading

But they need to execute the downloader in an Asynio loop for both synchronous and asynchronous, correct?

> * Makes some functionality contained only in the changeset available in the downloaders directly. Specifically
> the GroupDownloader was functionality you could only get via changeset usage. Changesets could easily be
> retooled to use this instead which would be great.
> * Each object can be used independently or in conjunction with others
> 
> # Demo
> * works with the HEAD of master of pulp_example [3] (not pulp_file) which shows that it actually runs
> 
> [0]: https://github.com/pulp/pulp/pull/3129
> [1]: http://file.rdu.redhat.com/~bbouters/plugins/plugin-api/download.html
> [2]: https://pulp.plan.io/issues/2951#note-17
> [3]: https://github.com/dkliban/pulp_example/
> 
> -Brian
> 
> 
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170824/4def0dae/attachment.sig>