<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<font face="DejaVu Sans">The pulp 3 alpha included two solutions for
providing downloading support in the plugin API. Each solution
was based on different concurrency technologies, protocol support
libs and abstractions. The goal was to test drive each, get
community feedback and make a second pass at documenting the <i>minimal</i>
requirements and characteristics deemed desirable for download
support in the API. This criteria appears at the end of this
email.<br>
<br>
Agreement has been reached to proceed with the </font><i><font
face="DejaVu Sans">asyncio</font></i><font face="DejaVu Sans">
based solution. The underlying asyncio/coroutine technology has
several advantages and none of the disadvantages imposed by python
threading. The abstractions in the proposed solution are
sufficient to meet the minimal criteria we've documented.<br>
<br>
Unless there are objections, the next steps will include writing
stories to:<br>
</font>
<ul>
<li>Remove the <i>futures</i> solution.</li>
<li>Move (promote) the <i>asyncio</i> solution to the <i>download</i>
package.<br>
</li>
<li>Update the ChangeSet to work with <i>asyncio</i> based
downloaders.</li>
<li>Update the File plugin to work with <i>asyncio</i>
downloaders.</li>
<li>Update the Example plugin imports.<br>
</li>
</ul>
<font face="DejaVu Sans"><br>
The feature set and a few characteristics of the <i>asyncio</i>
solution need (some) further discussion and may necessitate follow
up stories. But, this set of stories will get us most of the way
there.<br>
<br>
<br>
<br>
<br>
--------------------------------------------<br>
<br>
</font>
<div id="magicdomid218" class="ace-line"><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki"><u>Requirements</u><br>
</span></div>
<div id="magicdomid232" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">support </span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">concurrent
downloading by changesets</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid233" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">uses</span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki"> a </span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki i"><i>Downloader</i></span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid240" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">support </span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">concurrent
downloading by streamer</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid7" class="">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">uses
a </span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki i"><i>Downloader</i></span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid9" class="">
<ul class="list-bullet1">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">importer
provides the right, configured downloader</span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8"> (defined
interface)</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid10" class="">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">streamer
and changeset does not choose</span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid11" class=""><br>
</div>
<div id="magicdomid150" class="ace-line"><u><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki"></span></u><u><span
class="author-a-hi4bcroz65zz77zsss4yz73z8">Desirable</span></u><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki"><br>
</span></div>
<div id="magicdomid151" class="ace-line"><br>
</div>
<div id="magicdomid152" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Platform
supply downloading support for known protocols.</span></li>
</ul>
</div>
<div id="magicdomid187" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Support
concurrent downloading</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid188" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">single
file</span></li>
</ul>
</div>
<div id="magicdomid189" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">groups of
files</span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid449" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">HTTP
downloader</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid450" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">size
verification</span></li>
</ul>
</div>
<div id="magicdomid451" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">hash
verification</span></li>
</ul>
</div>
<div id="magicdomid452" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">use
the importer settings</span></li>
</ul>
</div>
<div id="magicdomid453" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">automatically
retry</span></li>
</ul>
</div>
<div id="magicdomid454" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">automatically
follow HTTP redirects</span></li>
</ul>
</div>
<div id="magicdomid455" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">raise
400+ HTTP responses as foreground errors</span></li>
</ul>
</div>
<div id="magicdomid456" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">e</span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8">xposes
underlying lib settings</span></li>
</ul>
</div>
<div id="magicdomid457" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">connection
pooling (concurrent)</span></li>
</ul>
</div>
<div id="magicdomid458" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">connection
reuse (concurrent)</span></li>
</ul>
</div>
<div id="magicdomid459" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">connection
keep-alive support (concurrent)</span></li>
</ul>
</div>
<div id="magicdomid460" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">extensible</span></li>
</ul>
</div>
</blockquote>
<blockquote>
<div id="magicdomid461" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">must
signal that headers are available prior to flowing data</span></li>
</ul>
</div>
<div id="magicdomid480" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">raises
fatal exceptions</span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid439" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">FILE
downloader</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid440" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">size
verification</span></li>
</ul>
</div>
<div id="magicdomid441" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">hash
verification</span></li>
</ul>
</div>
<div id="magicdomid442" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">use the
importer settings</span></li>
</ul>
</div>
<div id="magicdomid443" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">extensible</span></li>
</ul>
</div>
<div id="magicdomid445" class="ace-line">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">r</span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">aises
</span><span class="author-a-hi4bcroz65zz77zsss4yz73z8">fatal
</span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">Exceptions</span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid149" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">Factory
to build a downloader from the importer settings</span></li>
</ul>
</div>
<div id="magicdomid350" class="ace-line">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">L</span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">everage
library/language as much as possible</span></li>
</ul>
</div>
<br>
<div id="magicdomid256" class="ace-line"><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki i"><i>Downloader</i></span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8 i"><i> - </i></span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8">characteristics</span></div>
<div id="magicdomid13" class="">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Start the
download</span></li>
</ul>
</div>
<div id="magicdomid14" class="">
<ul class="list-bullet1">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Do
something with the downloaded content.</span></li>
</ul>
</div>
<blockquote>
<div id="magicdomid15" class="">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Write to
file</span></li>
</ul>
</div>
<div id="magicdomid16" class="">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Write to
output stream (Eg: streamer)</span></li>
</ul>
</div>
<div id="magicdomid17" class="">
<ul class="list-bullet2">
<li><span class="author-a-hi4bcroz65zz77zsss4yz73z8">Calc</span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">ulate
and </span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8">provide</span><span
class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki"> the</span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8"> size and
digests</span></li>
</ul>
</div>
</blockquote>
<div id="magicdomid18" class="">
<ul class="list-bullet1">
<li><span class="author-a-3z79zv1z71zz68zhz70zbntz79zlz80zki">Verify
size and/or digest(s) values</span><span
class="author-a-hi4bcroz65zz77zsss4yz73z8"> (requirement)</span></li>
</ul>
<p><br>
</p>
<p><br>
</p>
</div>
</body>
</html>