<div dir="ltr"><div><div>No opinion on name; foreman will name it whatever they want on the front end user experience. Devs working on pulp-2 to pulp-3 foreman transition may desire maintaining existing names.<br><br></div>Yes, I'd say everything but step 14 in that diagram. In addition, I would ensure that the squid cache size is configurable to zero so that it is effectively a straight pull through.<br><br></div>I assume that all pulp-3 content types will have this as an option as well, if the type supports it? I want straight proxy of container images, for example. An straight proxy of files. etc.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 30, 2018 at 11:34 AM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Actually, what about these as names?</div><div><br></div><div>policy=immediate -> downloads now while the task runs (no lazy). Also the default if unspecified.</div><div>policy=cache-and-save -> All the steps in the diagram. Content that is downloaded is saved so that it's only ever downloaded once.<br></div><div>policy=cache -> All the steps in the diagram except step 14. If squid pushes the bits
out of the cache, it will be re-downloaded again to serve to other
clients requesting the same bits.<br></div><div><br></div><div>If ^ is better I can update the stories. Other naming ideas and use cases are welcome.<br></div><div><br></div><div>Thanks,</div><div>Brian<br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 30, 2018 at 10:50 AM, Brian Bouterse <span dir="ltr"><<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, May 30, 2018 at 8:57 AM, Tom McKay <span dir="ltr"><<a href="mailto:thomasmckay@redhat.com" target="_blank">thomasmckay@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I think there is a usecase for "proxy only" like is being described here. Several years ago there was a project called thumbslug[1] that was used in a version of katello instead of pulp. It's job was to check entitlements and then proxy content from a cdn. The same functionality could be implemented in pulp. (Perhaps it's even as simple as telling squid not to cache anything so the content would never make it from cache to pulp in current pulp-2.)</div></div></blockquote><div> </div></span><div>What would you call this policy?</div><div>policy=proxy?</div><div>policy=stream-dont-save?</div><div>policy=stream-no-save?<br></div><div><br></div><div>Are the names 'on-demand' and 'immediate' clear enough? Are there better names?<br></div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>Overall I'm +1 to the idea of an only-squid version, if others think it would be useful.<br></div></blockquote><div><br></div></span><div>I understand describing this as a "only-squid" version, but for clarity, the streamer would still be required because it is what requests the bits with the correctly configured downloader (certs, proxy, etc). The streamer streams the bits into squid which provides caching and client multiplexing.</div><div><br></div><div>To confirm my understanding this "squid-only" policy would be the same as on-demand except that it would *not* perform step 14 from the diagram here (<a href="https://pulp.plan.io/issues/3693" target="_blank">https://pulp.plan.io/issues/3<wbr>693</a>). Is that right?<br></div><div><div class="m_-1341307190864262055h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div><br>[1] <a href="https://github.com/candlepin/thumbslug" target="_blank">https://github.com/candlepin/t<wbr>humbslug</a><br></div></div><div class="m_-1341307190864262055m_202346191337334644m_2357111328823709421gmail-HOEnZb"><div class="m_-1341307190864262055m_202346191337334644m_2357111328823709421gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 30, 2018 at 8:34 AM, Milan Kovacik <span dir="ltr"><<a href="mailto:mkovacik@redhat.com" target="_blank">mkovacik@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Tue, May 29, 2018 at 9:31 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote:<br>
> On Tue, May 29, 2018 at 11:42 AM, Milan Kovacik <<a href="mailto:mkovacik@redhat.com" target="_blank">mkovacik@redhat.com</a>> wrote:<br>
>><br>
>> On Tue, May 29, 2018 at 5:13 PM, Dennis Kliban <<a href="mailto:dkliban@redhat.com" target="_blank">dkliban@redhat.com</a>> wrote:<br>
>> > On Tue, May 29, 2018 at 10:41 AM, Milan Kovacik <<a href="mailto:mkovacik@redhat.com" target="_blank">mkovacik@redhat.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Good point!<br>
>> >> More the second; it might be a bit crazy to utilize Squid for that but<br>
>> >> first, let's answer the why ;)<br>
>> >> So why does Pulp need to store the content here?<br>
>> >> Why don't we point the users to the Squid all the time (for the lazy<br>
>> >> repos)?<br>
>> ><br>
>> ><br>
>> > Pulp's Streamer needs to fetch and store the content because that's<br>
>> > Pulp's<br>
>> > primary responsibility.<br>
>><br>
>> Maybe not that much the storing but rather the content views management?<br>
>> I mean the partitioning into repositories, promoting.<br>
>><br>
><br>
> Exactly this. We want Pulp users to be able to reuse content that was<br>
> brought in using the 'on_demand' download policy in other repositories.<br>
</span>I see.<br>
<span><br>
><br>
>><br>
>> If some of the content lived in Squid and some lived<br>
>> > in Pulp, it would be difficult for the user to know what content is<br>
>> > actually<br>
>> > available in Pulp and what content needs to be fetched from a remote<br>
>> > repository.<br>
>><br>
>> I'd say the rule of the thumb would be: lazy -> squid, regular -> pulp<br>
>> so not that difficult.<br>
>> Maybe Pulp could have a concept of Origin, where folks upload stuff to<br>
>> a Pulp repo, vs. Proxy for it's repo storage policy?<br>
>><br>
><br>
> Squid removes things from the cache at some point. You can probably<br>
> configure it to never remove anything from the cache, but then we would need<br>
> to implement orphan cleanup that would work across two systems: pulp and<br>
> squid.<br>
<br>
</span>Actually "remote" units wouldn't need orphan cleaning from the disk,<br>
just dropping them from the DB would suffice.<br>
<span><br>
><br>
> Answering that question would still be difficult. Not all content that is in<br>
> the repository that was synced using on_demand download policy will be in<br>
> Squid - only the content that has been requested by clients. So it's still<br>
> hard to know which of the content units have been downloaded and which have<br>
> not been.<br>
<br>
</span>But the beauty is exactly in that: we don't have to track whether the<br>
content is downloaded if it is reverse-proxied[1][2].<br>
Moreover, this would work both with and without a proxy between Pulp<br>
and the Origin of the remote unit.<br>
A "remote" content artifact might just need to carry it's URL in a DB<br>
column for this to work; so the async artifact model, instead of the<br>
"policy=on-demand" would have a mandatory remote "URL" attribute; I<br>
wouldn't say it's more complex than tracking the "policy" attribute.<br>
<span><br>
><br>
><br>
>><br>
>> ><br>
>> > As Pulp downloads an Artifact, it calculates all the checksums and it's<br>
>> > size. It then performs validation based on information that was provided<br>
>> > from the RemoteArtifact. After validation is performed, the Artifact, is<br>
>> > saved to the database and it's final place in<br>
>> > /var/lib/content/artifacts/.<br>
>><br>
>> This could be still achieved by storing the content just temporarily<br>
>> in the Squid proxy i.e use Squid as the content source, not the disk.<br>
>><br>
>> > Once this information is in the database, Pulp's web server can serve<br>
>> > the<br>
>> > content without having to involve the Streamer or Squid.<br>
>><br>
>> Pulp might serve just the API and the metadata, the content might be<br>
>> redirected to the Proxy all the time, correct?<br>
>> Doesn't Crane do that btw?<br>
><br>
><br>
> Theoretically we could do this, but in practice we would run into problems<br>
> when we needed to scale out the Content app. Right now when the Content app<br>
> needs to be scaled, a user can launch another machine that will run the<br>
> Content app. Squid does not support that kind of scaling. Squid can only<br>
> take advantage of additional cores in a single machine<br>
<br>
</span>I don't think I understand; proxies are actually designed to scale[1]<br>
and are used as tools to scale the web too.<br>
<br>
This is all about the How question but when it comes to my original<br>
Why, please correct me if I'm being wrong, the answer so far has been:<br>
Pulp always downloads the content because that's what it is supposed to do.<br>
<br>
Cheers,<br>
milan<br>
<br>
[1] <a href="https://en.wikipedia.org/wiki/Reverse_proxy" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/<wbr>Reverse_proxy</a><br>
[2] <a href="https://paste.fedoraproject.org/paste/zkBTyxZjm330FsqvPP0lIA" rel="noreferrer" target="_blank">https://paste.fedoraproject.or<wbr>g/paste/zkBTyxZjm330FsqvPP0lIA</a><br>
[3] <a href="https://wiki.squid-cache.org/Features/CacheHierarchy?highlight=%28faqlisted.yes%29" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/F<wbr>eatures/CacheHierarchy?highlig<wbr>ht=%28faqlisted.yes%29</a><br>
<div class="m_-1341307190864262055m_202346191337334644m_2357111328823709421gmail-m_-552244167370227429HOEnZb"><div class="m_-1341307190864262055m_202346191337334644m_2357111328823709421gmail-m_-552244167370227429h5"><br>
><br>
>><br>
>><br>
>> Cheers,<br>
>> milan<br>
>><br>
>> ><br>
>> > -dennis<br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >> cheers<br>
>> >> milan<br>
>> >><br>
>> >> On Tue, May 29, 2018 at 4:25 PM, Brian Bouterse <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>><br>
>> >> wrote:<br>
>> >> ><br>
>> >> > On Mon, May 28, 2018 at 9:57 AM, Milan Kovacik <<a href="mailto:mkovacik@redhat.com" target="_blank">mkovacik@redhat.com</a>><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> Hi,<br>
>> >> >><br>
>> >> >> Looking at the diagram[1] I'm wondering what's the reasoning behind<br>
>> >> >> Pulp having to actually fetch the content locally?<br>
>> >> ><br>
>> >> ><br>
>> >> > Is the question "why is Pulp doing the fetching and not Squid?" or<br>
>> >> > "why<br>
>> >> > is<br>
>> >> > Pulp storing the content after fetching it?" or both?<br>
>> >> ><br>
>> >> >> Couldn't Pulp just rely on the proxy with regards to the content<br>
>> >> >> streaming?<br>
>> >> >><br>
>> >> >> Thanks,<br>
>> >> >> milan<br>
>> >> >><br>
>> >> >><br>
>> >> >> [1] <a href="https://pulp.plan.io/attachments/130957" rel="noreferrer" target="_blank">https://pulp.plan.io/attachmen<wbr>ts/130957</a><br>
>> >> >><br>
>> >> >> On Fri, May 25, 2018 at 9:11 PM, Brian Bouterse<br>
>> >> >> <<a href="mailto:bbouters@redhat.com" target="_blank">bbouters@redhat.com</a>><br>
>> >> >> wrote:<br>
>> >> >> > A mini-team of core devs** met to talk through lazy use cases for<br>
>> >> >> > Pulp3.<br>
>> >> >> > It's effectively the same lazy from Pulp2 except:<br>
>> >> >> ><br>
>> >> >> > * it's now built into core (not just RPM)<br>
>> >> >> > * It disincludes repo protection use cases because we haven't<br>
>> >> >> > added<br>
>> >> >> > repo<br>
>> >> >> > protection to Pulp3 yet<br>
>> >> >> > * It disincludes the "background" policy which based on feedback<br>
>> >> >> > from<br>
>> >> >> > stakeholders provided very little value<br>
>> >> >> > * it will no longer will depend on Twisted as a dependency. It<br>
>> >> >> > will<br>
>> >> >> > use<br>
>> >> >> > asyncio instead.<br>
>> >> >> ><br>
>> >> >> > While it is being built into core, it will require minimal support<br>
>> >> >> > by<br>
>> >> >> > a<br>
>> >> >> > plugin writer to add support for it. Details in the epic below.<br>
>> >> >> ><br>
>> >> >> > The current use cases along with a technical plan are written on<br>
>> >> >> > this<br>
>> >> >> > epic:<br>
>> >> >> > <a href="https://pulp.plan.io/issues/3693" rel="noreferrer" target="_blank">https://pulp.plan.io/issues/36<wbr>93</a><br>
>> >> >> ><br>
>> >> >> > We're putting it out for comment, questions, and feedback before<br>
>> >> >> > we<br>
>> >> >> > start<br>
>> >> >> > into the code. I hope we are able to add this into our next<br>
>> >> >> > sprint.<br>
>> >> >> ><br>
>> >> >> > ** ipanova, jortel, ttereshc, dkliban, bmbouter<br>
>> >> >> ><br>
>> >> >> > Thanks!<br>
>> >> >> > Brian<br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > ______________________________<wbr>_________________<br>
>> >> >> > Pulp-dev mailing list<br>
>> >> >> > <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
>> >> >> > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
>> >> >> ><br>
>> >> ><br>
>> >> ><br>
>> >><br>
>> >> ______________________________<wbr>_________________<br>
>> >> Pulp-dev mailing list<br>
>> >> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
>> >> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
>> ><br>
>> ><br>
><br>
><br>
<br>
______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/pulp-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>