<div dir="ltr"><div>I agree Austin's suggestion of a mechanism like that sounds good. Can a more detailed description be written in Redmine and sent to the list? This change is in Stages API which has such broad impact that I'd feel most comfortable if we could get to that level of detail together somehow.<br></div><div><br></div><div>The Stages API details I'm sure we can work out, but the concern I keep thinking about has to do with on-demand content. An Artifact that doesn't also have a single url to download it from couldn't have a RemoteArtifact created for it. If we can't make a RemoteArtifact for it, then it won't work with on-demand sync modes. To me that is the more challenging aspect of this goal. Will this work with on-demand content? Do others see this as an issue? What do you think?</div><div><br></div><div>@ttereshc thank you for bringing this up, and @asmacdo thank you for jumping in and suggesting.<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko <<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Austin, thank you.</div><div>Your option is more explicit, so it's probably better.</div><div><br></div><div>Tanya<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald <<a href="mailto:amacdona@redhat.com" target="_blank">amacdona@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko <<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Bump.</div><div><br></div><div>Please provide feedback if you have any.</div><div>I'll start working on the PR to make the suggested change this week otherwise.</div><div><br></div><div>Thank you,</div><div>Tanya<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko <<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>In RPM plugin we have Modulemd content. It comes from metadata as one file and we parse it and then save each modulemd as a separate file/artifact.</div><div><br></div><div>The question is how to handle this content in the sync pipeline.</div><div>Modulemd content is artifactless on a remote source (metadata) but it's not artifactless in Pulp, so it can't follow a standard path - it needs an artifact but doesn't have any remote source to download it from.<br></div><div><br></div><div>The suggestion: find a way to skip ArtifactDownloader and RemoteArtifactSaver stages.</div><div>The content and its artifact still need to go through all the stages,
except the ones which deal with artifact's url in some way - download artifact (ArtifactDownloader stage) and or create a RemoteArtifact for downloading later (RemoteArtifactSaver stage).</div><div><br></div><div>The straightforward way is just to check if DeclarativeArtifact has url (or check some special value) and skip the stage otherwise. Any concerns about this approach (apart form being somewhat hacky)?</div></div></blockquote></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Any other solutions to the problem?</div></div></blockquote></div></blockquote></div><div class="gmail_quote"><br></div><div class="gmail_quote">That solution seems fine to me. I'll toss out another idea just to have options.<br></div><div class="gmail_quote"><br><div>New bool on Declarative Artifact. Similar to deferred downloads, certain stages are no-ops if DeclarativeArtifact.deferred_download is True.</div><div><a href="https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152" target="_blank">https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152</a></div><div><a href="https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29" target="_blank">https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29</a></div></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Thank you,</div><div>Tanya</div><div><br></div><div>P.S. FWIW, Reasons to store modulemd as a file are:<div> - the format is not very stable and new information can be added</div><div>- it can be large</div><div>- we don't need all the info in the DB, we use only small subset of fields (to search by or for copy operations)</div></div></div>
</blockquote></div>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote></div></div>
</blockquote></div>
_______________________________________________<br>
Pulp-dev mailing list<br>
<a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a><br>
</blockquote></div>