<div dir="ltr"><div>During the pulp_rpm meeting today @ttereshc identified that the on-demand content concern I brought up is not an issue because modulemd is a metadata file so it will always be downloaded and its Artifact saved regardless of the on-demand policy. So that's a non-issue. :) </div><div> </div><div>@mdellweg I will think more about what you are linking to applies and how it related to what @asmacdo was saying. Thank you for sharing. </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 29, 2019 at 11:11 AM Matthias Dellweg <<a href="mailto:dellweg@atix.de">dellweg@atix.de</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I am not entirely sure, whether it is the same kind of problem, but in [0] there is an example of a stage that adds a declarative artifact in the middle of the pipeline. [0] <a href="https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/tasks/synchronizing.py#L204" rel="noreferrer" target="_blank">https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/tasks/synchronizing.py#L204</a> On Thu, 29 Aug 2019 10:42:33 -0400 Brian Bouterse <<a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a>> wrote: > I agree Austin's suggestion of a mechanism like that sounds good. Can > a more detailed description be written in Redmine and sent to the > list? This change is in Stages API which has such broad impact that > I'd feel most comfortable if we could get to that level of detail > together somehow. > > The Stages API details I'm sure we can work out, but the concern I > keep thinking about has to do with on-demand content. An Artifact > that doesn't also have a single url to download it from couldn't have > a RemoteArtifact created for it. If we can't make a RemoteArtifact > for it, then it won't work with on-demand sync modes. To me that is > the more challenging aspect of this goal. Will this work with > on-demand content? Do others see this as an issue? What do you think? > > @ttereshc thank you for bringing this up, and @asmacdo thank you for > jumping in and suggesting. > > > On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko > <<a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote: > > > Austin, thank you. > > Your option is more explicit, so it's probably better. > > > > Tanya > > > > On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald > > <<a href="mailto:amacdona@redhat.com" target="_blank">amacdona@redhat.com</a>> wrote: > > > >> > >> > >> On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko < > >> <a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote: > >> > >>> Bump. > >>> > >>> Please provide feedback if you have any. > >>> I'll start working on the PR to make the suggested change this > >>> week otherwise. > >>> > >>> Thank you, > >>> Tanya > >>> > >>> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko < > >>> <a href="mailto:ttereshc@redhat.com" target="_blank">ttereshc@redhat.com</a>> wrote: > >>> > >>>> In RPM plugin we have Modulemd content. It comes from metadata > >>>> as one file and we parse it and then save each modulemd as a > >>>> separate file/artifact. > >>>> > >>>> The question is how to handle this content in the sync pipeline. > >>>> Modulemd content is artifactless on a remote source (metadata) > >>>> but it's not artifactless in Pulp, so it can't follow a standard > >>>> path - it needs an artifact but doesn't have any remote source > >>>> to download it from. > >>>> > >>>> The suggestion: find a way to skip ArtifactDownloader and > >>>> RemoteArtifactSaver stages. > >>>> The content and its artifact still need to go through all the > >>>> stages, except the ones which deal with artifact's url in some > >>>> way - download artifact (ArtifactDownloader stage) and or create > >>>> a RemoteArtifact for downloading later (RemoteArtifactSaver > >>>> stage). > >>>> > >>>> The straightforward way is just to check if DeclarativeArtifact > >>>> has url (or check some special value) and skip the stage > >>>> otherwise. Any concerns about this approach (apart form being > >>>> somewhat hacky)? > >>> > >>>> Any other solutions to the problem? > >>>> > >>> > >> That solution seems fine to me. I'll toss out another idea just to > >> have options. > >> > >> New bool on Declarative Artifact. Similar to deferred downloads, > >> certain stages are no-ops if DeclarativeArtifact.deferred_download > >> is True. > >> > >> <a href="https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152" rel="noreferrer" target="_blank">https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152</a> > >> > >> <a href="https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29" rel="noreferrer" target="_blank">https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29</a> > >> > >> > >> > >>> Thank you, > >>>> Tanya > >>>> > >>>> P.S. FWIW, Reasons to store modulemd as a file are: > >>>> - the format is not very stable and new information can be added > >>>> - it can be large > >>>> - we don't need all the info in the DB, we use only small subset > >>>> of fields (to search by or for copy operations) > >>>> > >>> _______________________________________________ > >>> Pulp-dev mailing list > >>> <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> > >>> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> > >>> > >> _______________________________________________ > > Pulp-dev mailing list > > <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> > > <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> _______________________________________________ Pulp-dev mailing list <a href="mailto:Pulp-dev@redhat.com" target="_blank">Pulp-dev@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-dev" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-dev</a> </blockquote></div>