[Pulp-dev] artifact stages adjustment or any other solution?

[Brian Bouterse]

During the pulp_rpm meeting today @ttereshc identified that the on-demand
content concern I brought up is not an issue because modulemd is a metadata
file so it will always be downloaded and its Artifact saved regardless of
the on-demand policy. So that's a non-issue. :)

@mdellweg I will think more about what you are linking to applies and how
it related to what @asmacdo was saying. Thank you for sharing.

On Thu, Aug 29, 2019 at 11:11 AM Matthias Dellweg <dellweg at atix.de> wrote:

> I am not entirely sure, whether it is the same kind of problem, but in
> [0] there is an example of a stage that adds a declarative artifact in
> the middle of the pipeline.
>
> [0]
>
> https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/tasks/synchronizing.py#L204
>
> On Thu, 29 Aug 2019 10:42:33 -0400
> Brian Bouterse <bmbouter at redhat.com> wrote:
>
> > I agree Austin's suggestion of a mechanism like that sounds good. Can
> > a more detailed description be written in Redmine and sent to the
> > list? This change is in Stages API which has such broad impact that
> > I'd feel most comfortable if we could get to that level of detail
> > together somehow.
> >
> > The Stages API details I'm sure we can work out, but the concern I
> > keep thinking about has to do with on-demand content. An Artifact
> > that doesn't also have a single url to download it from couldn't have
> > a RemoteArtifact created for it. If we can't make a RemoteArtifact
> > for it, then it won't work with on-demand sync modes. To me that is
> > the more challenging aspect of this goal. Will this work with
> > on-demand content? Do others see this as an issue? What do you think?
> >
> > @ttereshc thank you for bringing this up, and @asmacdo thank you for
> > jumping in and suggesting.
> >
> >
> > On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko
> > <ttereshc at redhat.com> wrote:
> >
> > > Austin, thank you.
> > > Your option is more explicit, so it's probably better.
> > >
> > > Tanya
> > >
> > > On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald
> > > <amacdona at redhat.com> wrote:
> > >
> > >>
> > >>
> > >> On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko <
> > >> ttereshc at redhat.com> wrote:
> > >>
> > >>> Bump.
> > >>>
> > >>> Please provide feedback if you have any.
> > >>> I'll start working on the PR to make the suggested change this
> > >>> week otherwise.
> > >>>
> > >>> Thank you,
> > >>> Tanya
> > >>>
> > >>> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko <
> > >>> ttereshc at redhat.com> wrote:
> > >>>
> > >>>> In RPM plugin we have Modulemd content. It comes from metadata
> > >>>> as one file and we parse it and then save each modulemd as a
> > >>>> separate file/artifact.
> > >>>>
> > >>>> The question is how to handle this content in the sync pipeline.
> > >>>> Modulemd content is artifactless on a remote source (metadata)
> > >>>> but it's not artifactless in Pulp, so it can't follow a standard
> > >>>> path - it needs an artifact but doesn't have any remote source
> > >>>> to download it from.
> > >>>>
> > >>>> The suggestion: find a way to skip ArtifactDownloader and
> > >>>> RemoteArtifactSaver stages.
> > >>>> The content and its artifact still need to go through all the
> > >>>> stages, except the ones which deal with artifact's url in some
> > >>>> way - download artifact (ArtifactDownloader stage) and or create
> > >>>> a RemoteArtifact for downloading later (RemoteArtifactSaver
> > >>>> stage).
> > >>>>
> > >>>> The straightforward way is just to check if DeclarativeArtifact
> > >>>> has url (or check some special value) and skip the stage
> > >>>> otherwise. Any concerns about this approach (apart form being
> > >>>> somewhat hacky)?
> > >>>
> > >>>> Any other solutions to the problem?
> > >>>>
> > >>>
> > >> That solution seems fine to me. I'll toss out another idea just to
> > >> have options.
> > >>
> > >> New bool on Declarative Artifact. Similar to deferred downloads,
> > >> certain stages are no-ops if DeclarativeArtifact.deferred_download
> > >> is True.
> > >>
> > >>
> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152
> > >>
> > >>
> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29
> > >>
> > >>
> > >>
> > >>> Thank you,
> > >>>> Tanya
> > >>>>
> > >>>> P.S. FWIW, Reasons to store modulemd as a file are:
> > >>>> - the format is not very stable and new information can be added
> > >>>> - it can be large
> > >>>> - we don't need all the info in the DB, we use only small subset
> > >>>> of fields (to search by or for copy operations)
> > >>>>
> > >>> _______________________________________________
> > >>> Pulp-dev mailing list
> > >>> Pulp-dev at redhat.com
> > >>> https://www.redhat.com/mailman/listinfo/pulp-dev
> > >>>
> > >> _______________________________________________
> > > Pulp-dev mailing list
> > > Pulp-dev at redhat.com
> > > https://www.redhat.com/mailman/listinfo/pulp-dev
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190829/0852d8f1/attachment.htm>