[Pulp-dev] artifact stages adjustment or any other solution?

Matthias Dellweg dellweg at atix.de
Thu Aug 29 15:02:25 UTC 2019


I am not entirely sure, whether it is the same kind of problem, but in
[0] there is an example of a stage that adds a declarative artifact in
the middle of the pipeline.

[0]
https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/tasks/synchronizing.py#L204

On Thu, 29 Aug 2019 10:42:33 -0400
Brian Bouterse <bmbouter at redhat.com> wrote:

> I agree Austin's suggestion of a mechanism like that sounds good. Can
> a more detailed description be written in Redmine and sent to the
> list? This change is in Stages API which has such broad impact that
> I'd feel most comfortable if we could get to that level of detail
> together somehow.
> 
> The Stages API details I'm sure we can work out, but the concern I
> keep thinking about has to do with on-demand content. An Artifact
> that doesn't also have a single url to download it from couldn't have
> a RemoteArtifact created for it. If we can't make a RemoteArtifact
> for it, then it won't work with on-demand sync modes. To me that is
> the more challenging aspect of this goal. Will this work with
> on-demand content? Do others see this as an issue? What do you think?
> 
> @ttereshc thank you for bringing this up, and @asmacdo thank you for
> jumping in and suggesting.
> 
> 
> On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko
> <ttereshc at redhat.com> wrote:
> 
> > Austin, thank you.
> > Your option is more explicit, so it's probably better.
> >
> > Tanya
> >
> > On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald
> > <amacdona at redhat.com> wrote:
> >  
> >>
> >>
> >> On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko <  
> >> ttereshc at redhat.com> wrote:  
> >>  
> >>> Bump.
> >>>
> >>> Please provide feedback if you have any.
> >>> I'll start working on the PR to make the suggested change this
> >>> week otherwise.
> >>>
> >>> Thank you,
> >>> Tanya
> >>>
> >>> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko <  
> >>> ttereshc at redhat.com> wrote:  
> >>>  
> >>>> In RPM plugin we have Modulemd content. It comes from metadata
> >>>> as one file and we parse it and then save each modulemd as a
> >>>> separate file/artifact.
> >>>>
> >>>> The question is how to handle this content in the sync pipeline.
> >>>> Modulemd content is artifactless on a remote source (metadata)
> >>>> but it's not artifactless in Pulp, so it can't follow a standard
> >>>> path - it needs an artifact but doesn't have any remote source
> >>>> to download it from.
> >>>>
> >>>> The suggestion: find a way to skip ArtifactDownloader and
> >>>> RemoteArtifactSaver stages.
> >>>> The content and its artifact still need to go through all the
> >>>> stages, except the ones which deal with artifact's url in some
> >>>> way - download artifact (ArtifactDownloader stage) and or create
> >>>> a RemoteArtifact for downloading later (RemoteArtifactSaver
> >>>> stage).
> >>>>
> >>>> The straightforward way is just to check if DeclarativeArtifact
> >>>> has url (or check some special value) and skip the stage
> >>>> otherwise. Any concerns about this approach (apart form being
> >>>> somewhat hacky)? 
> >>>  
> >>>> Any other solutions to the problem?
> >>>>  
> >>>  
> >> That solution seems fine to me. I'll toss out another idea just to
> >> have options.
> >>
> >> New bool on Declarative Artifact. Similar to deferred downloads,
> >> certain stages are no-ops if DeclarativeArtifact.deferred_download
> >> is True.
> >>
> >> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152
> >>
> >> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29
> >>
> >>
> >>  
> >>> Thank you,  
> >>>> Tanya
> >>>>
> >>>> P.S. FWIW, Reasons to store modulemd as a file are:
> >>>> - the format is not very stable and new information can be added
> >>>> - it can be large
> >>>> - we don't need all the info in the DB, we use only small subset
> >>>> of fields (to search by or for copy operations)
> >>>>  
> >>> _______________________________________________
> >>> Pulp-dev mailing list
> >>> Pulp-dev at redhat.com
> >>> https://www.redhat.com/mailman/listinfo/pulp-dev
> >>>  
> >> _______________________________________________  
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190829/86e6e72d/attachment.sig>


More information about the Pulp-dev mailing list