[Pulp-dev] artifact stages adjustment or any other solution?

[Brian Bouterse]

I agree Austin's suggestion of a mechanism like that sounds good. Can a
more detailed description be written in Redmine and sent to the list? This
change is in Stages API which has such broad impact that I'd feel most
comfortable if we could get to that level of detail together somehow.

The Stages API details I'm sure we can work out, but the concern I keep
thinking about has to do with on-demand content. An Artifact that doesn't
also have a single url to download it from couldn't have a RemoteArtifact
created for it. If we can't make a RemoteArtifact for it, then it won't
work with on-demand sync modes. To me that is the more challenging aspect
of this goal. Will this work with on-demand content? Do others see this as
an issue? What do you think?

@ttereshc thank you for bringing this up, and @asmacdo thank you for
jumping in and suggesting.


On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko <ttereshc at redhat.com>
wrote:

> Austin, thank you.
> Your option is more explicit, so it's probably better.
>
> Tanya
>
> On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald <amacdona at redhat.com>
> wrote:
>
>>
>>
>> On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko <
>> ttereshc at redhat.com> wrote:
>>
>>> Bump.
>>>
>>> Please provide feedback if you have any.
>>> I'll start working on the PR to make the suggested change this week
>>> otherwise.
>>>
>>> Thank you,
>>> Tanya
>>>
>>> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko <
>>> ttereshc at redhat.com> wrote:
>>>
>>>> In RPM plugin we have Modulemd content. It comes from metadata as one
>>>> file and we parse it and then save each modulemd as a separate
>>>> file/artifact.
>>>>
>>>> The question is how to handle this content in the sync pipeline.
>>>> Modulemd content is artifactless on a remote source (metadata) but it's
>>>> not artifactless in Pulp, so it can't follow a standard path - it needs an
>>>> artifact but doesn't have any remote source to download it from.
>>>>
>>>> The suggestion: find a way to skip ArtifactDownloader and
>>>> RemoteArtifactSaver stages.
>>>> The content and its artifact still need to go through all the stages,
>>>> except the ones which deal with artifact's url in some way - download
>>>> artifact (ArtifactDownloader stage) and or create a RemoteArtifact for
>>>> downloading later (RemoteArtifactSaver stage).
>>>>
>>>> The straightforward way is just to check if DeclarativeArtifact has url
>>>> (or check some special value) and skip the stage otherwise. Any concerns
>>>> about this approach (apart form being somewhat hacky)?
>>>>
>>>
>>>> Any other solutions to the problem?
>>>>
>>>
>> That solution seems fine to me. I'll toss out another idea just to have
>> options.
>>
>> New bool on Declarative Artifact. Similar to deferred downloads, certain
>> stages are no-ops if DeclarativeArtifact.deferred_download is True.
>>
>> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152
>>
>> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29
>>
>>
>>
>>> Thank you,
>>>> Tanya
>>>>
>>>> P.S. FWIW, Reasons to store modulemd as a file are:
>>>> - the format is not very stable and new information can be added
>>>> - it can be large
>>>> - we don't need all the info in the DB, we use only small subset of
>>>> fields (to search by or for copy operations)
>>>>
>>> _______________________________________________
>>> Pulp-dev mailing list
>>> Pulp-dev at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20190829/cfe07104/attachment.htm>