[Pulp-dev] Importers/Exporters

[Ina Panova]

I am in favour of letting the core handle the workflow and provide the
RepositoryVersion.
I don't know if it in general makes sense to export a publication
especially when we'll get to the incremental export implementation.

I do think we should separate the Export and Exporter concepts:
- while exporting/importing repo version it makes sense to export only the
repo vesion, so we can then plugin it in.
- Exporter workflow ( rsync for example) will need to "export" the repo
version content with repodata, if available (publication) so when the task
finishes the content is already consumable. So I think exporter should stay
master/details because the presence/absence of the publication depending on
the plugin.

+1 to uuids

I am not sure (or maybe not following) if exporting complete dataset will
help in case other repo versions have been created in between full export
and incremental export.


--------
Regards,

Ina Panova
Senior Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."


On Fri, Feb 14, 2020 at 7:11 PM David Davis <daviddavis at redhat.com> wrote:

> Grant and I met today to discuss importers and exporters[0] and we'd like
> some feedback before we proceed with the design. To sum up this feature
> briefly: users can export a repository version from one Pulp instance and
> import it to another.
>
> # Master/Detail vs Core
>
> So one fundamental question is whether we should use a Master/Detail
> approach or just have core control the flow but call out to plugins to get
> export formats.
>
> To give some background: we currently define Exporters (ie
> FileSystemExporter) in core as Master models. Plugins extend this model
> which allows them to configure or customize the Exporter. This was
> necessary because some plugins need to export Publications (along with
> repository metadata) while other plugins who don't have Publications or
> metadata export RepositoryVersions.
>
> The other option is to have core handle the workflow. The user would call
> a core endpoint and provide a RepositoryVersion. This would work because
> for importing/exporting, you wouldn't ever use Publications because
> metadata won't be used for importing back into Pulp. If needed, core could
> provide a way for plugin writers to write custom handlers/exporters for
> content types.
>
> If we go with the second option, the question then becomes whether we
> should divorce the concept of Exporters and import/export. Or do we also
> switch Exporters from Master/Detail to core only?
>
> # Foreign Keys
>
> Content can be distributed across multiple tables (eg UpdateRecord has
> UpdateCollection, etc). In our export, we could either use primary keys
> (UUIDs) or natural keys to relate records. The former assumes that UUIDs
> are unique across Pulp instances. The safer but more complex alternative is
> to use natural keys. This would involve storing a set of fields on a record
> that would be used to identify a related record.
>
> # Incremental Exports
>
> There are two big pieces of data contained in an export: the dataset of
> Content from the database and the artifact files. An incremental export
> cuts down on the size of an export by only exporting the differences.
> However, when performing an incremental export, we could still export the
> complete dataset instead of just a set of differences
> (additions/removals/updates). This approach would be simpler and it would
> allow us to ensure that the new repo version matches the exported repo
> version exactly. It would however increase the export size but not by much
> I think--probably some number of megabytes at most.
>
> [0] https://pulp.plan.io/issues/6134
>
> David
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200217/8ea00191/attachment.htm>