[Pulp-dev] Importers/Exporters

Justin Sherrill jsherril at redhat.com
Wed Feb 19 15:28:24 UTC 2020

Distributions definitely do not need to be migrated (i see them more as 
configuration similar to importers, than i do as something that can be 

Publications i think would be a nice to have, but can be re-generated on 
the other side based on the repository version. While the publication 
won't be 100% exact (due to timestamps and ordering differences), i'm 
not sure that it matters.  The biggest drawback will just be that the 
importing side needs to be smarter, and it will take a little longer.    
I could see import/export of publications being an optimization or 
extension later on, but not necessary for a first pass.  If i'm missing 
some issue or use case, please do speak up!


On 2/17/20 1:05 PM, David Davis wrote:
> The user stories that Katello gave us (which I've entered into redmine 
> here[0]) don't mention publications or distributions. I will follow up 
> with Katello though.
> [0] https://pulp.plan.io/issues/6134
> David
> On Mon, Feb 17, 2020 at 12:49 PM Dennis Kliban <dkliban at redhat.com 
> <mailto:dkliban at redhat.com>> wrote:
>     On Fri, Feb 14, 2020 at 1:11 PM David Davis <daviddavis at redhat.com
>     <mailto:daviddavis at redhat.com>> wrote:
>         Grant and I met today to discuss importers and exporters[0]
>         and we'd like some feedback before we proceed with the design.
>         To sum up this feature briefly: users can export a repository
>         version from one Pulp instance and import it to another.
>     Exporting just repository versions is not sufficient for
>     reproducing a Pulp instance in an air gapped environment. Users
>     need to be able to use the "export" to populate a Pulp instance
>     without needing to create any publications and/or distributions
>     afterwards. What about letting users specify a repository to
>     export and then have pulpcore figure out which repository
>     versions, publications, distributions, content, metadata, and
>     artifacts need to be exported?
>         # Master/Detail vs Core
>         So one fundamental question is whether we should use a
>         Master/Detail approach or just have core control the flow but
>         call out to plugins to get export formats.
>         To give some background: we currently define Exporters (ie
>         FileSystemExporter) in core as Master models. Plugins extend
>         this model which allows them to configure or customize the
>         Exporter. This was necessary because some plugins need to
>         export Publications (along with repository metadata) while
>         other plugins who don't have Publications or metadata export
>         RepositoryVersions.
>         The other option is to have core handle the workflow. The user
>         would call a core endpoint and provide a RepositoryVersion.
>         This would work because for importing/exporting, you wouldn't
>         ever use Publications because metadata won't be used for
>         importing back into Pulp. If needed, core could provide a way
>         for plugin writers to write custom handlers/exporters for
>         content types.
>         If we go with the second option, the question then becomes
>         whether we should divorce the concept of Exporters and
>         import/export. Or do we also switch Exporters from
>         Master/Detail to core only?
>         # Foreign Keys
>         Content can be distributed across multiple tables (eg
>         UpdateRecord has UpdateCollection, etc). In our export, we
>         could either use primary keys (UUIDs) or natural keys to
>         relate records. The former assumes that UUIDs are unique
>         across Pulp instances. The safer but more complex alternative
>         is to use natural keys. This would involve storing a set of
>         fields on a record that would be used to identify a related
>         record.
>         # Incremental Exports
>         There are two big pieces of data contained in an export: the
>         dataset of Content from the database and the artifact files.
>         An incremental export cuts down on the size of an export by
>         only exporting the differences. However, when performing an
>         incremental export, we could still export the complete dataset
>         instead of just a set of differences
>         (additions/removals/updates). This approach would be simpler
>         and it would allow us to ensure that the new repo version
>         matches the exported repo version exactly. It would however
>         increase the export size but not by much I think--probably
>         some number of megabytes at most.
>         [0] https://pulp.plan.io/issues/6134
>         David
>         _______________________________________________
>         Pulp-dev mailing list
>         Pulp-dev at redhat.com <mailto:Pulp-dev at redhat.com>
>         https://www.redhat.com/mailman/listinfo/pulp-dev
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200219/3f424963/attachment.htm>

More information about the Pulp-dev mailing list