[Pulp-dev] migration tool for Pulp 3

[Kodiak Firesmith]

As a bare-Pulp (no Katello / Sat 6) user with.... customizations... I think
I'd really like to be able to get a config exporter tool for a host -> host
migration as I'll probably move to a new host at the same time and run both
side-by side while I work out how to re-do our authentication scheme which
is based on mod_auth_gssapi and Active Directory.

Ideally I'd be able to run a script to dump all my current
repo/user/permission/role/etc settings to a text table, then run another
script on a newly installed separate Pulp 3 server to import all those
configs, and SCP /var/lib/pulp to the new host.  It's about 700GB but at
10Gbe it shouldn't take too long to sync.

On Wed, Apr 19, 2017 at 9:25 AM, Patrick Creech <pcreech at redhat.com> wrote:

> On Tue, 2017-04-18 at 14:42 -0400, Dennis Kliban wrote:
> > Do we want to provide a tool for migrating from Pulp 2 to 3? If yes,
> then ...
>
> Yes, indubitably (imo).
>
> > Would the tool be able to migrate repository definitions and require the
> user to sync and upload
> > content to restore /var/lib/pulp/content?
>
> Jeff has stated that we have no changes in pulp3 for this location, but
> even if we did, we probably
> shouldn't touch anything in this directory if we don't have to.   We have
> no guaruntees of the IOPS
> or latency of the underlying storage device, and have seen pain around
> this before.
>
> > Would this tool support installing Pulp 3 along side Pulp 2 and
> performing a migration of database
> > and /var/lib/pulp/content?
>
> If we can do a same-machine upgrade alongside pulp 2, that would be ideal.
>
> > Would this tool be able to accept a mongodump of Pulp 2 MongoDB and a
> path to a copy of Pulp 2's
> > /var/lib/pulp directory and use that information to populate Pulp 3?
>
> As a simple matter of performance, this tool should probably operate
> outside of the ORM interface
> (at least for the initial dump/load).  Most databases have a optimized way
> to bulk import/export
> data.  Mongodump is probably a bad tool for this though, as it is a binary
> export of data for use
> with mongorestore.
>
> One strategy we can use for the database is to mongoexport pulp2 data to
> json or csv, then convert
> this output to a format recognized by postgresql's COPY command for our
> table structure in pulp3.
> This will help us limit our memory footprint for migration, and could be
> done in steps to limit our
> filesystem footprint.  It appears mongoexport has some advanced options to
> help us here as well.
> Ideally we could just mongoexport the data into the desired postgresql
> import csv format for the
> tables.
>
> As noted by Sean, we should work to ensure we have minimal data loss and
> migration pain from pulp2
> to pulp3, and if we have to modify our pulp3 data model to do such, then
> we should while striving
> for a balance.
>
>
> >
> > _______________________________________________
> > Pulp-dev mailing list
> > Pulp-dev at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-dev
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170419/fa23a036/attachment.htm>