[Pulp-dev] migration tool for Pulp 3

[Patrick Creech]

On Tue, 2017-04-18 at 14:42 -0400, Dennis Kliban wrote:
> Do we want to provide a tool for migrating from Pulp 2 to 3? If yes, then ...

Yes, indubitably (imo).

> Would the tool be able to migrate repository definitions and require the user to sync and upload
> content to restore /var/lib/pulp/content?

Jeff has stated that we have no changes in pulp3 for this location, but even if we did, we probably
shouldn't touch anything in this directory if we don't have to.   We have no guaruntees of the IOPS
or latency of the underlying storage device, and have seen pain around this before.  

> Would this tool support installing Pulp 3 along side Pulp 2 and performing a migration of database
> and /var/lib/pulp/content?

If we can do a same-machine upgrade alongside pulp 2, that would be ideal.

> Would this tool be able to accept a mongodump of Pulp 2 MongoDB and a path to a copy of Pulp 2's
> /var/lib/pulp directory and use that information to populate Pulp 3?

As a simple matter of performance, this tool should probably operate outside of the ORM interface
(at least for the initial dump/load).  Most databases have a optimized way to bulk import/export
data.  Mongodump is probably a bad tool for this though, as it is a binary export of data for use
with mongorestore.  

One strategy we can use for the database is to mongoexport pulp2 data to json or csv, then convert
this output to a format recognized by postgresql's COPY command for our table structure in pulp3. 
This will help us limit our memory footprint for migration, and could be done in steps to limit our
filesystem footprint.  It appears mongoexport has some advanced options to help us here as well. 
Ideally we could just mongoexport the data into the desired postgresql import csv format for the
tables.

As noted by Sean, we should work to ensure we have minimal data loss and migration pain from pulp2
to pulp3, and if we have to modify our pulp3 data model to do such, then we should while striving
for a balance.


> 
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20170419/58abb91a/attachment.sig>