[Pulp-list] 2to3-migration takes a long time

[Winberg Adam]

'not' = now


________________________________
From: pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on behalf of Winberg Adam <Adam.Winberg at smhi.se>
Sent: 25 October 2020 15:59
To: Daniel Alley
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] 2to3-migration takes a long time



have tested not with 0.5.0 and it did indeed fix the problem. The initial migrations was way faster, about 2.5hrs, and the subsequent migrations are really fast.

Thanks!

//Adam

________________________________
From: Daniel Alley <dalley at redhat.com>
Sent: 23 October 2020 19:02
To: Winberg Adam
Cc: pulp-list at redhat.com
Subject: Re: [Pulp-list] 2to3-migration takes a long time

No, this is with 0.4.0, since 0.5.0 is quite new I haven't updated yet. Will do so and try again.

If it's 0.4.0, then this is actually not surprising after all.  This was a bug that was fixed in 0.5.0

https://pulp.plan.io/issues/7280

On Fri, Oct 23, 2020 at 12:45 PM Winberg Adam <Adam.Winberg at smhi.se<mailto:Adam.Winberg at smhi.se>> wrote:

> This is with the very latest version of the migration plugin (0.5.0)?

No, this is with 0.4.0, since 0.5.0 is quite new I haven't updated yet. Will do so and try again.

> Is it actually equally long, or just not 'reasonably' faster? Is it only the "migrating rpm content to Pulp 3 erratum" sub-item which reports doing a lot of work or are many of them doing so?
I would say that the runtime is very similar. And it looks like many if not all sub-items are repeating their work but since the 'pulp3 erratum' is the longest running that's the one that stands out. Here are some progress reports on a subsequent run which should only resulted in a handful of new rpm's:
    {
      "code": "migrating.rpm.content",
      "done": 32348,
      "message": "Migrating rpm content to Pulp 3 rpm",
      "state": "completed",
      "suffix": null,
      "total": 32348
    },
    {
      "code": "premigrating.content.general",
      "done": 89690,
      "message": "Pre-migrating Pulp 2 ERRATUM content (general info)",
      "state": "completed",
      "suffix": null,
      "total": 89690
    },
    {
      "code": "premigrating.content.general",
      "done": 39,
      "message": "Pre-migrating Pulp 2 RPM content (general info)",
      "state": "completed",
      "suffix": null,
      "total": 43
    },
    {
      "code": "migrating.rpm.content",
      "done": 89690,
      "message": "Migrating rpm content to Pulp 3 erratum",
      "state": "completed",
      "suffix": null,
      "total": 89690
    },

Should these numbers be 0 on a subsequent run if nothing has changed?

//Adam


________________________________
From: Daniel Alley <dalley at redhat.com<mailto:dalley at redhat.com>>
Sent: 23 October 2020 17:33
To: Winberg Adam
Cc: pulp-list at redhat.com<mailto:pulp-list at redhat.com>
Subject: Re: [Pulp-list] 2to3-migration takes a long time

Hey Adam,

It's not expected for subsequent migrations to take equally long, that is strange and concerning.  This is with the very latest version of the migration plugin (0.5.0)?  Is it actually equally long, or just not 'reasonably' faster? Is it only the "migrating rpm content to Pulp 3 erratum" sub-item which reports doing a lot of work or are many of them doing so?

I do have some good news though, the 6-7 hour runtime will improve significantly once you can upgrade to 3.7.2 (which doesn't look like it has been pushed to the Foreman repositories yet).  There was a significant performance regression introduced by 3.7.0 which has been fixed, and in my testing migrations only took a bit more than 1/3 as long as they had been taking previously.

On Fri, Oct 23, 2020 at 10:47 AM Winberg Adam <Adam.Winberg at smhi.se<mailto:Adam.Winberg at smhi.se>> wrote:

Depending on the size of your pulp2 installation I understand that the 2to3migration can take quite some time. In my environment it takes approx. 6-7hrs. However, I expect consecutive migrations to be faster, based on the documentation:

"When you are ready to switch to Pulp 3: run migration, then stop Pulp 2 services (so no new data is coming in), run migration for the last time (it should not take long)."


But all my migration runs are equally long, the 'sub-task' "Migrating rpm content to Pulp 3 erratum" processes about 90000 items every time which takes a long time. Is this expected behaviour?


Thanks,

Adam

_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201025/47177179/attachment.htm>