[Pulp-list] Long upgrade times from 2.6 -> 2.8
Ashby, Jason (IMS)
AshbyJ at imsweb.com
Fri Jul 1 15:29:52 UTC 2016
The pain point for me was that I first thought pulp-manage-db was hung up, but then did some Googling and saw a post re: one or two specific migrations that were expected to be slow based on amount of pulp data.
As a user, I‘d like to see a progress bar, even a "dumb" progress indicator, e.g. spit out a dot every couple seconds, to see that the migration hasn't frozen would be hugely beneficial.
From: pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com] On Behalf Of Eric Helms
Sent: Friday, July 01, 2016 11:23 AM
To: Brian Bouterse <bbouters at redhat.com>
Cc: pulp-list <pulp-list at redhat.com>
Subject: Re: [Pulp-list] Long upgrade times from 2.6 -> 2.8
I think there are a couple of considerations.
1) The first issue is that a 6-18 hour upgrade window is not something users expect and we've not been warning them of such so they can plan an outage accordingly. Lengthy upgrades also have that tendency to make users feel something is wrong or increase the risk that something can go wrong in between.
2) The fundamental question of - is this a bug or does this make perfect sense and how it has to work?
3) Applying the upgrade on an existing 2.6 if it changed nothing of the environment could work, the tough part is having to distribute that backwards. Pulp would have to distribute it back to 2.6, and Katello would have to push out patches to our 2.4 release channel.
Eric
On Fri, Jul 1, 2016 at 11:14 AM, Brian Bouterse <bbouters at redhat.com<mailto:bbouters at redhat.com>> wrote:
I'm trying to understand if the pain point is related to downtime or total runtime.
For instance, what if these migration could be run as a pre-migration step, ahead of time while Pulp was still online? The upgrade would still take just as long but you could use your (in this case) 2.6 install normally while the migrations are applying. Once they are done then the actual upgrade of the codebase could be very short.
-Brian
On 07/01/2016 09:20 AM, Eric Helms wrote:
On Fri, Jul 1, 2016 at 8:52 AM, Ashby, Jason (IMS) <AshbyJ at imsweb.com<mailto:AshbyJ at imsweb.com>
<mailto:AshbyJ at imsweb.com<mailto:AshbyJ at imsweb.com>>> wrote:
FWIW I just upgraded from 2.7 -> 2.8 and it was approx. 1-2 hr
upgrade to get through the migrations in pulp-manage-db.____
__ __
290 GB /var/lib/pulp____
16 GB MongoDB____
__ __
*From:*pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>
<mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>>
[mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>
<mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>>] *On Behalf Of *Michael Hrivnak
*Sent:* Friday, July 01, 2016 8:31 AM
*To:* Eric Helms <ehelms at redhat.com<mailto:ehelms at redhat.com> <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>>>
*Cc:* pulp-list <pulp-list at redhat.com<mailto:pulp-list at redhat.com> <mailto:pulp-list at redhat.com<mailto:pulp-list at redhat.com>>>
*Subject:* Re: [Pulp-list] Long upgrade times from 2.6 -> 2.8____
__ __
Did you get any feedback on whether one particular migration seemed
to be running for a lot of that time?
For the 1.5TB/100GB MongoDB scenario here is what I am able to glean
from user logs (which I can share privately with anyone debugging):
~5 hours: Applying pulp_puppet.plugins.migrations version 4
~10 hours: Applying pulp_rpm.plugins.migrations version 28
Use reports "lots of stating, unlinking, and linking of all the symlinks
in /var/lib/pulb" if that helps.
Another user reports ~6 hours on 176G of data.
Eric
____
__ __
Michael____
__ __
On Fri, Jul 1, 2016 at 7:23 AM, Eric Helms <ehelms at redhat.com<mailto:ehelms at redhat.com>
<mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>>> wrote:____
Howdy,____
__ __
We (Katello) have had users reporting incredibly long upgrade
times when upgrading from 2.6 to 2.8. This occurs during the
pulp-manage-db step that is run as the beginning of our
installers upgrade process. Based on the numbers below, does
this make sense at all?____
__ __
Some numbers:____
__ __
18 hour upgrade____
1.5 TB /var/lib/pulp____
100GB MongoDB____
__ __
__ __
Thanks,____
Eric____
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com> <mailto:Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>>
https://www.redhat.com/mailman/listinfo/pulp-list____
__ __
------------------------------------------------------------------------
Information in this e-mail may be confidential. It is intended only
for the addressee(s) identified above. If you are not the
addressee(s), or an employee or agent of the addressee(s), please
note that any dissemination, distribution, or copying of this
communication is strictly prohibited. If you have received this
e-mail in error, please notify the sender of the error.
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list
________________________________
Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20160701/6eccb8aa/attachment.htm>
More information about the Pulp-list
mailing list