[Pulp-list] Long upgrade times from 2.6 -> 2.8

Ashby, Jason (IMS) AshbyJ at imsweb.com
Sun Jul 3 12:25:33 UTC 2016


Yeah our content is on NFS mounts backed by regular non-SSD disks, so IO is pretty crappy.  But its cheap and perfect for serving yum content.

My recommendation would be having pulp-manage-db spit out a message before each big migration that says “this next migration is a big deal!” that would be a nice fyi.  And perhaps a URL to see migration details. This doesn’t prep people ahead of time to schedule downtime, but at least keeps them from thinking something is wrong.

From: pulp-list-bounces at redhat.com [mailto:pulp-list-bounces at redhat.com] On Behalf Of Michael Hrivnak
Sent: Friday, July 01, 2016 4:19 PM
To: Eric Helms <ehelms at redhat.com>
Cc: pulp-list <pulp-list at redhat.com>
Subject: Re: [Pulp-list] Long upgrade times from 2.6 -> 2.8

I expect the performance will be highly dependent on disk access speed. Latency is far more important than throughput for those two migrations. Deployments using NFS for example may benefit from finding ways to reduce latency during the migration; there may be mount options that can be adjusted only while the migration runs, or perhaps the code itself can be run on a machine closer to the data than where pulp usually runs.

Michael

On Fri, Jul 1, 2016 at 11:53 AM, Eric Helms <ehelms at redhat.com<mailto:ehelms at redhat.com>> wrote:
My main worry or first check I wanted to put to rest was that it was an actual issue. Given that everything is sane, I think warning users that upgrades will scale with the amount of content and giving them some rough estimates based on other users data is the best option. That way they can at least prepare themselves with knowledge for planning their upgrade outages.

Thanks for looking into this so quickly,
Eric

On Fri, Jul 1, 2016 at 11:50 AM, Brian Bouterse <bbouters at redhat.com<mailto:bbouters at redhat.com>> wrote:
With 2.8 specifically there is a *lot* of work done by the migrations and the runtime should scale with the content size. In other words I don't think it's doing useless work, there is just a lot of it.

I just did a quick audit of the two major offending migrations and they are implemented sanely so I don't think there is a quick fix to improve the runtime.

I believe to get a shorter runtime we would have to parallelize the migrations.

Regarding things going wrong, we do assume that migrations should be re-entrant meaning that if for some reason something strange occurs it should always be safe to run pulp-manage-db and it will pick up where it left off. Just a behavioral FYI.

These migration with 2.8 should not be norm, so I don't expect this to be very common in the future until maybe a major release (3.0).

Given all ^, what do you think we should do?

-Brian

On 07/01/2016 11:23 AM, Eric Helms wrote:
I think there are a couple of considerations.

 1) The first issue is that a 6-18 hour upgrade window is not something
users expect and we've not been warning them of such so they can plan an
outage accordingly. Lengthy upgrades also have that tendency to make
users feel something is wrong or increase the risk that something can go
wrong in between.
 2) The fundamental question of - is this a bug or does this make
perfect sense and how it has to work?
 3) Applying the upgrade on an existing 2.6 if it changed nothing of the
environment could work, the tough part is having to distribute that
backwards. Pulp would have to distribute it back to 2.6, and Katello
would have to push out patches to our 2.4 release channel.

Eric

On Fri, Jul 1, 2016 at 11:14 AM, Brian Bouterse <bbouters at redhat.com<mailto:bbouters at redhat.com>
<mailto:bbouters at redhat.com<mailto:bbouters at redhat.com>>> wrote:

    I'm trying to understand if the pain point is related to downtime or
    total runtime.

    For instance, what if these migration could be run as a
    pre-migration step, ahead of time while Pulp was still online? The
    upgrade would still take just as long but you could use your (in
    this case) 2.6 install normally while the migrations are applying.
    Once they are done then the actual upgrade of the codebase could be
    very short.

    -Brian

    On 07/01/2016 09:20 AM, Eric Helms wrote:



        On Fri, Jul 1, 2016 at 8:52 AM, Ashby, Jason (IMS)
        <AshbyJ at imsweb.com<mailto:AshbyJ at imsweb.com> <mailto:AshbyJ at imsweb.com<mailto:AshbyJ at imsweb.com>>
        <mailto:AshbyJ at imsweb.com<mailto:AshbyJ at imsweb.com> <mailto:AshbyJ at imsweb.com<mailto:AshbyJ at imsweb.com>>>> wrote:

            FWIW I just upgraded from 2.7 -> 2.8 and it was approx. 1-2 hr
            upgrade to get through the migrations in pulp-manage-db.____

            __ __

            290 GB /var/lib/pulp____

            16 GB MongoDB____

            __ __

            *From:*pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>
        <mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>>
            <mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>
        <mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>>>
            [mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>
        <mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>>
            <mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>
        <mailto:pulp-list-bounces at redhat.com<mailto:pulp-list-bounces at redhat.com>>>] *On Behalf Of *Michael
        Hrivnak
            *Sent:* Friday, July 01, 2016 8:31 AM
            *To:* Eric Helms <ehelms at redhat.com<mailto:ehelms at redhat.com>
        <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>> <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>
        <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>>>>
            *Cc:* pulp-list <pulp-list at redhat.com<mailto:pulp-list at redhat.com>
        <mailto:pulp-list at redhat.com<mailto:pulp-list at redhat.com>> <mailto:pulp-list at redhat.com<mailto:pulp-list at redhat.com>
        <mailto:pulp-list at redhat.com<mailto:pulp-list at redhat.com>>>>
            *Subject:* Re: [Pulp-list] Long upgrade times from 2.6 ->
        2.8____

            __ __

            Did you get any feedback on whether one particular migration
        seemed
            to be running for a lot of that time?


        For the 1.5TB/100GB MongoDB scenario here is what I am able to glean
        from user logs (which I can share privately with anyone debugging):

        ~5 hours: Applying pulp_puppet.plugins.migrations version 4
        ~10 hours: Applying pulp_rpm.plugins.migrations version 28

        Use reports "lots of stating, unlinking, and linking of all the
        symlinks
        in /var/lib/pulb" if that helps.

        Another user reports ~6 hours on 176G of data.

        Eric


            ____

            __ __

            Michael____

            __ __

            On Fri, Jul 1, 2016 at 7:23 AM, Eric Helms
        <ehelms at redhat.com<mailto:ehelms at redhat.com> <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>>
            <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com> <mailto:ehelms at redhat.com<mailto:ehelms at redhat.com>>>>
        wrote:____

                Howdy,____

                __ __

                We (Katello) have had users reporting incredibly long
        upgrade
                times when upgrading from 2.6 to 2.8. This occurs during the
                pulp-manage-db step that is run as the beginning of our
                installers upgrade process. Based on the numbers below, does
                this make sense at all?____

                __ __

                Some numbers:____

                __ __

                18 hour upgrade____

                1.5 TB /var/lib/pulp____

                100GB MongoDB____

                __ __

                __ __

                Thanks,____

                Eric____


                _______________________________________________
                Pulp-list mailing list
                Pulp-list at redhat.com<mailto:Pulp-list at redhat.com> <mailto:Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>>
        <mailto:Pulp-list at redhat.com<mailto:Pulp-list at redhat.com> <mailto:Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>>>
                https://www.redhat.com/mailman/listinfo/pulp-list____

            __ __



        ------------------------------------------------------------------------

            Information in this e-mail may be confidential. It is
        intended only
            for the addressee(s) identified above. If you are not the
            addressee(s), or an employee or agent of the addressee(s),
        please
            note that any dissemination, distribution, or copying of this
            communication is strictly prohibited. If you have received this
            e-mail in error, please notify the sender of the error.




        _______________________________________________
        Pulp-list mailing list
        Pulp-list at redhat.com<mailto:Pulp-list at redhat.com> <mailto:Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>>
        https://www.redhat.com/mailman/listinfo/pulp-list


    _______________________________________________
    Pulp-list mailing list
    Pulp-list at redhat.com<mailto:Pulp-list at redhat.com> <mailto:Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>>
    https://www.redhat.com/mailman/listinfo/pulp-list



_______________________________________________
Pulp-list mailing list
Pulp-list at redhat.com<mailto:Pulp-list at redhat.com>
https://www.redhat.com/mailman/listinfo/pulp-list


________________________________

Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20160703/c93a4d89/attachment.htm>


More information about the Pulp-list mailing list