[Pulp-list] Migration failure, 2.16 -> 2.18

Mon Apr 1 19:19:43 UTC 2019

Kodiak and I were able to resolve the issue by applying the patch found
here[0].

[0] https://pulp.plan.io/issues/4617#note-1

On Mon, Apr 1, 2019 at 1:38 PM Kodiak Firesmith <kfiresmith at gmail.com>
wrote:

> Hi Dennis,
> Sadly the RHEL 8 beta yum metadata appears to be in good order as a
> properly extensioned gz file with compressed data.  I went ahead and ran a
> quick tree of the /var/lib/pulp/content/units/yum_repo_metadata_file/
> path(*) to show how there's an odd mix of gzip and non-gzip metadata.  If
> you have any ideas on how to inject a debug option that prints the file
> it's working on during the migration, I'd love to shim that in.  I started
> poking around looking for the failed unit code
> in /usr/lib/python2.7/site-packages/pulp/server/db/migrations but came up
> empty.
>
> Please also feel free to ping me on #pulp  (I just hopped on)  if you have
> any questions that you wanted less lag on response.  Sadly I have a
> 14:00-15:00 but will be back after that.
>
> (*)  https://paste.fedoraproject.org/paste/QEa8YPiB5O3mMGot2wo-gQ
>
> Thanks!
>  - Kodiak
>
> On Mon, Apr 1, 2019 at 1:31 PM Dennis Kliban <dkliban at redhat.com> wrote:
>
>> I think that RHEL beta repos just had a 'modules' file or 'modules.yaml'
>> . If you find that file, compress it and try running the migration again.
>> Please reply on list with your findings.
>>
>> On Mon, Apr 1, 2019 at 12:54 PM Kodiak Firesmith <kfiresmith at gmail.com>
>> wrote:
>>
>>> Thanks very much for this lead.  This is probably something that will be
>>> good for everyone to know about as RHEL 8 beta includes modular
>>> repositories.  In my case, the only modular yum metadata was RHEL 8 and was
>>> in fact a gzip file (so sayeth the 'file' command and from head $file it
>>> appeared to be binary).
>>>
>>> I went ahead and traversed
>>> /var/lib/pulp/content/units/yum_repo_metadata_file -type f -name *.gz, fed
>>> them into the file command, and they all came back as compressed files.
>>>
>>> Should I run 'find' in any other paths by chance?  Any other .gz files
>>> seem to be symlinks back to the file in /var/lib/pulp/content/units :(
>>>
>>> Thanks!
>>>  - Kodiak
>>>
>>> On Mon, Apr 1, 2019 at 12:35 PM Dennis Kliban <dkliban at redhat.com>
>>> wrote:
>>>
>>>> I did not provide enough information. This is only happening to your
>>>> Fedora 28+ modular repositories. The file that you are looking for is
>>>> supposed to be called modules.yaml.gz. It will have a hash prepended to it.
>>>> So the name will look more similar to
>>>> 256554a6d4e29b3ab128655ad4e72932765cc7e2da3192199efd11ae2cb7cf7a-modules.yaml.gz.
>>>> You want to find all these files and make sure that they are actually
>>>> gzipped.
>>>>
>>>> On Mon, Apr 1, 2019 at 12:01 PM Kodiak Firesmith <kfiresmith at gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Dennis,
>>>>> I'm not sure I follow :(
>>>>> I'm guessing since there are a hundred YUM repos being served, there
>>>>> are YUM metadata files scattered all over the various symlink trees.  Do
>>>>> you know which files I should be looking for, and if it's safe to just gzip
>>>>> them all in place?
>>>>> Apologies if this is basic stuff.  I'm moving my priorities around to
>>>>> do a better job of twinning our beta and prod environment because it sounds
>>>>> like the bug is hitting something that might be out of the norm due to one
>>>>> of the repos that I'm syncing on prod but not on beta.
>>>>>
>>>>>  - Kodiak
>>>>>
>>>>> On Mon, Apr 1, 2019 at 11:57 AM Dennis Kliban <dkliban at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> I was investigating a somewhat related problem and discovered that
>>>>>> when Pulp is downloading files that was encoded using gzip, Pulp decodes
>>>>>> the file and writes it to disk in decoded form.
>>>>>>
>>>>>> You can search for all the yum metadata files and make sure that they
>>>>>> are gzipped. Then try migrating again.
>>>>>>
>>>>>> On Mon, Apr 1, 2019 at 7:59 AM Kodiak Firesmith <kfiresmith at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Folks,
>>>>>>> I have a beta and a prod pulp server, beta is much smaller but
>>>>>>> otherwise I keep them identical.  Last week I updated the beta host from
>>>>>>> 2.16 to latest without any issues.  Today I attempted to upgrade prod in
>>>>>>> the same way and ended up with a fatal migration error :(
>>>>>>>
>>>>>>> Any chance a Pulp dev can parse this traceback?  The final error
>>>>>>> that bombs the migration is 'IOError: Not a gzipped file', on migration
>>>>>>> unit 'pulp_rpm.plugins.migrations.0043_add_modulemd_modulemd-defaults'.
>>>>>>>
>>>>>>> https://paste.fedoraproject.org/paste/wEIu5a3Tf8OejEzMNsXN9w
>>>>>>>
>>>>>>>
>>>>>>> Packages installed are:
>>>>>>> $rpm -qa | egrep 'pulp|qpid|gofer|celery|mongo' | sort
>>>>>>> mongodb-2.6.12-6.el7.x86_64
>>>>>>> mongodb-server-2.6.12-6.el7.x86_64
>>>>>>> pulp-admin-client-2.18.1-2.el7.noarch
>>>>>>> pulp-docker-admin-extensions-3.2.2-1.el7.noarch
>>>>>>> pulp-docker-plugins-3.2.2-1.el7.noarch
>>>>>>> pulp-puppet-admin-extensions-2.18.1-1.el7.noarch
>>>>>>> pulp-puppet-plugins-2.18.1-1.el7.noarch
>>>>>>> pulp-rpm-admin-extensions-2.18.1-1.el7.noarch
>>>>>>> pulp-rpm-plugins-2.18.1-1.el7.noarch
>>>>>>> pulp-selinux-2.18.1-2.el7.noarch
>>>>>>> pulp-server-2.18.1-2.el7.noarch
>>>>>>> python2-celery-4.0.2-7.el7.noarch
>>>>>>> python2-qpid-1.37.0-4.el7.noarch
>>>>>>> python2-qpid-qmf-1.39.0-1.el7.x86_64
>>>>>>> python-gofer-2.12.5-1.el7.noarch
>>>>>>> python-gofer-qpid-2.12.5-1.el7.noarch
>>>>>>> python-isodate-0.5.0-4.pulp.el7.noarch
>>>>>>> python-mongoengine-0.10.5-1.el7.noarch
>>>>>>> python-pulp-bindings-2.18.1-2.el7.noarch
>>>>>>> python-pulp-client-lib-2.18.1-2.el7.noarch
>>>>>>> python-pulp-common-2.18.1-2.el7.noarch
>>>>>>> python-pulp-docker-common-3.2.2-1.el7.noarch
>>>>>>> python-pulp-oid_validation-2.18.1-2.el7.noarch
>>>>>>> python-pulp-puppet-common-2.18.1-1.el7.noarch
>>>>>>> python-pulp-repoauth-2.18.1-2.el7.noarch
>>>>>>> python-pulp-rpm-common-2.18.1-1.el7.noarch
>>>>>>> python-pymongo-3.2-2.el7.x86_64
>>>>>>> python-pymongo-gridfs-3.2-2.el7.x86_64
>>>>>>> qpid-cpp-client-1.39.0-1.el7.x86_64
>>>>>>> qpid-cpp-server-1.39.0-1.el7.x86_64
>>>>>>> qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
>>>>>>> qpid-proton-c-0.26.0-1.el7.x86_64
>>>>>>> qpid-qmf-1.39.0-1.el7.x86_64
>>>>>>> qpid-tools-1.39.0-1.el7.noarch
>>>>>>>
>>>>>>> Thanks very much for any input you can give on this issue!
>>>>>>>  - Kodiak Firesmith
>>>>>>> _______________________________________________
>>>>>>> Pulp-list mailing list
>>>>>>> Pulp-list at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20190401/86dc0e22/attachment.htm>