[Pulp-list] Can't get CentOS 7 Updates repo published

Sat Oct 3 05:51:45 UTC 2020

Thanks for the info Dennis!

I tried 'rpm repo sync sun --force-full' but it didn't help to clear the
corrupted entry. How can I drop this on the server side? Should I just
directly edit an entry in mongodb?

пт, 2 окт. 2020 г. в 14:39, Dennis Kliban <dkliban at redhat.com>:

> Filelists are stored in the database in a compressed form[0]. During
> publishing of metadata, each fillist is decompressed before being written
> to a file[1]. It's hard to speculate when data corruption occurred. It's
> possible that the data was corrupted in the remote repository the first
> time that this RPM was downloaded. That metadata could have been fixed
> since then, but Pulp thinks it already has the information and is not
> trying to re-download.
>
> [0]
> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L1051
> [1]
> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L1064
>
> On Fri, Oct 2, 2020 at 3:27 AM Konstantin M. Khankin <
> khankin.konstantin at gmail.com> wrote:
>
>> Sorry for reviving this thread but I think I found something.
>>
>> Today I tried to install CentOS 7 and got an error:
>> "TypeError: Parsing filelists.xml error: expected '>'"
>>
>> So I did this:
>>
>> >>> import xml.etree.ElementTree as ET
>> >>> tree =
>> ET.parse('/mnt/sysimage/var/tmp/yum.cache/centos_updates/gen/filelists.xml')
>> ...
>>   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in
>> _raiseerror
>>     raise err
>> xml.etree.ElementTree.ParseError: not well-formed (invalid token): line
>> 1884123, column 136
>>
>> I opened filelists.xml and found that line 1884123 is indeed corrupt and
>> looks like this:
>>
>> <file>/usr/share/javadoc/java-1.8.0-openjdk-1.8.0.111-1.b15.el7_2-debug/api/javax/sql/rowset/serial/compact3-package-frame.html</fil%
>>   !,^@ 0
>> 480!share/javadoc/java-1.8.0-openjdk-1.8.0.111-1.b15.el7_2-debug/api/javax/sql/rowset/serial/compact3-package-summary.html</file>
>>
>> I verified another consumer has exactly the same issue with this file.
>>
>> Do I understand correctly Pulp generates filelists.xml? If so, can it be
>> because of a bug in Pulp or should I look for a silent data corruption
>> issue on my pulp server?
>>
>> Thanks!
>>
>>
>> вт, 1 сент. 2020 г. в 17:14, Dennis Kliban <dkliban at redhat.com>:
>>
>>> The repoview feature requires the sqlite db to be generated. This
>>> feature generates HTML pages for browsing the repository in a web browser.
>>> I believe that users are still able to browse the repository without it,
>>> but in that case the HTML listing of directories is generated by the web
>>> server each time a user requests it.
>>>
>>> On Tue, Sep 1, 2020 at 4:53 AM Konstantin M. Khankin <
>>> khankin.konstantin at gmail.com> wrote:
>>>
>>>> Thank you, that obviously helped. Still would be useful to know the
>>>> root cause.
>>>>
>>>> OTOH, if sqlite creation functionality is not that critical - why would
>>>> we ever enable something which only consumes time during sync for no
>>>> benefit?
>>>>
>>>> вт, 1 сент. 2020 г. в 00:26, Dennis Kliban <dkliban at redhat.com>:
>>>>
>>>>> Looks like everything is up to date. I have no idea what the root
>>>>> cause is, but according to this comment[0], you can work around the problem
>>>>> by disabling the generation of sqlite db. I am not sure what the exact
>>>>> effect of this will be for the clients that are consuming this content, but
>>>>> the repository will be usable.
>>>>>
>>>>> [0] https://pulp.plan.io/issues/2019#note-2
>>>>>
>>>>> On Mon, Aug 31, 2020 at 4:10 PM Konstantin M. Khankin <
>>>>> khankin.konstantin at gmail.com> wrote:
>>>>>
>>>>>> # rpm -qa | grep pulp | sort
>>>>>> pulp-admin-client-2.21.3-1.el7.noarch
>>>>>> pulp-agent-2.21.3-1.el7.noarch
>>>>>> pulp-consumer-client-2.21.3-1.el7.noarch
>>>>>> pulp-deb-admin-extensions-1.10.1-1.el7.noarch
>>>>>> pulp-deb-plugins-1.10.1-1.el7.noarch
>>>>>> pulp-docker-admin-extensions-3.2.6-1.el7.noarch
>>>>>> pulp-docker-plugins-3.2.6-1.el7.noarch
>>>>>> pulp-puppet-consumer-extensions-2.21.3-1.el7.noarch
>>>>>> pulp-puppet-handlers-2.21.3-1.el7.noarch
>>>>>> pulp-python-admin-extensions-2.0.4-1.el7.noarch
>>>>>> pulp-python-plugins-2.0.4-1.el7.noarch
>>>>>> pulp-rpm-admin-extensions-2.21.3-1.el7.noarch
>>>>>> pulp-rpm-consumer-extensions-2.21.3-1.el7.noarch
>>>>>> pulp-rpm-handlers-2.21.3-1.el7.noarch
>>>>>> pulp-rpm-plugins-2.21.3-1.el7.noarch
>>>>>> pulp-rpm-yumplugins-2.21.3-1.el7.noarch
>>>>>> pulp-selinux-2.21.3-1.el7.noarch
>>>>>> pulp-server-2.21.3-1.el7.noarch
>>>>>> python-pulp-agent-lib-2.21.3-1.el7.noarch
>>>>>> python-pulp-bindings-2.21.3-1.el7.noarch
>>>>>> python-pulp-client-lib-2.21.3-1.el7.noarch
>>>>>> python-pulp-common-2.21.3-1.el7.noarch
>>>>>> python-pulp-deb-common-1.10.1-1.el7.noarch
>>>>>> python-pulp-docker-common-3.2.6-1.el7.noarch
>>>>>> python-pulp-oid_validation-2.21.3-1.el7.noarch
>>>>>> python-pulp-puppet-common-2.21.3-1.el7.noarch
>>>>>> python-pulp-python-common-2.0.4-1.el7.noarch
>>>>>> python-pulp-repoauth-2.21.3-1.el7.noarch
>>>>>> python-pulp-rpm-common-2.21.3-1.el7.noarch
>>>>>>
>>>>>> # rpm -qa | grep createrepo
>>>>>> createrepo-0.9.9-28.el7.noarch
>>>>>> createrepo_c-0.10.0-20.el7.x86_64
>>>>>> createrepo_c-libs-0.10.0-20.el7.x86_64
>>>>>>
>>>>>> # cat /etc/redhat-release
>>>>>> CentOS Linux release 7.8.2003 (Core)
>>>>>>
>>>>>> пн, 31 авг. 2020 г. в 22:48, Dennis Kliban <dkliban at redhat.com>:
>>>>>>
>>>>>>> This looks exactly like the issue that was reported here[0].
>>>>>>>
>>>>>>> What version of pulp are you using? What version of createrepo_c is
>>>>>>> installed?
>>>>>>>
>>>>>>> [0] https://pulp.plan.io/issues/2019
>>>>>>>
>>>>>>> On Mon, Aug 31, 2020 at 3:16 PM Konstantin M. Khankin <
>>>>>>> khankin.konstantin at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> The issue still persists. Could someone take a look please?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> ср, 19 авг. 2020 г., 13:15 Konstantin M. Khankin <
>>>>>>>> khankin.konstantin at gmail.com>:
>>>>>>>>
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> I found that my pulp2-managed mirror of
>>>>>>>>> http://mirror.yandex.ru/centos/7/updates/x86_64 has not been
>>>>>>>>> successfully published since April 30. I ran publish manually and received
>>>>>>>>> an error:
>>>>>>>>>
>>>>>>>>> '''
>>>>>>>>> Generating sqlite files
>>>>>>>>> [/]
>>>>>>>>> ... failed
>>>>>>>>> Error occurred during 'sqliterepo_c' execution: Preparing sqlite
>>>>>>>>> DBs
>>>>>>>>>
>>>>>>>>> ::
>>>>>>>>> C_CREATEREPOLIB: Critical: cr_xml_parser_generic: parsing error
>>>>>>>>> '/var/cache/pulp/reserved_resource_worker-1 at hive.gsk.loc
>>>>>>>>> /12257118-33e7-4294-ad68
>>>>>>>>>
>>>>>>>>> -98e0515f2627/repodata/9e958c09c7880d130ef3321332000f43a4e4e701e416801a0f54313f5
>>>>>>>>> 766d03b-filelists.xml.gz': not well-formed (invalid token)
>>>>>>>>> Parse error
>>>>>>>>> '/var/cache/pulp/reserved_resource_worker-1 at hive.gsk.loc
>>>>>>>>> /12257118-33e7-4294-ad68
>>>>>>>>>
>>>>>>>>> -98e0515f2627/repodata/9e958c09c7880d130ef3321332000f43a4e4e701e416801a0f54313f5
>>>>>>>>> 766d03b-filelists.xml.gz' at line: 1884123 (not well-formed
>>>>>>>>> (invalid token))
>>>>>>>>> '''
>>>>>>>>>
>>>>>>>>> I can't open .sqlite files in /tmp from CLI either:
>>>>>>>>>
>>>>>>>>> '''
>>>>>>>>> -rw-------. 1 apache apache  54066176 Aug 19 15:12
>>>>>>>>> filelists.211JP0.sqlite
>>>>>>>>> -rw-------. 1 apache apache 157097984 Aug 19 15:12
>>>>>>>>> primary.PQ3JP0.sqlite
>>>>>>>>> -rw-------. 1 apache apache         0 Aug 19 15:12
>>>>>>>>> other.X201JP0.sqlite
>>>>>>>>>
>>>>>>>>> [root at hive tmp]# sqlite3 primary.PQ3JP0.sqlite
>>>>>>>>> SQLite version 3.7.17 2013-05-20 00:56:22
>>>>>>>>> Enter ".help" for instructions
>>>>>>>>> Enter SQL statements terminated with a ";"
>>>>>>>>> sqlite> .databases
>>>>>>>>> Error: file is encrypted or is not a database
>>>>>>>>> '''
>>>>>>>>>
>>>>>>>>> I have multiple repos managed by pulp, some of which originate
>>>>>>>>> also from mirror.yandex.ru, and they are synced and published
>>>>>>>>> normally. --force-full doesn't help.
>>>>>>>>>
>>>>>>>>> What issue could be here?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Konstantin Khankin
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pulp-list mailing list
>>>>>>>> Pulp-list at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ханкин Константин
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Ханкин Константин
>>>>
>>>
>>
>> --
>> Ханкин Константин
>>
>

-- 
Ханкин Константин
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20201003/e05a638b/attachment.htm>