[Pulp-list] Errors in pulp.log : BSON document too large
Brian Bouterse
bbouters at redhat.com
Tue Nov 25 17:10:10 UTC 2014
Hi Paul,
Mongo does have a 16MB document limit [0], and this repo must have metadata that is a little larger than that. I was able to reproduce this bug, and I documented it here [1]. I did find some old BSON related Pulp bugs [2] [3], but those all seemed to be resolved so [1] is the only open BSON related bug I know of.
[0]: http://docs.mongodb.org/manual/reference/limits/#bson-documents
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1167950
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=748812
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=620495
-Brian
----- Original Message -----
> From: "Paul Gonin" <paul.gonin at gmail.com>
> To: "Michael Hrivnak" <mhrivnak at redhat.com>, pulp-list at redhat.com
> Sent: Monday, November 24, 2014 4:32:01 PM
> Subject: Re: [Pulp-list] Errors in pulp.log : BSON document too large
>
> Hi Michael,
>
> I ran manually the sync and it completed successfully from the admin
> client point of view:
> $ pulp-admin rpm repo sync run --repo-id=asis-chef-rhel6
> +----------------------------------------------------------------------+
> Synchronizing Repository [asis-chef-rhel6]
> +----------------------------------------------------------------------+
>
> This command may be exited by pressing ctrl+c without affecting the actual
> operation on the server.
>
> Downloading metadata...
> [\]
> ... completed
>
> Downloading repository content...
> [== ] 5%
> RPMs: 1/5 items
> Delta RPMs: 0/0 items
>
> ... completed
>
> Downloading distribution files...
> [==================================================] 100%
> Distributions: 0/0 items
> ... completed
>
> Importing errata...
> [-]
> ... completed
>
> Importing package groups/categories...
> [-]
> ... completed
>
> Publishing packages...
> [==== ] 9%
> Packages: 5/54 items
> Publishing distributions...
> [==================================================] 100%
> Distributions: 0/0 items
> [==================================================] 100%
> Packages: 54/54 items
> ... completed
>
> Generating metadata
> [|]
> ... completed
>
> Publishing repository over HTTP
> [-]
> ... completed
>
> no error in ~/.pulp/admin.log
>
> In pulp.log :
> 2014-11-24 21:55:26,888 pulp.plugins.conduits.mixins:ERROR: Content
> unit association failed [Unit [key={'name': 'private-chef',
> 'checksum': '2d068c98c36c4eb548773efc616fec5c36bd2bdd', 'epoch': '0',
> 'version': '11.2.3', 'release': '1.el6', 'arch': 'x86_64',
> 'checksumtype': 'sha'}] [type=rpm] [id=None]]
> 2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document
> too large (16967489 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> 2014-11-24 21:56:36,058 pulp.plugins.conduits.mixins:ERROR: Content
> unit association failed [Unit [key={'name': 'private-chef',
> 'checksum': 'd5d8bda7a3c260d42bbc79adb318cc05c794ae55', 'epoch': '0',
> 'version': '11.2.5', 'release': '1.el6', 'arch': 'x86_64',
> 'checksumtype': 'sha'}] [type=rpm] [id=None]]
> 2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document
> too large (16968021 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> 2014-11-24 21:56:45,599 pulp.plugins.conduits.mixins:ERROR: Content
> unit association failed [Unit [key={'name': 'private-chef',
> 'checksum': '8ffc55a8eaaf5ce31d9a37aa257bbf6ffdd0aa34', 'epoch': '0',
> 'version': '11.2.2', 'release': '1.el6', 'arch': 'x86_64',
> 'checksumtype': 'sha'}] [type=rpm] [id=None]]
> 2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document
> too large (16959923 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> 2014-11-24 21:57:03,051 pulp.plugins.conduits.mixins:ERROR: Content
> unit association failed [Unit [key={'name': 'private-chef',
> 'checksum': 'a0ef74235e3826dbf21ebea40070832d8430d988', 'epoch': '0',
> 'version': '11.2.4', 'release': '1.el6', 'arch': 'x86_64',
> 'checksumtype': 'sha'}] [type=rpm] [id=None]]
> 2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document
> too large (16992417 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
>
> In nectar.log
> 2014-11-24 21:50:52,988
> requests.packages.urllib3.connectionpool:DEBUG: "GET
> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/904.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=PuNQ0kK68ejp4jkxhVgmWlgpHHY%3D&Expires=1416862551
> HTTP/1.1" 200 576836991
> 2014-11-24 21:50:53,197
> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
> to <object object at 0x7fe99d14c100>
> 2014-11-24 21:50:53,310
> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
> to <object object at 0x7fe99d14c100>
> 2014-11-24 21:50:53,430
> requests.packages.urllib3.connectionpool:DEBUG: "GET
> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1290.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=AUyaEOnkxhQ7nRAHeGDuxsFLZJE%3D&Expires=1416862553
> HTTP/1.1" 200 577050972
> 2014-11-24 21:50:53,550
> requests.packages.urllib3.connectionpool:DEBUG: "GET
> https://packagecloud.io/chef/stable/el/6/x86_64/private-chef-11.2.5-1.el6.x86_64.rpm
> HTTP/1.1" 302 0
> 2014-11-24 21:50:53,552 requests.packages.urllib3.connectionpool:INFO:
> Starting new HTTPS connection (1):
> packagecloud-repositories.s3.amazonaws.com
> 2014-11-24 21:50:53,758
> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
> to <object object at 0x7fe99d14c100>
> 2014-11-24 21:50:53,998
> requests.packages.urllib3.connectionpool:DEBUG: "GET
> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1366.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=bgbKpTUhe3bNNWJHTgLJDsTcJZ0%3D&Expires=1416862552
> HTTP/1.1" 200 578311972
> 2014-11-24 21:50:54,248
> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
> to <object object at 0x7fe99d14c100>
> 2014-11-24 21:50:54,478
> requests.packages.urllib3.connectionpool:DEBUG: "GET
> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1448.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=vGIQTaz8OYV8cIrUCQY4dsxvQ6U%3D&Expires=1416862553
> HTTP/1.1" 200 577030942
> 2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document
> too large (16967489 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
> line 110, in _fire_event_to_listener
> event_listener_callback(*args, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
> line 97, in download_succeeded
> self.sync_conduit.save_unit(unit)
> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
> line 485, in save_unit
> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
> File
> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
> line 47, in add_content_unit
> collection.insert(unit_doc, safe=True)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 114, in retry
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
> line 357, in insert
> continue_on_error, self.__uuid_subtype), safe)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 143, in _with_end_request
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 910, in _send_message
> (request_id, data) = self.__check_bson_size(message)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 881, in __check_bson_size
> (max_doc_size, self.__max_bson_size))
> ImporterConduitException: BSON document too large (16967489 bytes) -
> the connected server supports BSON document sizes up to 16777216
> bytes.
> 2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document
> too large (16968021 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
> line 110, in _fire_event_to_listener
> event_listener_callback(*args, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
> line 97, in download_succeeded
> self.sync_conduit.save_unit(unit)
> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
> line 485, in save_unit
> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
> File
> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
> line 47, in add_content_unit
> collection.insert(unit_doc, safe=True)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 114, in retry
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
> line 357, in insert
> continue_on_error, self.__uuid_subtype), safe)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 143, in _with_end_request
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 910, in _send_message
> (request_id, data) = self.__check_bson_size(message)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 881, in __check_bson_size
> (max_doc_size, self.__max_bson_size))
> ImporterConduitException: BSON document too large (16968021 bytes) -
> the connected server supports BSON document sizes up to 16777216
> bytes.
> 2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document
> too large (16959923 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
> line 110, in _fire_event_to_listener
> event_listener_callback(*args, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
> line 97, in download_succeeded
> self.sync_conduit.save_unit(unit)
> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
> line 485, in save_unit
> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
> File
> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
> line 47, in add_content_unit
> collection.insert(unit_doc, safe=True)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 114, in retry
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
> line 357, in insert
> continue_on_error, self.__uuid_subtype), safe)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 143, in _with_end_request
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 910, in _send_message
> (request_id, data) = self.__check_bson_size(message)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 881, in __check_bson_size
> (max_doc_size, self.__max_bson_size))
> ImporterConduitException: BSON document too large (16959923 bytes) -
> the connected server supports BSON document sizes up to 16777216
> bytes.
> 2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document
> too large (16992417 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
> line 110, in _fire_event_to_listener
> event_listener_callback(*args, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
> line 97, in download_succeeded
> self.sync_conduit.save_unit(unit)
> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
> line 485, in save_unit
> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
> File
> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
> line 47, in add_content_unit
> collection.insert(unit_doc, safe=True)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 114, in retry
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
> line 357, in insert
> continue_on_error, self.__uuid_subtype), safe)
> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
> line 143, in _with_end_request
> return method(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 910, in _send_message
> (request_id, data) = self.__check_bson_size(message)
> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
> line 881, in __check_bson_size
> (max_doc_size, self.__max_bson_size))
> ImporterConduitException: BSON document too large (16992417 bytes) -
> the connected server supports BSON document sizes up to 16777216
> bytes.
> 2014-11-24 21:57:03,062 nectar.downloaders.threaded:DEBUG: starting workers
>
>
> mongodb.log empty
>
> Thanks !
> Regards
> Paul
>
> On Mon, Nov 24, 2014 at 7:44 PM, Michael Hrivnak <mhrivnak at redhat.com> wrote:
> > Paul,
> >
> > Is there a full stack trace in the log that you can share?
> >
> > Thanks,
> > Michael
> >
> > ----- Original Message -----
> > From: "Paul Gonin" <paul.gonin at gmail.com>
> > To: pulp-list at redhat.com
> > Sent: Monday, November 24, 2014 5:36:30 AM
> > Subject: [Pulp-list] Errors in pulp.log : BSON document too large
> >
> > Hi !
> >
> > A few days ago I started synchronizing chef repo from packagecloud.io
> > https://packagecloud.io/chef/stable/el/6/x86_64
> >
> > I scheduled the sync to be ran every night at 2am.
> >
> > Since then I have the following MongoDB related error in pulp.log
> > which seems related to MongoDB document size limit to 16MB
> >
> > 2014-11-22 02:36:56,824 nectar.downloaders.base:ERROR: BSON document
> > too large (16968021 bytes) - the connected server supports BSON
> > document sizes up to 16777216 bytes.
> >
> > Any idea how to address it ?
> > I looked in Bugzilla if it is known issue but could not anything
> > related to BSON.
> >
> > Thanks
> > Paul
> >
> > _______________________________________________
> > Pulp-list mailing list
> > Pulp-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
More information about the Pulp-list
mailing list