[Pulp-list] Errors in pulp.log : BSON document too large

Paul Gonin paul.gonin at gmail.com
Mon Nov 24 21:32:01 UTC 2014


Hi Michael,

I ran manually the sync and it completed successfully from the admin
client point of view:
$ pulp-admin rpm repo sync run --repo-id=asis-chef-rhel6
+----------------------------------------------------------------------+
               Synchronizing Repository [asis-chef-rhel6]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==                                                ] 5%
RPMs:       1/5 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Publishing packages...
[====                                              ] 9%
Packages: 5/54 items
Publishing distributions...
[==================================================] 100%
Distributions: 0/0 items
[==================================================] 100%
Packages: 54/54 items
... completed

Generating metadata
[|]
... completed

Publishing repository over HTTP
[-]
... completed

no error in ~/.pulp/admin.log

In pulp.log :
2014-11-24 21:55:26,888 pulp.plugins.conduits.mixins:ERROR: Content
unit association failed [Unit [key={'name': 'private-chef',
'checksum': '2d068c98c36c4eb548773efc616fec5c36bd2bdd', 'epoch': '0',
'version': '11.2.3', 'release': '1.el6', 'arch': 'x86_64',
'checksumtype': 'sha'}] [type=rpm] [id=None]]
2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document
too large (16967489 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
2014-11-24 21:56:36,058 pulp.plugins.conduits.mixins:ERROR: Content
unit association failed [Unit [key={'name': 'private-chef',
'checksum': 'd5d8bda7a3c260d42bbc79adb318cc05c794ae55', 'epoch': '0',
'version': '11.2.5', 'release': '1.el6', 'arch': 'x86_64',
'checksumtype': 'sha'}] [type=rpm] [id=None]]
2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document
too large (16968021 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
2014-11-24 21:56:45,599 pulp.plugins.conduits.mixins:ERROR: Content
unit association failed [Unit [key={'name': 'private-chef',
'checksum': '8ffc55a8eaaf5ce31d9a37aa257bbf6ffdd0aa34', 'epoch': '0',
'version': '11.2.2', 'release': '1.el6', 'arch': 'x86_64',
'checksumtype': 'sha'}] [type=rpm] [id=None]]
2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document
too large (16959923 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
2014-11-24 21:57:03,051 pulp.plugins.conduits.mixins:ERROR: Content
unit association failed [Unit [key={'name': 'private-chef',
'checksum': 'a0ef74235e3826dbf21ebea40070832d8430d988', 'epoch': '0',
'version': '11.2.4', 'release': '1.el6', 'arch': 'x86_64',
'checksumtype': 'sha'}] [type=rpm] [id=None]]
2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document
too large (16992417 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.

In nectar.log
2014-11-24 21:50:52,988
requests.packages.urllib3.connectionpool:DEBUG: "GET
https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/904.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=PuNQ0kK68ejp4jkxhVgmWlgpHHY%3D&Expires=1416862551
HTTP/1.1" 200 576836991
2014-11-24 21:50:53,197
requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
to <object object at 0x7fe99d14c100>
2014-11-24 21:50:53,310
requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
to <object object at 0x7fe99d14c100>
2014-11-24 21:50:53,430
requests.packages.urllib3.connectionpool:DEBUG: "GET
https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1290.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=AUyaEOnkxhQ7nRAHeGDuxsFLZJE%3D&Expires=1416862553
HTTP/1.1" 200 577050972
2014-11-24 21:50:53,550
requests.packages.urllib3.connectionpool:DEBUG: "GET
https://packagecloud.io/chef/stable/el/6/x86_64/private-chef-11.2.5-1.el6.x86_64.rpm
HTTP/1.1" 302 0
2014-11-24 21:50:53,552 requests.packages.urllib3.connectionpool:INFO:
Starting new HTTPS connection (1):
packagecloud-repositories.s3.amazonaws.com
2014-11-24 21:50:53,758
requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
to <object object at 0x7fe99d14c100>
2014-11-24 21:50:53,998
requests.packages.urllib3.connectionpool:DEBUG: "GET
https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1366.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=bgbKpTUhe3bNNWJHTgLJDsTcJZ0%3D&Expires=1416862552
HTTP/1.1" 200 578311972
2014-11-24 21:50:54,248
requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout
to <object object at 0x7fe99d14c100>
2014-11-24 21:50:54,478
requests.packages.urllib3.connectionpool:DEBUG: "GET
https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1448.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=vGIQTaz8OYV8cIrUCQY4dsxvQ6U%3D&Expires=1416862553
HTTP/1.1" 200 577030942
2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document
too large (16967489 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
line 110, in _fire_event_to_listener
    event_listener_callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
line 97, in download_succeeded
    self.sync_conduit.save_unit(unit)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
line 485, in save_unit
    unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
line 47, in add_content_unit
    collection.insert(unit_doc, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 114, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
line 357, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 143, in _with_end_request
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 910, in _send_message
    (request_id, data) = self.__check_bson_size(message)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 881, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
ImporterConduitException: BSON document too large (16967489 bytes) -
the connected server supports BSON document sizes up to 16777216
bytes.
2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document
too large (16968021 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
line 110, in _fire_event_to_listener
    event_listener_callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
line 97, in download_succeeded
    self.sync_conduit.save_unit(unit)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
line 485, in save_unit
    unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
line 47, in add_content_unit
    collection.insert(unit_doc, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 114, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
line 357, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 143, in _with_end_request
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 910, in _send_message
    (request_id, data) = self.__check_bson_size(message)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 881, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
ImporterConduitException: BSON document too large (16968021 bytes) -
the connected server supports BSON document sizes up to 16777216
bytes.
2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document
too large (16959923 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
line 110, in _fire_event_to_listener
    event_listener_callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
line 97, in download_succeeded
    self.sync_conduit.save_unit(unit)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
line 485, in save_unit
    unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
line 47, in add_content_unit
    collection.insert(unit_doc, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 114, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
line 357, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 143, in _with_end_request
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 910, in _send_message
    (request_id, data) = self.__check_bson_size(message)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 881, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
ImporterConduitException: BSON document too large (16959923 bytes) -
the connected server supports BSON document sizes up to 16777216
bytes.
2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document
too large (16992417 bytes) - the connected server supports BSON
document sizes up to 16777216 bytes.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py",
line 110, in _fire_event_to_listener
    event_listener_callback(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py",
line 97, in download_succeeded
    self.sync_conduit.save_unit(unit)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py",
line 485, in save_unit
    unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py",
line 47, in add_content_unit
    collection.insert(unit_doc, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 114, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py",
line 357, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py",
line 143, in _with_end_request
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 910, in _send_message
    (request_id, data) = self.__check_bson_size(message)
  File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py",
line 881, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
ImporterConduitException: BSON document too large (16992417 bytes) -
the connected server supports BSON document sizes up to 16777216
bytes.
2014-11-24 21:57:03,062 nectar.downloaders.threaded:DEBUG: starting workers


mongodb.log empty

Thanks !
Regards
Paul

On Mon, Nov 24, 2014 at 7:44 PM, Michael Hrivnak <mhrivnak at redhat.com> wrote:
> Paul,
>
> Is there a full stack trace in the log that you can share?
>
> Thanks,
> Michael
>
> ----- Original Message -----
> From: "Paul Gonin" <paul.gonin at gmail.com>
> To: pulp-list at redhat.com
> Sent: Monday, November 24, 2014 5:36:30 AM
> Subject: [Pulp-list] Errors in pulp.log : BSON document too large
>
> Hi !
>
> A few days ago I started synchronizing chef repo from packagecloud.io
> https://packagecloud.io/chef/stable/el/6/x86_64
>
> I scheduled the sync to be ran every night at 2am.
>
> Since then I have the following MongoDB related error in pulp.log
> which seems related to MongoDB document size limit to 16MB
>
> 2014-11-22 02:36:56,824 nectar.downloaders.base:ERROR: BSON document
> too large (16968021 bytes) - the connected server supports BSON
> document sizes up to 16777216 bytes.
>
> Any idea how to address it ?
> I looked in Bugzilla if it is known issue but could not anything
> related to BSON.
>
> Thanks
> Paul
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list




More information about the Pulp-list mailing list