<div dir="ltr"><div dir="ltr"><div>Hi Venkataramana,</div><div> </div><div dir="ltr">It seems like you are running into this issue <a href="https://pulp.plan.io/issues/2540">https://pulp.plan.io/issues/2540</a>.</div><div>You probably have a very large repo and you probably run your Pulp 2.11 on RHEL 6 with MongoDB 2.4 which is very old and has certain issues.</div><div></div><div>See the suggested way of upgrading MongoDB to 2.6 (if you are on RHEL 6) <a href="https://pulp.plan.io/issues/2540#note-14">https://pulp.plan.io/issues/2540#note-14</a>.</div><div>This should solve your problem if my assumptions above are correct.</div><div>Though I encourage you to upgrade to the latest RHEL 7 and to the latest Pulp 2.18 and MongoDB 3.4. </div><div> </div><div>Let me know if your problem is solved or if you have any questions, </div><div>Tanya </div><div> </div><div> </div></div></div> <div class="gmail_quote"><div dir="ltr">On Tue, Jan 1, 2019 at 8:50 PM Venkataramana Bora <<a href="mailto:venkbora@in.ibm.com">venkbora@in.ibm.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Team, here is /var/log/messages error getting while trying to sync . Please help.<tt> </tt> Jan 1 10:10:07 dpa-pulpslv-01 puppet-agent[3657]: Caught TERM; exiting Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred) Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[c2f818fd-c871-4538-a091-5451e32885cf] Jan 1 10:11:27 dpa-pulpslv-01 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[1f04ce4f-c093-43a3-80bb-4f71b6ab8695] Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[c2f818fd-c871-4538-a091-5451e32885cf] succeeded in 0.060901439s: None Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[1f04ce4f-c093-43a3-80bb-4f71b6ab8695] succeeded in 0.130910374s: None Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) command SON([('mapreduce', u'units_rpm'), ('map', Code("\n function () {\n var key_fields = [<a href="http://this.name" target="_blank">this.name</a>, this.epoch, this.version, this.release, this.arch]\n emit(key_fields.join('-'), {ids: [this._id]});\n }\n ", {})), ('reduce', Code("\n function (key, values) {\n // collect mapped values into the first value to build the list of ids for this key/nevra\n var collector = values[0]\n // since collector is values[0] start this loop at index 1\n // reduce isn't called if map only emits one result for key,\n // so there is at least one value to collect\n for (var i = 1; i < values.length; i++) {\n collector.ids = collector.ids.concat(values[i].ids)\n }\n return collector\n }\n ", {})), ('out', {'inline': 1}), ('query', {}), ('finalize', Code("\n function (key, reduced) {\n if (reduced.ids.length > 1) {\n return reduced;\n }\n // if there's only one value after reduction, this key is useless\n // undefined is implicitly returned here, which saves space\n }\n ", {}))]) on namespace pulp_database.$cmd failed: exception: BSONObj size: 17162604 (0x6CE10501) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: { _id: "0ad-0-0.0.22-1.el7-x86_64", value: null } Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) Traceback (most recent call last): Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 296, in run Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) purge.remove_repo_duplicate_nevra(self.conduit.repo_id) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 304, in remove_repo_duplicate_nevra Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) for unit_ids in _duplicate_key_id_generator(unit_type): Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 493, in _duplicate_key_id_generator_mapreduce Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) for reduced in reduced_units: Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 1157, in map_reduce Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) map_f, reduce_f, **mr_args) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1903, in inline_map_reduce Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) res = self._command(sock_info, cmd, slave_ok) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 205, in _command Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) read_concern=read_concern) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/pool.py", line 211, in command Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) read_concern) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/network.py", line 100, in command Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) helpers._check_command_response(response_doc, msg, allowable_errors) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/helpers.py", line 196, in _check_command_response Sincerely, Venkataramana Bora IBM Visakha Hills Visakhapatnam – 530 045, India Direct:+91.891-4499606 / VoIP:92-29207 Mob:+91.9949358804 <img src="cid:1680e7e6bb8308fca931" alt="Inactive hide details for pulp-list-request---01/01/2019 10:02:18 PM---Send Pulp-list mailing list submissions to pulp-list@re" width="16" height="16" border="0">pulp-list-request---01/01/2019 10:02:18 PM---Send Pulp-list mailing list submissions to <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a> From: <a href="mailto:pulp-list-request@redhat.com" target="_blank">pulp-list-request@redhat.com</a> To: <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a> Date: 01/01/2019 10:02 PM Subject: Pulp-list Digest, Vol 110, Issue 1 Sent by: <a href="mailto:pulp-list-bounces@redhat.com" target="_blank">pulp-list-bounces@redhat.com</a> <hr style="color:rgb(128,145,165)" width="100%" size="2" noshade align="left"> <tt>Send Pulp-list mailing list submissions to <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a> To subscribe or unsubscribe via the World Wide Web, visit </tt><tt><a href="https://www.redhat.com/mailman/listinfo/pulp-list" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></tt><tt> or, via email, send a message with subject or body 'help' to <a href="mailto:pulp-list-request@redhat.com" target="_blank">pulp-list-request@redhat.com</a> You can reach the person managing the list at <a href="mailto:pulp-list-owner@redhat.com" target="_blank">pulp-list-owner@redhat.com</a> When replying, please edit your Subject line so it is more specific than "Re: Contents of Pulp-list digest..." Today's Topics: 1. Task Failed:Importer indicated a failed response (Venkataramana Bora) ---------------------------------------------------------------------- Message: 1 Date: Tue, 1 Jan 2019 22:00:58 +0530 From: "Venkataramana Bora" <<a href="mailto:venkbora@in.ibm.com" target="_blank">venkbora@in.ibm.com</a>> To: <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a> Subject: [Pulp-list] Task Failed:Importer indicated a failed response Message-ID: <<a href="mailto:OFD49419D1.B0C5E372-ON65258375.00586335-65258375.005ABA25@notes.na.collabserv.com" target="_blank">OFD49419D1.B0C5E372-ON65258375.00586335-65258375.005ABA25@notes.na.collabserv.com</a>> Content-Type: text/plain; charset="utf-8" Hi Team, can some some one kindly help me with Pulp sync error "Task Failed Importer indicated a failed response ? Also pasted here some lines of /var/log/messages. Please let me know if you need any further info to trouble shoot on this. This is Pulp slave server with version : pulp-server-2.11.2-1.el6. syncs with master Pulp server 2.16. It was working till few days ago with out any issues. Pulp sync error: [root@dpa-pulpslv-01 ~]# pulp-admin rpm repo sync run --repo-id=rhel-6-server-rpms-x86_64 +----------------------------------------------------------------------+ Synchronizing Repository [rhel-6-server-rpms-x86_64] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [/] ... completed Downloading repository content... [-] [==================================================] 100% RPMs: 0/0 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing package groups/categories... [|] ... completed Cleaning duplicate packages... [\] ... failed Task Failed Importer indicated a failed response [root@dpa-pulpslv-01 ~]# /var/log/messages: Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred) Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred [c2f818fd-c871-4538-a091-5451e32885cf] Jan 1 10:11:27 dpa-pulpslv-01 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred [1f04ce4f-c093-43a3-80bb-4f71b6ab8695] Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred [c2f818fd-c871-4538-a091-5451e32885cf] succeeded in 0.060901439s: None Jan 1 10:11:27 dpa-pulpslv-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred [1f04ce4f-c093-43a3-80bb-4f71b6ab8695] succeeded in 0.130910374s: None Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) command SON ([('mapreduce', u'units_rpm'), ('map', Code("\n function () {\n var key_fields = [<a href="http://this.name" target="_blank">this.name</a>, this.epoch, this.version, this.release, this.arch]\n emit(key_fields.join('-'), {ids: [this._id]});\n }\n ", {})), ('reduce', Code("\n function (key, values) {\n // collect mapped values into the first value to build the list of ids for this key/nevra\n var collector = values[0]\n // since collector is values[0] start this loop at index 1\n // reduce isn't called if map only emits one result for key,\n // so there is at least one value to collect\n for (var i = 1; i < values.length; i++) {\n collector.ids = collector.ids.concat(values[i].ids)\n }\n return collector\n }\n ", {})), ('out', {'inline': 1}), ('query', {}), ('finalize', Code("\n function (key, reduced) {\n if (reduced.ids.length > 1) {\n return reduced;\n }\n // if there's only one value after reduction, this key is useless\n // undefined is implicitly returned here, which saves space\n }\n ", {}))]) on namespace pulp_database.$cmd failed: exception: BSONObj size: 17162604 (0x6CE10501) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: { _id: "0ad-0-0.0.22-1.el7-x86_64", value: null } Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) Traceback (most recent call last): Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 296, in run Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) purge.remove_repo_duplicate_nevra(self.conduit.repo_id) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 304, in remove_repo_duplicate_nevra Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) for unit_ids in _duplicate_key_id_generator(unit_type): Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 493, in _duplicate_key_id_generator_mapreduce Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) for reduced in reduced_units: Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 1157, in map_reduce Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) map_f, reduce_f, **mr_args) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1903, in inline_map_reduce Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) res = self._command(sock_info, cmd, slave_ok) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 205, in _command Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) read_concern=read_concern) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/pool.py", line 211, in command Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) read_concern) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/network.py", line 100, in command Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) helpers._check_command_response(response_doc, msg, allowable_errors) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) File "/usr/lib64/python2.6/site-packages/pymongo/helpers.py", line 196, in _check_command_response Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) raise OperationFailure(msg %% errmsg, code, response) Jan 1 10:18:48 dpa-pulpslv-01 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (3304-06528) OperationFailure: command SON([('mapreduce', u'units_rpm'), ('map', Code("\n function () {\n var key_fields = [<a href="http://this.name" target="_blank">this.name</a>, this.epoch, this.version, this.release, this.arch]\n emit(key_fields.join('-'), {ids: [this._id]});\n }\n ", {})), ('reduce', Code("\n function (key, values) {\n // collect mapped values into the first value to build the list of ids for this key/nevra\n var collector = values[0]\n // since collector is values[0] start this loop at index 1\n // reduce isn't called if map only emits one result for key,\n // so there is at least one value to collect\n for (var i = 1; i < values.length; i++) {\n collector.ids = collector.ids.concat(values[i].ids)\n }\n return collector\n }\n ", {})), ('out', {'inline': 1}), ('query', {}), ('finalize', Code("\n function (key, reduced) {\n if (reduced.ids.length > 1) {\n return reduced;\n }\n // if there's only one value after reduction, this key is useless\n // undefined is implicitly returned here, which saves space\n }\n ", {}))]) on namespace pulp_database.$cmd failed: exception: BSONObj size: 17162604 (0x6CE10501) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: { _id: "0ad-0-0.0.22-1.el7-x86_64", value: null } Jan 1 10:18:49 dpa-pulpslv-01 pulp: pulp.server.async.tasks:INFO: Task failed : [6daefad5-b4a2-4a30-acbc-e6a2cd4dd697] Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) Task pulp.server.managers.repo.sync.sync [6daefad5-b4a2-4a30-acbc-e6a2cd4dd697] raised unexpected: PulpExecutionException('Importer indicated a failed response',) Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) Traceback (most recent call last): Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) R = retval = fun(*args, **kwargs) Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 491, in __call__ Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) return super(Task, self).__call__(*args, **kwargs) Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 103, in __call__ Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) return super(PulpTask, self).__call__(*args, **kwargs) Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) return self.run(*args, **kwargs) Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 820, in sync Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response')) Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:ERROR: (3221-06528) PulpExecutionException: Importer indicated a failed response Jan 1 10:18:49 dpa-pulpslv-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource [1850458c-9817-40d2-b295-d1ad1198269d] succeeded in 0.00577054099995s: None ~ Sincerely, Venkataramana Bora IBM Visakha Hills Visakhapatnam ? 530 045, India Direct:+91.891-4499606 / VoIP:92-29207 Mob:+91.9949358804 -------------- next part -------------- An HTML attachment was scrubbed... URL: <</tt><tt><a href="https://www.redhat.com/archives/pulp-list/attachments/20190101/fa070532/attachment.html" target="_blank">https://www.redhat.com/archives/pulp-list/attachments/20190101/fa070532/attachment.html</a></tt><tt>> ------------------------------ _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> </tt><tt><a href="https://www.redhat.com/mailman/listinfo/pulp-list" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></tt><tt> End of Pulp-list Digest, Vol 110, Issue 1 ***************************************** </tt> </div> _______________________________________________ Pulp-list mailing list <a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div>