<div dir="ltr"><div>Hi Venkataramana,</div><div><br></div><div>Thanks for the logs. In there, I only see one error, but I see it a huge number of times: ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused To me that means environmentally celery and/or the WSGI environments cannot connect to your mongod. Unfortunately, I don't have more info than that.</div><div><br></div><div>All the best,</div><div>Brian</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 4, 2021 at 2:29 PM Venkataramana Bora <<a href="mailto:venkbora@in.ibm.com">venkbora@in.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr">Hi Brian, here have attached "pulp:" logs from /var/log/messages .Sorry for attaching bit lengthy text files.</div>
<div dir="ltr">Other than those, I did not find any other pulp logs . Thank you !</div>
<div dir="ltr"><br><br><br>Sincerely,<br>Ramana Bora</div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: Brian Bouterse <<a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a>><br>To: Venkataramana Bora <<a href="mailto:venkbora@in.ibm.com" target="_blank">venkbora@in.ibm.com</a>><br>Cc: pulp-list <<a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a>>, Donal Keane <<a href="mailto:donal.keane@ie.ibm.com" target="_blank">donal.keane@ie.ibm.com</a>><br>Subject: [EXTERNAL] Re: [Pulp-list] pulp mongodb upgrade<br>Date: Wed, Jun 2, 2021 9:29 PM<br> <br> <span style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden"> Hi Venkataramana and Donal, I looked at the logs, but unfortunately I don't see any exceptions related to Pulp's use of Mongo. Mostly I just see a mongodb error: `FileRenameFailed Renaming file /var/log/mongodb/mongodb.log to /var/log/mongodb/mongodb.log.2021-05-27T07-33-08 </span> <span style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerStart</span>
<div style="margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)"><div style="display:inline-block;margin:0px 0px 1px;max-width:600px"><div style="color:rgb(0,0,0);font-family:"Arial",sans-serif;font-weight:bold;font-size:14px;line-height:18px;display:block">This Message Is From an External Sender</div>
<div style="color:rgb(0,0,0);font-weight:normal;font-family:"Arial",sans-serif;font-size:12px;line-height:18px;margin-top:2px;display:block">This message came from outside your organization.</div></div></div> <span style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerEnd</span><br>
<div dir="ltr"><div>Hi Venkataramana and Donal,</div>
<div> </div>
<div>I looked at the logs, but unfortunately I don't see any exceptions related to Pulp's use of Mongo. Mostly I just see a mongodb error: `FileRenameFailed Renaming file /var/log/mongodb/mongodb.log to /var/log/mongodb/mongodb.log.2021-05-27T07-33-08 failed; Cannot verify whether destination already exists`.</div>
<div> </div>
<div>I believe the pulp logs themselves would be needed.</div>
<div> </div>
<div>-Brian</div>
<div> </div></div>
<div><div dir="ltr">On Fri, May 28, 2021 at 12:18 PM Venkataramana Bora <<a href="mailto:venkbora@in.ibm.com" target="_blank">venkbora@in.ibm.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr"> </div>
<div dir="ltr">Hi Brian,</div>
<div dir="ltr"> </div>
<div dir="ltr">Please find attached MongoDB logs from /var/log/mongodb/. There are no logs in /var/log/pulp. Please let us know if any other logs required . Thanks a lot for your reply!<br><br><br>Sincerely,<br>Ramana Bora</div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: Donal Keane/Ireland/IBM<br>To: <a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a><br>Cc: <a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a>, Venkataramana Bora/India/IBM@IBM<br>Subject: Re: [EXTERNAL] Re: [Pulp-list] pulp mongodb upgrade<br>Date: Fri, May 28, 2021 2:19 AM<br>
<div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr">Hi Brian,</div>
<div dir="ltr"> </div>
<div dir="ltr">Thank you kindly for your response.</div>
<div dir="ltr"> </div>
<div dir="ltr">The plan here originally was to move to Pulp 3 - and definitely still is. However, our brand within IBM has recently been sold off and it is happening sooner rather than later. As part of this acquisition we are losing a good proportion of expertise around our Linux operating systems. We have been tasked as a high priority to make sure all of our open source tools are patched to the latest security level before the acquisition goes through. This is why Ramana is attempting to get us to 2.21.x asap and from there, concentrate on getting to 3 when time and resources allow. I hope this explains our current situation and why any help would be graciously accepted!</div>
<div dir="ltr"> </div>
<div dir="ltr">Ramana can you post relevant log info when you get a chance tomorrow?</div>
<div dir="ltr"> </div>
<div dir="ltr">Thanks,</div>
<div dir="ltr"> </div>
<div dir="ltr">Donal</div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:2px solid rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: Brian Bouterse <<a href="mailto:bmbouter@redhat.com" target="_blank">bmbouter@redhat.com</a>><br>To: Venkataramana Bora <<a href="mailto:venkbora@in.ibm.com" target="_blank">venkbora@in.ibm.com</a>><br>Cc: pulp-list <<a href="mailto:pulp-list@redhat.com" target="_blank">pulp-list@redhat.com</a>>, Donal Keane <<a href="mailto:donal.keane@ie.ibm.com" target="_blank">donal.keane@ie.ibm.com</a>><br>Subject: [EXTERNAL] Re: [Pulp-list] pulp mongodb upgrade<br>Date: Thu, May 27, 2021 8:12 PM<br> <br><span style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden">I don't have ready advice, but it might be helpful to post a snippet of the logs. Without that, all we know is there is a 500 server error, not what the cause could be. Have you considered migrating to Pulp3? Pulp2 is in maintenance mode, </span> <span style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerStart</span>
<div style="margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)"><div style="display:inline-block;margin:0px 0px 1px;max-width:600px"><div style="color:rgb(0,0,0);font-family:"Arial",sans-serif;font-weight:bold;font-size:14px;line-height:18px;display:block">This Message Is From an External Sender</div>
<div style="color:rgb(0,0,0);font-weight:normal;font-family:"Arial",sans-serif;font-size:12px;line-height:18px;margin-top:2px;display:block">This message came from outside your organization.</div></div></div><span style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerEnd</span>
<div dir="ltr"><div>I don't have ready advice, but it might be helpful to post a snippet of the logs. Without that, all we know is there is a 500 server error, not what the cause could be.</div>
<div> </div>
<div>Have you considered migrating to Pulp3? Pulp2 is in maintenance mode, and likely has had its terminal release. Pulp3 is at 3.13 and has been out since Dec 2019. Here's more information on how to upgrade <a href="https://pulpproject.org/migrate-to-pulp-3/" target="_blank">https://pulpproject.org/migrate-to-pulp-3/</a> We would be very interested in helping you do that.</div></div>
<div><div dir="ltr">On Thu, May 27, 2021 at 12:03 PM Venkataramana Bora <<a href="mailto:venkbora@in.ibm.com" target="_blank">venkbora@in.ibm.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr"> </div>
<div dir="ltr"><div>Hi team,</div>
<div>We are using pulp-server-2.21.5-1.el7 with mongodb-server-2.6.12-6.el7.x86_64 , this version of mongodb 2.6.12-16 got installed from our config manage server puppet .<br>Time and again pulp-admin commands run, throwing below errors , I think it is due to old version of mongodb. After running /var/lib/mongodb --repair able to access pulp-admin commands for some time/few days as shown here below.<br>Some one kindly help us on how to upgrade mongodb-server-2.6.12-6 to higher mongodb version that is compatible with pulp-server-2.21.5-1 ? This specific pulp-server we are using instead of higher versions due to some internal dependencies. Thanks a lot in advance.</div>
<div> <br>[root@drtp1opsndbx04 ~]# pulp-admin tasks list<br>+----------------------------------------------------------------------+<br> Tasks<br>+----------------------------------------------------------------------+</div>
<div>There was an internal server error while trying to access the Pulp application.<br>One possible cause is that the database needs to be migrated to the latest<br>version. If this is the case, run pulp-manage-db and restart the services. More<br>information may be found in Apache's log.</div>
<div>root@drtp1opsndbx04 ~]# pulp-admin repo list<br>+----------------------------------------------------------------------+<br> Repositories<br>+----------------------------------------------------------------------+</div>
<div>There was an internal server error while trying to access the Pulp application.<br>One possible cause is that the database needs to be migrated to the latest<br>version. If this is the case, run pulp-manage-db and restart the services. More<br>information may be found in Apache's log.</div>
<div> </div>
<div>After running /var/lib/mongodb --repair working a little while:</div>
<div> </div>
<div>[root@drtp1opsndbx04 ~]# pulp-admin tasks list<br>+----------------------------------------------------------------------+<br> Tasks<br>+----------------------------------------------------------------------+</div>
<div>No tasks found</div>
<div><br>[root@drtp1opsndbx04 ~]# pulp-admin repo list<br>+----------------------------------------------------------------------+<br> Repositories<br>+----------------------------------------------------------------------+</div>
<div>Id: virtualbox<br>Display Name: virtualbox<br>Description: virtualbox<br>Content Unit Counts:<br> Rpm: 107</div>
<div>.....</div>
<div> </div>
<div> </div>
<div> </div>
<div><br>Sincerely,<br>Venkataramana Bora</div>
<div>IBM - TMS cloud operations</div>
<div>Plot #1, Hill # 3, Apiic & Ites Sez, Rishikonda, Madhuravada<br>Visakhapatnam, AP 530045, <span>India.</span></div></div></div><br>_______________________________________________<br>Pulp-list mailing list<br><a href="mailto:Pulp-list@redhat.com" target="_blank">Pulp-list@redhat.com</a><br><a href="https://listman.redhat.com/mailman/listinfo/pulp-list" rel="noreferrer" target="_blank">https://listman.redhat.com/mailman/listinfo/pulp-list</a></blockquote></div></blockquote>
<div dir="ltr"> </div></div></blockquote>
<div dir="ltr"> </div></div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210527"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210425"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210418"> </div></div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210527"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210425"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210418"> </div></div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210527"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210425"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210418"> </div></div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210527"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210425"> </div>
<div dir="ltr" id="gmail-m_712116268669971475gmail-m_-2163674892973822199MIMEAttachInfoDiv" style="display:none" title="octet-stream|mongodb.log-20210418"> </div></div></blockquote></div></blockquote>
<div dir="ltr"> </div></div>
<br>
</blockquote></div>