<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">I am seeing this behavior as well after upgrading from 2.6 to 2.7. However, I am using qpid not rabbitmq.
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Jeffrey<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> pulp-list-bounces@redhat.com [mailto:pulp-list-bounces@redhat.com]
<b>On Behalf Of </b>Ashby, Jason (IMS)<br>
<b>Sent:</b> Wednesday, November 18, 2015 8:29 AM<br>
<b>To:</b> pulp-list@redhat.com<br>
<b>Subject:</b> [Pulp-list] Missing celery workers<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal">I’m hitting another issue with the upgrade to Pulp 2.7.0 + changing from qpid to rabbitmq for messaging. The workers are continuously going missing, every minute or so. The effect is that the tasks in the task list stay in a Waiting state
and are never completed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Rabbitmq looks healthy; I see successful accepted connections per the logs and can see a bunch of connections in the rabbitmq management GUI. I’m kind of stuck as far as troubleshooting goes. Any tips on what else to investigate?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Pulp and rabbitmq servers are both CentOS 6.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># /var/log/messages<o:p></o:p></p>
<p class="MsoNormal">Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed heartbeat from resource_manager@pulp01<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-3@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-1@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-2@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-0@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:56 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'resource_manager@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without<o:p></o:p></p>
<p class="MsoNormal">at least one pulp_resource_mananger process running.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication
is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
</body>
</html>