<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal">I’m hitting another issue with the upgrade to Pulp 2.7.0 + changing from qpid to rabbitmq for messaging. The workers are continuously going missing, every minute or so. The effect is that the tasks in the task list stay in a Waiting state
and are never completed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Rabbitmq looks healthy; I see successful accepted connections per the logs and can see a bunch of connections in the rabbitmq management GUI. I’m kind of stuck as far as troubleshooting goes. Any tips on what else to investigate?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Pulp and rabbitmq servers are both CentOS 6.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># /var/log/messages<o:p></o:p></p>
<p class="MsoNormal">Nov 18 08:53:56 pulp01 pulp: celery.worker.consumer:INFO: missed heartbeat from resource_manager@pulp01<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-3@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-1@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-2@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:46 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'reserved_resource_worker-0@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:05:56 pulp01 pulp: pulp.server.async.worker_watcher:INFO: New worker 'resource_manager@pulp01' discovered<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-3@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-3@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-1@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-1@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-2@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-2@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'reserved_resource_worker-0@pulp01' has gone missing, removing from list of work<o:p></o:p></p>
<p class="MsoNormal">ers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named reserved_resource_worker-0@pulp01 is missing. Canceling the tasks in its q<o:p></o:p></p>
<p class="MsoNormal">ueue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: Worker 'resource_manager@pulp01' has gone missing, removing from list of workers<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.tasks:ERROR: The worker named resource_manager@pulp01 is missing. Canceling the tasks in its queue.<o:p></o:p></p>
<p class="MsoNormal">Nov 18 09:06:46 pulp01 pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without<o:p></o:p></p>
<p class="MsoNormal">at least one pulp_resource_mananger process running.<o:p></o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication
is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error.<br>
</font>
</body>
</html>