<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 06/05/2013 12:14 AM, Gary Kotton
wrote:<br>
</div>
<blockquote cite="mid:51AEE572.70409@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
On 06/05/2013 01:17 AM, Steven Dake wrote:
<blockquote cite="mid:51AE6783.5040602@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 06/04/2013 01:09 PM, S Manoo
wrote:<br>
</div>
<blockquote
cite="mid:CA+iL8QBFr3A28++0cHrFfhBXGr+LvhakNhtZbA_ieW1bZRbp1Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Looking into this further, I'm observing the same error
message relating to timeouts talking to qpid in
dhcp-agent.log after every restart, perhaps this is why
I'm unable to get any dhcp responses to instances? Any
suggestions on what's causing this and where I might look
to troubleshoot this further?<br>
</div>
</div>
</blockquote>
</blockquote>
<br>
When one restarts a host each process needs to register with the
message broker. If you are running all of the services on the same
host then they will only be able to connect when the qpid service
is up and running. This usually takes a few seconds after reboot.
If a service does not receive an answer from the qpid service then
it will wait and retry again. This is why you see the timeouts.
The wait is incremental. I have seen that all service are usually
able to connect within a minute of booting a host (we should try
and reduce this time).<br>
<br>
Please note that the quantum cli has an option: quantum
agent-list. This provides the list of agents, their status and
hosts that they are running on.<br>
<br>
If you spin up an instance after the dhcp agent is up and running
do you see the problem?<br>
<br>
<blockquote cite="mid:51AE6783.5040602@redhat.com" type="cite">
<blockquote
cite="mid:CA+iL8QBFr3A28++0cHrFfhBXGr+LvhakNhtZbA_ieW1bZRbp1Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div> <br>
</div>
</div>
</blockquote>
S Manoo,<br>
<br>
We may have just fixed a bug related to this problem which is
not fixed in the preview. Please try the workaround in this
bugzilla:<br>
<br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=970453">https://bugzilla.redhat.com/show_bug.cgi?id=970453</a><br>
</blockquote>
<br>
This fix is good for an all in one setup but will not help if the
DHCP agent is running on another host. In Quantum we have the
notion of a network node. Please look at
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://docs.google.com/drawings/d/167gegaoTBZpd318b2JTgF_Qi9YdkIX8pcQ6YBJLUtGY/edit?usp=sharing">https://docs.google.com/drawings/d/167gegaoTBZpd318b2JTgF_Qi9YdkIX8pcQ6YBJLUtGY/edit?usp=sharing</a><br>
<br>
If the message broker goes down (say for example host reboot or
network problems) then the dhcp agent will try and reconnect.<br>
<br>
</blockquote>
Gary,<br>
<br>
I have found dhcp agent stops responding permanently in this
condition on a all in one setup. Perhaps the same is true for
multinode (ie the retry logic doesn't work as expected). I don't
have multiple nodes to test, but might be worth double-checking if
you do.<br>
<br>
Regards<br>
-steve<br>
<br>
<blockquote cite="mid:51AEE572.70409@redhat.com" type="cite">
<blockquote cite="mid:51AE6783.5040602@redhat.com" type="cite"> <br>
Regards<br>
-steve<br>
<br>
<br>
<blockquote
cite="mid:CA+iL8QBFr3A28++0cHrFfhBXGr+LvhakNhtZbA_ieW1bZRbp1Q@mail.gmail.com"
type="cite">
<div dir="ltr"><b>/var/log/quantum/dhcp-agent.log:</b><br>
<div>2013-06-04 12:50:44 INFO [quantum.common.config]
Logging enabled!<br>
2013-06-04 12:50:44 INFO
[quantum.openstack.common.rpc.impl_qpid] Connected to AMQP
server on localhost:5672<br>
2013-06-04 12:50:44 INFO
[quantum.openstack.common.rpc.impl_qpid] Connected to AMQP
server on localhost:5672<br>
2013-06-04 12:50:44 INFO [quantum.agent.dhcp_agent]
DHCP agent started<br>
2013-06-04 12:51:44 ERROR [quantum.agent.dhcp_agent]
Failed reporting state!<br>
Traceback (most recent call last):<br>
File
"/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py",
line 700, in _report_state<br>
self.agent_state)<br>
File
"/usr/lib/python2.6/site-packages/quantum/agent/rpc.py",
line 66, in report_state<br>
topic=self.topic)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py",
line 80, in call<br>
return rpc.call(context, self._get_topic(topic), msg,
timeout)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py",
line 140, in call<br>
return _get_impl().call(CONF, context, topic, msg,
timeout)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 611, in call<br>
rpc_amqp.get_connection_pool(conf, Connection))<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
line 613, in call<br>
rv = list(rv)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
line 555, in __iter__<br>
self.done()<br>
File "/usr/lib64/python2.6/contextlib.py", line 23, in
__exit__<br>
self.gen.next()<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
line 552, in __iter__<br>
self._iterator.next()<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 436, in iterconsume<br>
yield self.ensure(_error_callback, _consume)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 380, in ensure<br>
error_callback(e)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 421, in _error_callback<br>
raise rpc_common.Timeout()<br>
Timeout: Timeout while waiting on RPC response.<br>
2013-06-04 12:51:44 WARNING
[quantum.openstack.common.loopingcall] task run outlasted
interval by 56.108887 sec<br>
2013-06-04 12:51:44 INFO [quantum.agent.dhcp_agent]
Synchronizing state<br>
<br>
<br>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On Mon, Jun 3, 2013 at
11:28 PM, S Manoo <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:smanoo76@gmail.com"
target="_blank">smanoo76@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<br>
<div><b>dhcp-agent.log:</b><br>
[root@grizzly ~(keystone_admin)]# cat
dhcp-agent.log <br>
2013-06-03 22:27:09 INFO
[quantum.common.config] Logging enabled!<br>
2013-06-03 22:27:09 INFO
[quantum.openstack.common.rpc.impl_qpid]
Connected to AMQP server on <a
moz-do-not-send="true"
href="http://10.0.0.19:5672"
target="_blank">10.0.0.19:5672</a><br>
2013-06-03 22:27:09 INFO
[quantum.openstack.common.rpc.impl_qpid]
Connected to AMQP server on <a
moz-do-not-send="true"
href="http://10.0.0.19:5672"
target="_blank">10.0.0.19:5672</a><br>
2013-06-03 22:27:10 INFO
[quantum.agent.dhcp_agent] DHCP agent
started<br>
2013-06-03 22:28:10 ERROR
[quantum.agent.dhcp_agent] Failed reporting
state!<br>
Traceback (most recent call last):<br>
File
"/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py",
line 700, in _report_state<br>
self.agent_state)<br>
File
"/usr/lib/python2.6/site-packages/quantum/agent/rpc.py",
line 66, in report_state<br>
topic=self.topic)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py",
line 80, in call<br>
return rpc.call(context,
self._get_topic(topic), msg, timeout)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py",
line 140, in call<br>
return _get_impl().call(CONF, context,
topic, msg, timeout)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 611, in call<br>
rpc_amqp.get_connection_pool(conf,
Connection))<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
line 613, in call<br>
rv = list(rv)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
line 555, in __iter__<br>
self.done()<br>
File "/usr/lib64/python2.6/contextlib.py",
line 23, in __exit__<br>
self.gen.next()<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py",
line 552, in __iter__<br>
self._iterator.next()<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 436, in iterconsume<br>
yield self.ensure(_error_callback,
_consume)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 380, in ensure<br>
error_callback(e)<br>
File
"/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py",
line 421, in _error_callback<br>
raise rpc_common.Timeout()<br>
Timeout: Timeout while waiting on RPC
response.<br>
2013-06-03 22:28:10 WARNING
[quantum.openstack.common.loopingcall] task
run outlasted interval by 56.133099 sec<br>
2013-06-03 22:28:10 INFO
[quantum.agent.dhcp_agent] Synchronizing
state<br>
[root@grizzly ~(keystone_admin)]#<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
rhos-list mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:rhos-list@redhat.com">rhos-list@redhat.com</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/rhos-list">https://www.redhat.com/mailman/listinfo/rhos-list</a></pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
rhos-list mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:rhos-list@redhat.com">rhos-list@redhat.com</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/rhos-list">https://www.redhat.com/mailman/listinfo/rhos-list</a></pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>