[EnMasse] Disconnect from time to time

Wed Mar 27 09:07:26 UTC 2019

Hi Ulf,

These are the logs from the admin pod:

    2019-03-21T00:10:05.458Z INFO  [AddressController] total: 435848727 ns, resolvedPlan: 7329 ns, calculatedUsage: 66505 ns, checkedQuota: 9831 ns, listClusters: 16603774 ns, provisionResources: 5975 ns, checkStatuses: 419078275 ns, deprovisionUnused: 11260 ns, upgradeClusters: 59877 ns, replaceAddresses: 4197 ns, gcTerminating: 1704 ns
    2019-03-21T00:10:35.458Z INFO  [AddressController] Total: 6, Active: 6, Configuring: 0, Pending: 0, Terminating: 0, Failed: 0

The logs from the qdrouterd is the one I already sent you (other logs from the qdrouterd are gone since I restarted the pod to enable PN_TRACE_FRM, if I can collect the logs from a stopped stateful set, please advise me:) ).

    2019-03-21 10:30:20.777139 +0000 ROUTER_CORE (info) Link Route Deactivated 'ragent-gw-gw-out' on container gw-out

Kind regards,
Bob

On 3/27/19 9:54 AM, Ulf Lilleengen wrote:
Hi Bob,

Do you happen to have the logs of the qdrouterd and admin pods as well? This seems to indicate one of the ends of router and broker is closing the TCP connection.

Best regards,

Ulf

On Wed, Mar 27, 2019 at 9:26 AM Bob Claerhout <bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io>> wrote:
Hi all,

Hereby the logging of the broker of this event.
2019-03-21T10:30:20.778Z INFO  [ProtonClientConnectionManager] Connection messaging-aloxy/172.30.9.50:56671<http://172.30.9.50:56671> destroyed
2019-03-21T10:30:20.778Z WARN  [server] AMQ222061: Client connection failed, clearing up resources for session a4ebe8f7-4676-11e9-bac7-0a580a800051
2019-03-21T10:30:20.988Z WARN  [server] AMQ222107: Cleared up resources for session a4ebe8f7-4676-11e9-bac7-0a580a800051
2019-03-21T10:30:20.988Z INFO  [amqp] connectionDestroyed for connector address-connector-out
2019-03-21T10:30:25.988Z INFO  [amqp] Starting connector address-connector-out
2019-03-21T10:30:25.999Z INFO  [amqp] connectionCreated for connector address-connector-out
2019-03-21T10:30:25.999Z INFO  [ProtonClientConnectionManager] Connection messaging-aloxy/172.30.9.50:56671<http://172.30.9.50:56671> created

The broker has not restarted.

Kind regards,
Bob

On 3/22/19 6:02 PM, Ulf Lilleengen wrote:
Hi Bob,

Regarding Gordon and Teds reply, do you have the logs from the broker? Alternatively could you also check if it got restarted for some reason?

Best regards,

Ulf

fre. 22. mar. 2019 kl. 12:19 skrev Bob Claerhout <bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io>>:
Hi Ulf,

Thanks for the quick response.
I've enabled this tracing. I'll come back to this when it occurs again.

Enjoy the weekend
Kind regards,
Bob

On 3/22/19 11:10 AM, Ulf Lilleengen wrote:
Hi Bob,

If your client is based on qpid-proton, turning on frame tracing might give some more info on why it is disconnected. You can do that by setting the environment PN_TRACE_FRM=1 for your client (I think this works with most qpid clients).

Another alternative is to manually edit the qdrouterd statefulset on your cluster and add this env variable to the router to get more info in the router logs:

kubectl set env statefulset/qdrouterd-$uuid PN_TRACE_FRM=1

Best regards,

Ulf

On Fri, Mar 22, 2019 at 10:55 AM Bob Claerhout <bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io>> wrote:
Hi all,

From time to time we experience a disconnect from enmasse. I'm trying to figure out what's actually happening. In the logs I'm getting following line:
    `2019-03-21 10:30:20.777139 +0000 ROUTER_CORE (info) Link Route Deactivated 'ragent-gw-gw-out' on container gw-out`.

The address I'm using is called "gw". Can you point me towards somehwere I can find more information about why this link route is deactivated?
Can I enable debug logging somewhere? Is it logged somewhere why a link is deactivated?

Kind regards,
Bob
--

[cid:part6.80F351B3.C03C5562 at aloxy.io]  Bob Claerhout
Software developer
M +32 479 34 84 92
The Beacon, Sint-Pietersvliet 7<https://maps.google.com/?q=Sint-Pietersvliet+7&entry=gmail&source=g>, 2000
Antwerp
bob.claerhout at aloxy.io<mailto:bob.claerhout at aloxy.io> | www.aloxy.io<http://www.aloxy.io>
_______________________________________________
enmasse mailing list
enmasse at redhat.com<mailto:enmasse at redhat.com>
https://www.redhat.com/mailman/listinfo/enmasse

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190327/5a78d330/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cmfaafbecndjnoom.png
Type: image/png
Size: 6330 bytes
Desc: cmfaafbecndjnoom.png
URL: <http://listman.redhat.com/archives/enmasse/attachments/20190327/5a78d330/attachment.png>