<div dir="ltr">I have the same issues with 2.5 and latest OSAD packages... the connection still looks like it's established at the client side, but for some reason it has stopped trying to send data. The master no longer sees the connection as open and therefore cannot send anything to it.<div><br></div><div>The only resolution I've found is to restart the client(s), but for so many systems this caused the dispatchers to become unresponsive during our maintenance windows. Essentially, Puppet would run, restart OSAD, and it would consume all the HTTP connections and make the GUI unresponsive. Update and reboot actions were picked up outside of the scheduled maintenance, and it was all around chaos.<div><br></div><div>So at this point I'm stuck babysitting OSAD status of systems because there is nothing easily found in /var/log/osad that indicates an issue, even though the client still has 5222 open to the dispatcher and the osad service is running. In the Spacewalk database, the system is marked down... I ran an strace on the OSAD process on the client for about 30 minutes, and didn't see any attempts to do anything.</div><div><br></div><div><div>[me@osad-client1 ~]$ sudo lsof -Pp 21996 | grep TCP</div><div>osad 21996 root 3u IPv4 7392569 0t0 TCP osad-client1:56939->spacewalk-master:5222 (ESTABLISHED)</div><div>[me@osad-client1 ~]$ service osad status</div><div>osad (pid 21996) is running...</div><div>[me@osad-client1 ~]$ sudo lsof -Pp 21996 | grep TCP</div><div>osad 21996 root 3u IPv4 7392569 0t0 TCP osad-client1:56939->spacewalk-master:5222 (ESTABLISHED)</div><div><div>[me@osad-client1 ~]$ sudo strace -fp 21996</div><div>Process 21996 attached</div><div>select(4, [3], [], [], NULL</div></div><div><br></div><div>---</div><div>rhnschema=# select <a href="http://s.name" target="_blank">s.name</a>,pc.state_id from rhnpushclient pc, rhnserver s where <a href="http://s.name" target="_blank">s.name</a>='osad-client1' and pc.server_id=<a href="http://s.id" target="_blank">s.id</a>;</div><div> name | state_id </div><div>-----------------------+----------</div><div> osad-client1 | 2</div><div>(1 row)</div></div><div><br></div><div>Even though osad-client1 thought it was still connected, the master didn't have a corresponding connection on 5222:</div><div><div>[me@spacewalk-master ~]$ netstat -a | grep osad-client1</div><div>[me@spacewalk-master ~]$</div></div><div><br></div><div>For me, changing the values in /etc/jabberd/*.xml as recommended in <a href="https://fedorahosted.org/spacewalk/wiki/JabberAndOSAD" target="_blank">https://fedorahosted.org/spacewalk/wiki/JabberAndOSAD</a> wasn't going to work... I tried that and all systems would be disconnected, then would reconnect, causing some (perhaps insignificant) load on the database as well as unnecessary network traffic and client processing. I could see the number systems marked as "online" in the database flapping wildly between 1,000 and 5,000 over time.</div><div><br></div><div>One thing I did notice on the systems that were marked offline... a netstat showed two connections, one in CLOSE_WAIT status and another in ESTABLISHED. On restart of OSAD, only one was there, in ESTABLISHED state and the system was marked online again.</div></div><div><br></div><div>I'm thinking that the OSAD Python code isn't closing the sockets properly when an error is encountered, and leaves the client thinking it's still connected, while the master doesn't have a corresponding connection to send data to.</div><div><br></div><div>Basically, as a workaround, I think I'm going to have systems restart OSAD if they see connections open on 5222 in CLOSE_WAIT status... until something better comes along and the client code is fixed up. Unfortunately the workaround isn't even a full one... not every system had multiple connections, but it's a step toward more systems staying usable than before.</div><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 1, 2016 at 1:26 PM Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">2.4, I tried, actually after I did spacewalk-service restart it helped for one day.<div><br></div><div>Now it's the same, but no any errors on both sides.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 31, 2016 at 9:06 AM, Matthew Madey <span dir="ltr"><<a href="mailto:mattmadey@gmail.com" target="_blank">mattmadey@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p dir="ltr">What version of Spacewalk are you running? You likely need to reset the osad credentials on the clients. This typically only occurs when the jabber database has been corrupted. </p> <p dir="ltr">On the clients, run the below commands:</p><p dir="ltr"><br></p><p>rm -f /etc/sysconfig/rhn/osad-auth.conf ; service osad restart </p><p>You may find the below links helpful</p><p><a href="https://fedorahosted.org/spacewalk/wiki/OsadHowTo" target="_blank">https://fedorahosted.org/spacewalk/wiki/OsadHowTo</a><br></p><p><a href="https://fedorahosted.org/spacewalk/wiki/JabberAndOSAD" target="_blank">https://fedorahosted.org/spacewalk/wiki/JabberAndOSAD</a><br></p><p><br></p><p><br></p> <div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Aug 30, 2016 4:43 PM, "Konstantin Raskoshnyi" <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<br type="attribution"></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div dir="ltr">Something strange with some of my osad clients ~1/3<div><br></div><div>They don't pickup any jobs from osa-dispatcher, no any errors during starting the service,</div><div><br></div><div>also if I restart osad on sp I see logs:</div><div><br></div><div><div>Aug 30 14:32:20 spacewalk15 jabberd/c2s[51907]: [142] [::ffff:172.90.7.220, port=43046] disconnect jid=osad-e43e3265db@spacewalk15.ooma.internal/osad, packets: 29, bytes: 3738</div><div>Aug 30 14:32:20 spacewalk15 jabberd/sm[51904]: session ended: jid=osad-e43e3265db@spacewalk15.ooma.internal/osad</div><div>Aug 30 14:32:20 spacewalk15 jabberd/sm[51904]: user unloaded jid=osad-e43e3265db@spacewalk15.ooma.internal</div><div>Aug 30 14:32:20 spacewalk15 jabberd/c2s[51907]: [142] traditional.digest authentication succeeded: osad-e43e3265db@/osad ::ffff:<a href="http://172.90.7.220:43454" target="_blank">172.90.7.220:43454</a> TLS</div><div>Aug 30 14:32:20 spacewalk15 jabberd/c2s[51907]: [142] requesting session: jid=osad-e43e3265db@spacewalk15.ooma.internal/osad</div><div>Aug 30 14:32:20 spacewalk15 jabberd/sm[51904]: session started: jid=osad-e43e3265db@spacewalk15.ooma.internal/osad</div></div><div><br></div><div>So looks like everything should be fine</div></div> <br></div></div>_______________________________________________<br> Spacewalk-list mailing list<br> <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><br></blockquote></div></div> </div> <br>_______________________________________________<br> Spacewalk-list mailing list<br> <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a><br></blockquote></div><br></div> _______________________________________________<br> Spacewalk-list mailing list<br> <a href="mailto:Spacewalk-list@redhat.com" target="_blank">Spacewalk-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a></blockquote></div></div></div>