<div dir="ltr">Hi All,<div><br></div><div>Been testing this to see if I could potentially recreate the fault. </div><div><br></div><div>I have taken a copy of the instance which was working (vanilla install with spacewalk applied). When the copy starts up. It appears to be loaded and running fine. When I made a change to the server. For example, the hostname. The OSAD failed after reboot. </div><div>I then made another machine from known working copy and changed the sshd_config file. Rebooted the machine and osad then failed.again.</div><div><br></div><div>The failure seems to be the same, and there is nothing within the logs which suggest why its failed. It appears to be a time out. anything else I could potentially look into? </div><div><br></div><div>Thanks</div><div><br></div><div>Ray <br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <strong class="gmail_sendername" dir="auto">Raymond Setchfield</strong> <span dir="auto"><<a href="mailto:raymond.setchfield@gmail.com">raymond.setchfield@gmail.com</a>></span><br>Date: Tue, Nov 5, 2019 at 10:00 AM<br>Subject: osa-dispatcher.service failed<br>To: <<a href="mailto:spacewalk-list@redhat.com">spacewalk-list@redhat.com</a>><br></div><br><br><div dir="ltr">Hi All,<br><div><br></div><div>I have been following this guide to install the SSL certificate, and it appears to have worked fine. As I can see the certificate when logging into spacewalk etc. </div><div><br></div><div>However, when restarting the services it appears that the osa-dispatcher has failed. There is little or no information to go with to figure out why it has failed. </div><div><br></div><div>● postgresql.service - PostgreSQL database server<br> Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2019-11-05 09:50:15 UTC; 4min 41s ago<br> Process: 12341 ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast (code=exited, status=0/SUCCESS)<br> Process: 12504 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)<br> Process: 12487 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)<br> Main PID: 12518 (postgres)<br> CGroup: /system.slice/postgresql.service<br> ├─12518 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432<br> ├─12529 postgres: logger process<br> ├─12531 postgres: checkpointer process<br> ├─12532 postgres: writer process<br> ├─12533 postgres: wal writer process<br> ├─12534 postgres: autovacuum launcher process<br> ├─12535 postgres: stats collector process<br> ├─12590 postgres: rhnuser rhnschema 127.0.0.1(44062) idle<br> ├─12591 postgres: rhnuser rhnschema 127.0.0.1(44065) idle<br> ├─12592 postgres: rhnuser rhnschema 127.0.0.1(44064) idle<br> ├─12593 postgres: rhnuser rhnschema 127.0.0.1(44068) idle<br> ├─12594 postgres: rhnuser rhnschema 127.0.0.1(44070) idle<br> ├─12762 postgres: rhnuser rhnschema 127.0.0.1(44074) idle<br> ├─12867 postgres: rhnuser rhnschema 127.0.0.1(44076) idle<br> ├─12868 postgres: rhnuser rhnschema 127.0.0.1(44078) idle<br> ├─12869 postgres: rhnuser rhnschema 127.0.0.1(44080) idle<br> ├─12870 postgres: rhnuser rhnschema 127.0.0.1(44082) idle<br> ├─12871 postgres: rhnuser rhnschema 127.0.0.1(44084) idle<br> ├─12872 postgres: rhnuser rhnschema 127.0.0.1(44086) idle<br> ├─12873 postgres: rhnuser rhnschema 127.0.0.1(44088) idle<br> ├─12874 postgres: rhnuser rhnschema 127.0.0.1(44090) idle<br> ├─12875 postgres: rhnuser rhnschema 127.0.0.1(44092) idle<br> ├─12876 postgres: rhnuser rhnschema 127.0.0.1(44094) idle<br> ├─12877 postgres: rhnuser rhnschema 127.0.0.1(44096) idle<br> ├─12878 postgres: rhnuser rhnschema 127.0.0.1(44098) idle<br> ├─12879 postgres: rhnuser rhnschema 127.0.0.1(44100) idle<br> ├─12880 postgres: rhnuser rhnschema 127.0.0.1(44102) idle<br> ├─12881 postgres: rhnuser rhnschema 127.0.0.1(44104) idle<br> ├─12887 postgres: rhnuser rhnschema 127.0.0.1(44106) idle<br> ├─12888 postgres: rhnuser rhnschema 127.0.0.1(44108) idle<br> ├─12889 postgres: rhnuser rhnschema 127.0.0.1(44110) idle<br> ├─12890 postgres: rhnuser rhnschema 127.0.0.1(44112) idle<br> ├─12891 postgres: rhnuser rhnschema 127.0.0.1(44114) idle<br> ├─12930 postgres: rhnuser rhnschema 127.0.0.1(44118) idle<br> ├─12931 postgres: rhnuser rhnschema 127.0.0.1(44120) idle<br> ├─12941 postgres: rhnuser rhnschema 127.0.0.1(44126) idle<br> ├─12942 postgres: rhnuser rhnschema 127.0.0.1(44128) idle<br> └─12943 postgres: rhnuser rhnschema 127.0.0.1(44130) idle<br><br>● jabberd.service - Jabber Server<br> Loaded: loaded (/usr/lib/systemd/system/jabberd.service; enabled; vendor preset: disabled)<br> Active: active (exited) since Tue 2019-11-05 09:50:14 UTC; 4min 42s ago<br> Process: 12486 ExecStart=/bin/true (code=exited, status=0/SUCCESS)<br> Main PID: 12486 (code=exited, status=0/SUCCESS)<br> CGroup: /system.slice/jabberd.service<br><br>● tomcat.service - Apache Tomcat Web Application Container<br> Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2019-11-05 09:50:14 UTC; 4min 42s ago<br> Main PID: 12480 (java)<br> CGroup: /system.slice/tomcat.service<br> └─12480 /usr/lib/jvm/jre/bin/java -ea -Xms256m -Xmx256m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dorg.apache.tomcat.util.htt...<br><br>● spacewalk-wait-for-tomcat.service - Spacewalk wait for tomcat<br> Loaded: loaded (/usr/lib/systemd/system/spacewalk-wait-for-tomcat.service; static; vendor preset: disabled)<br> Active: active (exited) since Tue 2019-11-05 09:50:31 UTC; 4min 25s ago<br> Process: 12481 ExecStart=/usr/sbin/spacewalk-startup-helper wait-for-tomcat (code=exited, status=0/SUCCESS)<br> Main PID: 12481 (code=exited, status=0/SUCCESS)<br> CGroup: /system.slice/spacewalk-wait-for-tomcat.service<br><br>● httpd.service - The Apache HTTP Server<br> Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2019-11-05 09:50:31 UTC; 4min 24s ago<br> Docs: man:httpd(8)<br> man:apachectl(8)<br> Process: 12464 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)<br> Main PID: 12676 (httpd)<br> Status: "Total requests: 13; Current requests/sec: 0; Current traffic: 0 B/sec"<br> CGroup: /system.slice/httpd.service<br> ├─12676 /usr/sbin/httpd -DFOREGROUND<br> ├─12677 /usr/sbin/httpd -DFOREGROUND<br> ├─12678 /usr/sbin/httpd -DFOREGROUND<br> ├─12680 /usr/sbin/httpd -DFOREGROUND<br> ├─12681 /usr/sbin/httpd -DFOREGROUND<br> ├─12683 /usr/sbin/httpd -DFOREGROUND<br> ├─12685 /usr/sbin/httpd -DFOREGROUND<br> ├─12686 /usr/sbin/httpd -DFOREGROUND<br> └─12687 /usr/sbin/httpd -DFOREGROUND<br><br>● spacewalk-wait-for-jabberd.service - Spacewalk wait for jabberd<br> Loaded: loaded (/usr/lib/systemd/system/spacewalk-wait-for-jabberd.service; static; vendor preset: disabled)<br> Active: active (exited) since Tue 2019-11-05 09:50:14 UTC; 4min 42s ago<br> Process: 12490 ExecStart=/usr/sbin/spacewalk-startup-helper wait-for-jabberd (code=exited, status=0/SUCCESS)<br> Main PID: 12490 (code=exited, status=0/SUCCESS)<br> CGroup: /system.slice/spacewalk-wait-for-jabberd.service<br><br>● osa-dispatcher.service - OSA Dispatcher daemon<br> Loaded: loaded (/usr/lib/systemd/system/osa-dispatcher.service; enabled; vendor preset: disabled)<br> Active: failed (Result: timeout) since Tue 2019-11-05 09:51:45 UTC; 3min 11s ago<br> Process: 12544 ExecStart=/usr/sbin/osa-dispatcher --pid-file /var/run/osa-dispatcher.pid (code=killed, signal=TERM)<br> Process: 12542 ExecStartPre=/bin/rm -f /var/run/osa-dispatcher.pid (code=exited, status=0/SUCCESS)<br><br>● rhn-search.service - Spacewalk search engine<br> Loaded: loaded (/usr/lib/systemd/system/rhn-search.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2019-11-05 09:50:32 UTC; 4min 24s ago<br> Process: 12350 ExecStop=/usr/sbin/rhn-search stop (code=exited, status=0/SUCCESS)<br> Process: 12679 ExecStart=/usr/sbin/rhn-search start (code=exited, status=0/SUCCESS)<br> Main PID: 12703 (rhnsearchd)<br> CGroup: /system.slice/rhn-search.service<br> ├─12703 /usr/bin/rhnsearchd /usr/share/rhn/config-defaults/rhn_search_daemon.conf wrapper.pidfile=/var/run/rhn-search.pid wrapper.daemonize=TRUE<br> └─12716 /usr/bin/java -Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/11.2/client/lib:/usr/lib/oracle/11.2/client64/lib:/usr/lib/gcj/postgresql-jdbc:/usr/lib6...<br><br>● cobblerd.service - Cobbler daemon<br> Loaded: loaded (/usr/lib/systemd/system/cobblerd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2019-11-05 09:50:31 UTC; 4min 24s ago<br> Main PID: 12682 (cobblerd)<br> CGroup: /system.slice/cobblerd.service<br> └─12682 /usr/bin/python2 -s /usr/bin/cobblerd --no-daemonize<br><br>● taskomatic.service - Taskomatic<br> Loaded: loaded (/usr/lib/systemd/system/taskomatic.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2019-11-05 09:50:32 UTC; 4min 24s ago<br> Process: 12348 ExecStop=/usr/sbin/taskomatic stop (code=exited, status=0/SUCCESS)<br> Process: 12684 ExecStart=/usr/sbin/taskomatic start (code=exited, status=0/SUCCESS)<br> Main PID: 12714 (taskomaticd)<br> CGroup: /system.slice/taskomatic.service<br> ├─12714 /usr/bin/taskomaticd /usr/share/rhn/config-defaults/rhn_taskomatic_daemon.conf wrapper.pidfile=/var/run//taskomatic.pid wrapper.daemonize=TRUE<br> └─12720 /usr/bin/java -Dibm.dst.compatibility=true -Dfile.encoding=UTF-8 -Xms256m -Xmx1024m -Djava.library.path=/usr/lib:/usr/lib64:/usr/lib/oracle/11.2/client64/lib:...<br>● spacewalk.target - Spacewalk<br> Loaded: loaded (/usr/lib/systemd/system/spacewalk.target; enabled; vendor preset: disabled)<br> Active: inactive (dead)<br></div><div><br></div><div>[root@smart-cent-mgmt ~]# tail -f /var/log/rhn/osa-dispatcher.log<br>2019/11/04 15:43:47 -00:00 1160 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 08:49:32 -00:00 7952 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 08:53:00 -00:00 8449 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 08:58:38 -00:00 8738 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 09:14:49 -00:00 9621 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 09:19:31 -00:00 10318 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 09:25:15 -00:00 11420 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 09:37:06 -00:00 12049 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 09:43:39 -00:00 12162 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br>2019/11/05 09:50:15 -00:00 12544 <a href="http://0.0.0.0" target="_blank">0.0.0.0</a>: osad/jabber_lib.__init__<br></div><div><br></div><div>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> systemd[1]: osa-dispatcher.service start operation timed out. Terminating.<br>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> systemd[1]: Failed to start OSA Dispatcher daemon.<br>-- Subject: Unit osa-dispatcher.service has failed<br>-- Defined-By: systemd<br>-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><br>--<br>-- Unit osa-dispatcher.service has failed.<br>--<br>-- The result is failed.<br>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> systemd[1]: Dependency failed for Spacewalk.<br>-- Subject: Unit spacewalk.target has failed<br>-- Defined-By: systemd<br>-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><br>--<br>-- Unit spacewalk.target has failed.<br>--<br>-- The result is dependency.<br>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> systemd[1]: Job spacewalk.target/start failed with result 'dependency'.<br>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> systemd[1]: Unit osa-dispatcher.service entered failed state.<br>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> systemd[1]: osa-dispatcher.service failed.<br>Nov 05 09:51:45 <a href="http://cent-mgmt.domain.com" target="_blank">cent-mgmt.domain.com</a> polkitd[653]: Unregistered Authentication Agent for unix-process:12474:6519546 (system bus name :1.133, object path /org/freedesktop/PolicyKit1/Au<br></div><div><br></div><div>Can any please advise? The only thing which I can think of is a change in hostname which took place before this, and this was ran using "spacewalk-hostname-rename" command</div><div><br></div><div>Thanks</div><div><br></div><div>Ray </div><div><br></div></div> </div></div></div>