[Spacewalk-list] 2.5 upgrade breaks osa-dispatcher, Tomcat

McCleary, Matthew matthew.mccleary at bia.gov
Wed Jun 8 19:13:45 UTC 2016 

Greetings,

I went through the 2.4-to-2.5 upgrade process earlier today and it went
smoothly, but afterward osa-dispatcher and RHN taskomatic refused to start
and this seemed to have a cascade effect on Tomcat -- it would only display
blank Web pages when I'd try to access my Spacewalk installation.

Fortunately, no harm done since I took a snapshot prior to the upgrade, but
I don't want to be stuck at version 2.4 forever. Wondering if anyone else
is experiencing similar issues.

Lots of error output follows:

[root at iiamabqzpcm02p log]# spacewalk-service restart
Shutting down spacewalk services...
Stopping RHN Taskomatic...
RHN Taskomatic was not running.
Stopping cobblerd (via systemctl):                         [  OK  ]
Redirecting to /bin/systemctl stop  rhn-search.service
Redirecting to /bin/systemctl stop  osa-dispatcher.service
Redirecting to /bin/systemctl stop  httpd.service
Redirecting to /bin/systemctl stop  tomcat.service
Redirecting to /bin/systemctl stop  jabberd.service
Redirecting to /bin/systemctl stop  postgresql.service
Done.
Starting spacewalk services...
Redirecting to /bin/systemctl start  postgresql.service
Redirecting to /bin/systemctl start  jabberd.service
Job for jabberd.service canceled.
Redirecting to /bin/systemctl start  tomcat.service
Waiting for tomcat to be ready ...
Redirecting to /bin/systemctl start  httpd.service
Redirecting to /bin/systemctl start  osa-dispatcher.service
Job for osa-dispatcher.service failed because the control process exited
with error code. See "systemctl status osa-dispatcher.service" and
"journalctl -xe" for details.
Redirecting to /bin/systemctl start  rhn-search.service
Starting cobblerd (via systemctl):                         [  OK  ]
Starting RHN Taskomatic...
Done.
[root at iiamabqzpcm02p log]# spacewalk-service status
Redirecting to /bin/systemctl status  postgresql.service
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled;
vendor preset: disabled)
   Active: active (running) since Wed 2016-06-08 12:57:38 MDT; 1min 20s ago
  Process: 14245 ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast
(code=exited, status=0/SUCCESS)
  Process: 14273 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p
${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 14267 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA}
(code=exited, status=0/SUCCESS)
 Main PID: 14276 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─14276 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─14277 postgres: logger process
           ├─14279 postgres: checkpointer process
           ├─14280 postgres: writer process
           ├─14281 postgres: wal writer process
           ├─14282 postgres: autovacuum launcher process
           └─14283 postgres: stats collector process

Jun 08 12:57:37 iiamabqzpcm02p.ia.doi.net systemd[1]: Starting PostgreSQL
dat...
Jun 08 12:57:38 iiamabqzpcm02p.ia.doi.net systemd[1]: Started PostgreSQL
data...
Hint: Some lines were ellipsized, use -l to show in full.
Redirecting to /bin/systemctl status  jabberd.service
● jabberd.service - Jabber Server
   Loaded: loaded (/usr/lib/systemd/system/jabberd.service; enabled; vendor
preset: disabled)
   Active: inactive (dead) since Wed 2016-06-08 12:57:38 MDT; 1min 20s ago
  Process: 14304 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 14304 (code=exited, status=0/SUCCESS)

Jun 08 12:57:38 iiamabqzpcm02p.ia.doi.net systemd[1]: Starting Jabber
Server...
Jun 08 12:57:38 iiamabqzpcm02p.ia.doi.net systemd[1]: Stopped Jabber Server.
Jun 08 12:57:38 iiamabqzpcm02p.ia.doi.net systemd[1]: Stopping Jabber
Server...
Hint: Some lines were ellipsized, use -l to show in full.
Redirecting to /bin/systemctl status  tomcat.service
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor
preset: disabled)
   Active: active (running) since Wed 2016-06-08 12:57:38 MDT; 1min 20s ago
  Process: 14178 ExecStop=/usr/libexec/tomcat/server stop (code=exited,
status=0/SUCCESS)
 Main PID: 14324 (java)
   CGroup: /system.slice/tomcat.service
           └─14324 /usr/lib/jvm/jre/bin/java -ea -Xms256m -Xmx256m
-Djava.awt...

Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: Jun 08, 2016
12:58:4...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: INFO: Deployment
of ...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: Jun 08, 2016
12:58:4...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: INFO: Starting
Proto...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: Jun 08, 2016
12:58:4...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: INFO: Starting
Proto...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: Jun 08, 2016
12:58:4...
Jun 08 12:58:49 iiamabqzpcm02p.ia.doi.net server[14324]: INFO: Starting
Proto...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net server[14324]: Jun 08, 2016
12:58:5...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net server[14324]: INFO: Server
startup...
Hint: Some lines were ellipsized, use -l to show in full.
Redirecting to /bin/systemctl status  httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor
preset: disabled)
   Active: active (running) since Wed 2016-06-08 12:58:50 MDT; 8s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 14158 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited,
status=0/SUCCESS)
 Main PID: 14760 (httpd)
   Status: "Processing requests..."
   CGroup: /system.slice/httpd.service
           ├─14760 /usr/sbin/httpd -DFOREGROUND
           ├─14761 /usr/sbin/httpd -DFOREGROUND
           ├─14762 /usr/sbin/httpd -DFOREGROUND
           ├─14763 /usr/sbin/httpd -DFOREGROUND
           ├─14764 /usr/sbin/httpd -DFOREGROUND
           ├─14765 /usr/sbin/httpd -DFOREGROUND
           ├─14766 /usr/sbin/httpd -DFOREGROUND
           ├─14767 /usr/sbin/httpd -DFOREGROUND
           └─14768 /usr/sbin/httpd -DFOREGROUND

Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]: Starting The Apache
HTT...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]: Started The Apache
HTTP...
Hint: Some lines were ellipsized, use -l to show in full.
Redirecting to /bin/systemctl status  osa-dispatcher.service
● osa-dispatcher.service - OSA Dispatcher daemon
   Loaded: loaded (/usr/lib/systemd/system/osa-dispatcher.service; enabled;
vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-06-08 12:58:50 MDT; 8s
ago
  Process: 14786 ExecStart=/usr/sbin/osa-dispatcher --pid-file
/var/run/osa-dispatcher.pid (code=exited, status=255)
  Process: 14784 ExecStartPre=/bin/rm -f /var/run/osa-dispatcher.pid
(code=exited, status=0/SUCCESS)

Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]: Starting OSA
Dispatcher...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net osa-dispatcher[14786]: Spacewalk
14...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net osa-dispatcher[14786]: Spacewalk
14...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net osa-dispatcher[14786]: ERROR:
unhan...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]:
osa-dispatcher.service:...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]: Failed to start OSA
Dis...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]: Unit
osa-dispatcher.ser...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]:
osa-dispatcher.service ...
Hint: Some lines were ellipsized, use -l to show in full.
Redirecting to /bin/systemctl status  rhn-search.service
● rhn-search.service - Spacewalk search engine
   Loaded: loaded (/usr/lib/systemd/system/rhn-search.service; enabled;
vendor preset: disabled)
   Active: active (running) since Wed 2016-06-08 12:58:51 MDT; 7s ago
  Process: 14074 ExecStop=/usr/sbin/rhn-search stop (code=exited,
status=0/SUCCESS)
  Process: 14806 ExecStart=/usr/sbin/rhn-search start (code=exited,
status=0/SUCCESS)
 Main PID: 14819 (rhnsearchd)
   CGroup: /system.slice/rhn-search.service
           ├─14819 /usr/bin/rhnsearchd
/usr/share/rhn/config-defaults/rhn_sea...
           └─14821 /usr/bin/java
-Djava.library.path=/usr/lib:/usr/lib64:/usr...

Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net systemd[1]: Starting Spacewalk
sear...
Jun 08 12:58:50 iiamabqzpcm02p.ia.doi.net rhn-search[14806]: Starting
rhn-sea...
Jun 08 12:58:51 iiamabqzpcm02p.ia.doi.net systemd[1]: Started Spacewalk
searc...
Hint: Some lines were ellipsized, use -l to show in full.
● cobblerd.service - LSB: daemon for libvirt virtualization API
   Loaded: loaded (/etc/rc.d/init.d/cobblerd)
   Active: active (running) since Wed 2016-06-08 12:58:52 MDT; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14051 ExecStop=/etc/rc.d/init.d/cobblerd stop (code=exited,
status=0/SUCCESS)
  Process: 14861 ExecStart=/etc/rc.d/init.d/cobblerd start (code=exited,
status=0/SUCCESS)
   CGroup: /system.slice/cobblerd.service
           └─14871 /usr/bin/python -s /bin/cobblerd --daemonize

Jun 08 12:58:51 iiamabqzpcm02p.ia.doi.net systemd[1]: Starting LSB: daemon
fo...
Jun 08 12:58:52 iiamabqzpcm02p.ia.doi.net cobblerd[14861]: Starting cobbler
d...
Jun 08 12:58:52 iiamabqzpcm02p.ia.doi.net systemd[1]: Started LSB: daemon
for...
Hint: Some lines were ellipsized, use -l to show in full.
RHN Taskomatic is not running.

And

[root at iiamabqzpcm02p log]# grep osa-dispatcher messages
Jun  8 11:54:17 iiamabqzpcm02p yum[22900]: Updated:
osa-dispatcher-5.11.71-1.el7.noarch
Jun  8 11:54:45 iiamabqzpcm02p yum[22900]: Updated:
osa-dispatcher-selinux-5.11.71-1.el7.noarch
Jun  8 12:40:07 iiamabqzpcm02p osa-dispatcher: Spacewalk 30124 2016/06/08
12:40:07 -06:00: ('Error connecting to jabber server: Unable to connect to
the host and port specified. See https://access.redhat.com/solutions/327903
for more information. ',)
Jun  8 12:40:07 iiamabqzpcm02p osa-dispatcher: Spacewalk 30124 2016/06/08
12:40:07 -06:00: ('Error caught:',)
Jun  8 12:40:07 iiamabqzpcm02p osa-dispatcher: ERROR: unhandled exception
occurred: (unicode argument expected, got 'str').
Jun  8 12:40:07 iiamabqzpcm02p systemd: osa-dispatcher.service: control
process exited, code=exited status=255
Jun  8 12:40:07 iiamabqzpcm02p systemd: Unit osa-dispatcher.service entered
failed state.
Jun  8 12:40:07 iiamabqzpcm02p systemd: osa-dispatcher.service failed.
Jun  8 12:53:03 localhost osa-dispatcher: Spacewalk 1397 2016/06/08
12:53:03 -06:00: ('Error connecting to jabber server: Unable to connect to
the host and port specified. See https://access.redhat.com/solutions/327903
for more information. ',)
Jun  8 12:53:03 localhost osa-dispatcher: Spacewalk 1397 2016/06/08
12:53:03 -06:00: ('Error caught:',)
Jun  8 12:53:03 localhost osa-dispatcher: ERROR: unhandled exception
occurred: (unicode argument expected, got 'str').
Jun  8 12:53:03 localhost systemd: osa-dispatcher.service: control process
exited, code=exited status=255
Jun  8 12:53:03 localhost systemd: Unit osa-dispatcher.service entered
failed state.
Jun  8 12:53:03 localhost systemd: osa-dispatcher.service failed.
Jun  8 12:54:22 localhost osa-dispatcher: Spacewalk 13013 2016/06/08
12:54:22 -06:00: ('Error connecting to jabber server: Unable to connect to
the host and port specified. See https://access.redhat.com/solutions/327903
for more information. ',)
Jun  8 12:54:22 localhost osa-dispatcher: Spacewalk 13013 2016/06/08
12:54:22 -06:00: ('Error caught:',)
Jun  8 12:54:22 localhost osa-dispatcher: ERROR: unhandled exception
occurred: (unicode argument expected, got 'str').
Jun  8 12:54:22 localhost systemd: osa-dispatcher.service: control process
exited, code=exited status=255
Jun  8 12:54:22 localhost systemd: Unit osa-dispatcher.service entered
failed state.
Jun  8 12:54:22 localhost systemd: osa-dispatcher.service failed.
Jun  8 12:58:50 iiamabqzpcm02p osa-dispatcher: Spacewalk 14786 2016/06/08
12:58:50 -06:00: ('Error connecting to jabber server: Unable to connect to
the host and port specified. See https://access.redhat.com/solutions/327903
for more information. ',)
Jun  8 12:58:50 iiamabqzpcm02p osa-dispatcher: Spacewalk 14786 2016/06/08
12:58:50 -06:00: ('Error caught:',)
Jun  8 12:58:50 iiamabqzpcm02p osa-dispatcher: ERROR: unhandled exception
occurred: (unicode argument expected, got 'str').
Jun  8 12:58:50 iiamabqzpcm02p systemd: osa-dispatcher.service: control
process exited, code=exited status=255
Jun  8 12:58:50 iiamabqzpcm02p systemd: Unit osa-dispatcher.service entered
failed state.
Jun  8 12:58:50 iiamabqzpcm02p systemd: osa-dispatcher.service failed.


*Matthew S. McCleary*
*Lead Linux Systems Administrator*
Contractor - Cherokee Nation Technologies

matthew.mccleary at bia.gov
<https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=matthew.mccleary@bia.gov>
cherokee-cnt.com
1011 Indian School Rd. NW
Albuquerque, NM 87104
505-563-5049 w
505-239-1044 m
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20160608/35758407/attachment.htm>

More information about the Spacewalk-list mailing list