[Spacewalk-list] Spacewalk stopped working

[Rob Sterenborg]

We're running Spacewalk 2.7 on CentOS 7.

Yesterday all yum commands requiring the Spacewalk server started failing on all of our Spacewalk clients.


The Apache ssl_error_log logfile has a lot of:
=====================================================
[Tue Jun 25 14:51:32.981919 2019] [proxy_ajp:error] [pid 48172] [client 192.168.x.y:37042] AH00877: read zero bytes, expecting 122 bytes
=====================================================


The Tomcat localhost_access_log.<date>.txt logfile has a lot of:
=====================================================
192.168.x.y - - [25/Jun/2019:14:51:33 +0200] "GET /rhn/errors/404.jsp HTTP/1.1" 404 -
=====================================================


=====================================================
# rhn-profile-sync -vvv
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Providename 0x400 mode=0x0
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login A protocol error occurred: Not Found , attempt #1, Could not find URL, spacewalk.domain.local /XMLRPC Check server name and/or URL, then retry

There was a RhnServerException while testing login:

D: closed   db index       /var/lib/rpm/Providename
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
=====================================================
/var/log/up2date:
[Wed Jun 26 10:54:02 2019] up2date logging into up2date server
[Wed Jun 26 10:54:02 2019] up2date A protocol error occurred: Not Found , attempt #1,
[Wed Jun 26 10:54:02 2019] up2date Could not find URL, spacewalk.domain.local /XMLRPC
[Wed Jun 26 10:54:02 2019] up2date Check server name and/or URL, then retry
=====================================================

I'm unsure why the message above says "Could not find URL, spacewalk.domain.local /XMLRPC" (the space between the hostname and path), because in /etc/sysconfig/rhn/up2date I have:
=====================================================
serverURL=https://spacewalk.domain.local/XMLRPC
=====================================================
Is that cosmetics?


I can resolve the actual "spacewalk.domain.local" just fine, tcp ports 80 and 443 are accessible from the client host experiencing issues.
I can use the web GUI without issues, and also the local spacecmd works just fine.


In the Apache config I have:
=====================================================
<IfModule !wsgi_module>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

WSGIPythonPath "/usr/share/rhn"

WSGIScriptAlias /APP /usr/share/rhn/wsgi/app.py
WSGIScriptAlias /APPLET /usr/share/rhn/wsgi/applet.py
WSGIScriptAlias /CONFIG-MANAGEMENT /usr/share/rhn/wsgi/config.py
WSGIScriptAlias /CONFIG-MANAGEMENT-TOOL /usr/share/rhn/wsgi/config_tool.py
WSGIScriptAlias /PACKAGE-PUSH /usr/share/rhn/wsgi/package_push.py
WSGIScriptAlias /SAT /usr/share/rhn/wsgi/sat.py
WSGIScriptAlias /SAT-DUMP-INTERNAL /usr/share/rhn/wsgi/sat_dump.py
WSGIScriptAlias /XMLRPC /usr/share/rhn/wsgi/xmlrpc.py

<IfVersion >= 2.4>
    <Directory  /usr/share/rhn/wsgi>
        Require all granted
    </Directory>
</IfVersion>
=====================================================

This python scripts exist, file owner/mode: root:root 644.
AFAICS it seems like this part is not working. But I'm simply not sure what is wrong.


Can someone please point me to exactly what it is that is not working anymore and how to fix it.
I can provide any additional config needed; right now I'm unsure which that would be.


Thanks,
Rob