[Freeipa-users] ipa-server-upgrade fails and CA cannot start

[Petr Vobornik]

On 05/08/2016 09:49 PM, Andrew C. Dingman wrote:
> For those of you who recognize me from non-public lists and chats, this
> is a whole different setup from the one we've been discussing there.
> 
> This is on a RHEL 7 system, and unfortunately for me the CA master in
> my personal IPA realm. When I attempted to update using yum on April
> 15th, the ipa-server-update script failed with what seems to be a dbus
> error, and I have been unable to start the CA (and therefore ipa in
> general) since. As a result, my personal systems are running on one IPA
> server, which makes me more than a little nervous.
> 
> The relevant bit of the upgrade log seems to be:
> 
> 2016-05-08T19:03:08Z DEBUG stderr=
> 2016-05-08T19:03:08Z INFO [Upgrading CA schema]
> 2016-05-08T19:03:08Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-
> ACDINGMAN-COM.socket from SchemaCache
> 2016-05-08T19:03:08Z DEBUG retrieving schema for SchemaCache
> url=ldapi://%2fvar%2frun%2fslapd-ACDINGMAN-COM.socket
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x576e368>
> 2016-05-08T19:03:08Z DEBUG Processing schema LDIF file
> /usr/share/pki/server/conf/schema-certProfile.ldif
> 2016-05-08T19:03:08Z DEBUG Not updating schema
> 2016-05-08T19:03:08Z INFO CA schema update complete (no changes)
> 2016-05-08T19:03:08Z INFO [Verifying that CA audit signing cert has 2
> year validity]
> 2016-05-08T19:03:08Z DEBUG caSignedLogCert.cfg profile validity range
> is 720
> 2016-05-08T19:03:08Z INFO [Update certmonger certificate renewal
> configuration to version 4]
> 2016-05-08T19:03:08Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2016-05-08T19:03:08Z ERROR Failed to get request: bus, object_path and
> dbus_interface must not be None.
> 2016-05-08T19:03:08Z ERROR IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
> 2016-05-08T19:03:08Z DEBUG   File "/usr/lib/python2.7/site-
> packages/ipapython/admintool.py", line 171, in execute
>     return_value = self.run()
>   File "/usr/lib/python2.7/site-
> packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
>     raise admintool.ScriptError(str(e))
> 
> 2016-05-08T19:03:08Z DEBUG The ipa-server-upgrade command failed,
> exception: ScriptError: bus, object_path and dbus_interface must not be
> None.
> 2016-05-08T19:03:08Z ERROR bus, object_path and dbus_interface must not
> be None.
> 
> There's a whole lot more, nearly 4MiB of log even when I reduce it to
> my most recent attempt to run the upgrade script.
> 
> "getcert list" successfully shows 8 certificate requests being tracked.
> Four are in "MONITORING" status, four in "NEED_CA". The NEED_CA
> requests all indicate expiration back in February, and look like
> crucial certificates: CN=CA Subsystem, CN=IPA RA, CN=CA Audit
> and CN=OCSP Subsystem.
> 
> On the working replica, all eight are in "MONITORING" status and have
> expiration dates in 2017 or later. I have not attempted the package
> update on that system. Should I consider promoting this one to CA
> master, force-deleting the old one, and reinstalling it as a new
> system?
> 
> Please let me know what other information would be helpful for
> diagnostics. The current state of all packages on the broken master is
> up to earlier today from the official Red Hat content distribution
> network.
> 

Hello Andrew,

Could you paste output of `ipactl start` ?

Also when upgrader fails it tends to leave directory server not
accessible by changing 389 and 636 port.

It could be verified by:

$ ldapsearch -ZZ -h `hostname` -D "cn=Directory Manager" -W -s base -b
"cn=config" | grep "nsslapd-security\|nsslapd-port"
Enter LDAP Password:
nsslapd-requiresrestart: cn=config:nsslapd-port
nsslapd-port: 389
nsslapd-security: on

If there are values other than '389' and 'on' (usually '0' and 'off')
then it might the reason why IPA doesn't start. Changing them back to
'on' and 389 might help.

But it won't say why the upgrader failed. Maybe it was a one-time glitch
or it was related to the expired certs.

The error message you got is in code which creates connection to
certmonger.

But if there are expired certificates. The usual recovery is to move
back time a day or two before the first certificate expires and let
certmonger to renew the certs. Optionally the renewal can be forced by
`getcert resubmit -i $certid` command.
-- 
Petr Vobornik