[Freeipa-users] IPA Startup issues

Mon May 16 14:56:34 UTC 2011

On 05/16/2011 08:43 AM, Sigbjorn Lie wrote:
> On 05/16/2011 03:52 PM, Simo Sorce wrote:
>> On Sat, 2011-05-14 at 16:46 +0200, Sigbjorn Lie wrote:
>>> I've noticed that if the machine running IPA is very busy at startup,
>>> the IPA services will not be online when the machine is started.
>>>
>>> I noticed this is as my test virtualization host has had it's power 
>>> cord
>>> knocked out a few times. When I restart the host machine, all the
>>> virtual machines is started at the same time, causing (a lot) higher
>>> than normal latency for each virtual machine.
>>>
>>> This causes the IPA daemons to start, while during the startup one or
>>> several IPA daemons fails due to dependencies of other daemons which is
>>> not started yet, and all the IPA daemons is stopped as not all the IPA
>>> daemons started successfully. I've noticed that the default behavior of
>>> the ipactl command is to shut down all the IPA daemons, if any of the
>>> IPA daemons should fail during startup.
>>>
>>> This can be seen in the logs of the individual services, as some is
>>> started successfully, just to receive a shutdown signal shortly after.
>>> It seem to be the pki-ca which shut down my IPA services this morning.
>>>
>>> When rebooting the virtual machine running the IPA daemons during 
>>> normal
>>> load of the host machine, all the IPA daemons start successfully.
>>> Logging on to the IPA server and manually starting the IPA daemons 
>>> after
>>> the load of the host machine has decreased also works.
>>>
>>> I suggest changing the startup scripts to allow (a lot) longer startup
>>> times for the IPA daemons prior to failing them.
>> At the moment we just run service<name>  start and wait until it is
>> done. If the pki-cad service timeouts and returns an error I think we
>> need to open a bug against the dogtag component as that is the cause.
>>
>> Can you open a bug in the freeipa trac with logs showing that service is
>> responsible for the failure ?
>
> I haven't been able to figure out which service that failed IPA yet. A 
> lot of log files scattered around. As you can see from the slapd 
> errors file, the slapd daemon was available for almost 3 minutes 
> before receiving the shutdown signal. I notice now that the PKI daemon 
> failed 8 seconds after slapd had shut down, so I was wrong in blaming 
> the PKI daemon.
>
> See below for a list of log files I've been trough. They all have on 
> thing in common, the daemons starts when the host machine is started, 
> at approx 06:34, then receives a shutdown signal around 06:37. Some 
> time later when the host has calmed down, I'm logging in and manually 
> starting IPA using "ipactl start", and all the daemons start without 
> any problem. And they keep running after my manual intervention.
>
> I wish I could be more specific, but I'm unsure where else to look. 
> Suggestions?
>
>
> /var/log/krb5kdc.log
> /var/log/pki-ca/catalina.out
> /var/log/dirsrv/slapd-IX-TEST-COM/errors
> /var/log/dirsrv/slapd-PKI-IPA/errors
> /var/log/httpd/error_log
> /var/log/messages (named log)
>
> slapd errors:
>
> [14/May/2011:06:33:52 +0200] - 389-Directory/1.2.8.rc1 B2011.062.1416 
> starting up
> [14/May/2011:06:33:54 +0200] - Detected Disorderly Shutdown last time 
> Directory Server was running, recovering database.
1) Disorderly Shutdown means a) crash b) kill -9 or similar - neither of 
which should be happening - is this the replica install or the first 
master install?
> [14/May/2011:06:34:39 +0200] schema-compat-plugin - warning: no 
> entries set up under , ou=SUDOers, dc=ix,dc=TEST,dc=com
> [14/May/2011:06:34:39 +0200] - Skipping CoS Definition cn=Password 
> Policy,cn=accounts,dc=ix,dc=TEST,dc=com--no CoS Templates found, which 
> should be added b
> efore the CoS Definition.
> [14/May/2011:06:34:40 +0200] - Skipping CoS Definition cn=Password 
> Policy,cn=accounts,dc=ix,dc=TEST,dc=com--no CoS Templates found, which 
> should be added b
> efore the CoS Definition.
> [14/May/2011:06:34:41 +0200] - slapd started.  Listening on All 
> Interfaces port 389 for LDAP requests
> [14/May/2011:06:34:41 +0200] - Listening on All Interfaces port 636 
> for LDAPS requests
> [14/May/2011:06:34:42 +0200] - Listening on 
> /var/run/slapd-IX-TEST-COM.socket for LDAPI requests
> [14/May/2011:06:37:30 +0200] - slapd shutting down - signaling 
> operation threads
> [14/May/2011:06:37:30 +0200] - slapd shutting down - closing down 
> internal subsystems and plugins
> [14/May/2011:06:37:31 +0200] - Waiting for 4 database threads to stop
> [14/May/2011:06:37:32 +0200] - All database threads now stopped
> [14/May/2011:06:37:32 +0200] - slapd stopped.
>
>
> /var/log/pki-ca/system:
> 1871.main - [14/May/2011:06:37:40 CEST] [8] [3] In Ldap (bound) 
> connection pool to host ipasrv01.ix.TEST.com port 7389, Cannot connect 
> to LDAP server. Error: netscape.ldap.LDAPException: failed to connect 
> to server ldap://ipasrv01.ix.TEST.com:7389 (91)
>
> _______________________________________________
> Freeipa-users mailing list
> Freeipa-users at redhat.com
> https://www.redhat.com/mailman/listinfo/freeipa-users