[Freeipa-devel] Check if server is fully installed and ready

[Petr Spacek]

On 14.4.2016 12:23, Martin Babinsky wrote:
> On 04/14/2016 11:16 AM, Christian Heimes wrote:
>> Hi,
>>
>> while I was working on my Ansible playbook I ran into an issue. It is
>> hard to detect if a FreeIPA server instance is fully installed and all
>> its services are ready to handle requests. It's even harder to check it
>> remotely. I have figured out some heuristics to detect that a sever is
>> *not* fully installed (e.g. /etc/ipa/default.conf is missing or
>> http://ipa-ca.ipa.example/ipa/crl/MasterCRL.bin returns 404). The
>> presence of these resources is no guarantee that all FreeIPA services
>> fully up and running.
>>
>> Two days ago on IRC Jan came up with the same problem with containers.
>> He ran into a problem related to containers and DNS updates. Since I'm
>> no longer alone with the problem and my own workarounds are not
>> completely stable, I like to address the problem in FreeIPA directly.
>> Now you might wonder why it is so hard to check if FreeIPA is ready or
>> why nobody ran into the issue before.
>>
>> Let's start with the second question. A typical admin first installs a
>> FreeIPA server on one machine. It takes a couple of seconds until he
>> notices that the installer has finished. The admin ssh-es into another
>> machine, sudo -s and then runs ipa-client-install with some arguments.
>> It takes a couple of seconds, maybe even a minute. With containers and
>> automation tools it's more like milliseconds.
>>
>> Now for the first question. Under some conditions a FreeIPA service
>> might be started but not yet ready to serve requests or aren't fully
>> operation yet. For example ticket
>> https://fedorahosted.org/freeipa/ticket/5813 is an example of a problem
>> with ipa-kra-install, 389-DS restarts and bind-dyndb-ldap.
>>
>>
>> Proposal
>>
>> 1) A new boolean attribute ipaReady=TRUE/FALSE in
>> cn=$FQDN,cn=masters,cn=ipa,cn=etc,$SUFFIX tracks whether or not an
>> FreeIPA instance is ready to handle requests.
>>
> 
> Each service entry in the service container contains
> 'ipaConfigString=enabledService' attribute that *should* tell that the service
> is enabled and running.
> 
> When the Server Roles[1] feature you could check whether the role that
> interests you (in this case implicit "IPA Master" role) is enabled (which
> means that all component services are marked as enabled, read the linked
> design for more info).
> 
> However the fact that something is marked as enabled/ready in LDAP backend
> does not reflect the real status of systemd service on local system it just
> tells you that someone (typically installer) marked this service as enabled.
> 
> For that we would need a daemon that would synchronize the state of local
> services with the state of LDAP backend and even then things may not be 100%
> guaranteed.

We can use systemd's feature 'notify' to inform systemd that service is
actually running and accepting requests (compare this with 'process is running').

See man sd_notify

I have a bind-dyndb-ldap ticket for this but it did not get sufficient
priority up until now:
https://fedorahosted.org/bind-dyndb-ldap/ticket/150

The very same thing can be done with all other services. When this is
implemented for all services, it will be very easy to add systemd unit like
'ipa-ready' which can be used either for local check or to trigger a file
generation somewhere so you can read the file over HTTP or so.

Petr^2 Spacek

> 
>> 2) A new HTTP route http[s]://$FQDN/ipa/ready is added. The route does
>> not need authentication. When ipaReady=TRUE the route simple returns 200
>> OK with some text like READY. When the attribute is not present or
>> FALSE, it returns an error to the client (412, 408?).
>>
> 
>> 3) All ipa install and upgrade commands set the attribute to FALSE
>> before any tasks.
>>
> Installers already mark the corresponding service as enabled when finished
> succesfully.
>> 4) A final step in all ipa install and upgrade commands checks that all
>> services have been started and are ready to handle requests. Eventually
>> the ipaReady attribute is set to true.
>>
> This is basically what ipactl status does.
>> Christian
>>
>>
>>
> http://www.freeipa.org/page/V4/Server_Roles
> 


-- 
Petr^2 Spacek