[Freeipa-users] FreeIPA JSON API does not work behind Load Balancer because Services4User
Klíma David
david.klima at vig.cz
Thu Oct 20 13:37:58 UTC 2016
Hi all, I need advice or help with freeIPA implementation behind F5 bigip loadbalancer. My goal is to have all freeIPA services (including json/xml API) behind loadbalancer for freeIPA clients.
>> Because RHEL support says me IPA behind loadbalancer is not supported I was coming out of these articles (I recommend you read and I thank the people who wrote them):
https://www.redhat.com/archives/freeipa-users/2015-March/msg00965.html
http://directory.fedoraproject.org/docs/389ds/howto/howto-loadbalance-gssapi.html
https://ssimo.org/blog/id_019.html
https://access.redhat.com/solutions/547723
http://firstyear.id.au/blog/html/2015/12/11/Load_balanced_389_instance_with_freeipa_kerberos_domain..html
http://www.freeipa.org/page/V4/Keytab_Retrieval#Use_Case:_A_load_balancing_cluster_of_HTTP_server_that_allow_GSSAPI.2FKrb5_negotiation_.28TBD.29
https://www.freeipa.org/page/V4/Service_Constraint_Delegation
http://vda.li/en/posts/2013/07/29/Setting-up-S4U2Proxy-with-FreeIPA/index.html
https://vda.li/en/posts/2015/05/28/talking-to-freeipa-api-with-sessions/
https://www.adelton.com/freeipa/freeipa-behind-proxy-with-different-name
https://www.adelton.com/freeipa/freeipa-behind-ssl-proxy
>> Now I have one pool with one freeIPA node (for easy debugging):
hostname: ipa-01.internal.services
>> And VIP hostname for clients:
hostname: hub.internal.services
hub.internal.services
+--------------+
| |
| |
+--------+ | Loadbalancer | ipa-01.internal.services
| | TLS | | TLS +--------------+
|Client +----->+ +----->+ |
| | | | | freeIPA node |
+--------+ | | | |
| | +--------------+
+--------------+
>> After ipa-server-install .... first, I created a fake host that I assign services. This is fake host for the load balancer:
ipa host-add hub.internal.services --force --random
ipa host-allow-retrieve-keytab hub.internal.services --users=admin
ipa-getkeytab -s ipa-01.internal.services -p host/hub.internal.services -k /etc/krb5.keytab \
-e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96,des3-cbc-sha1,arcfour-hmac,camellia128-cts-cmac,camellia256-cts-cmac
>> Second I created LDAP service - because I need keytab for ldap/hub.internal.services (after retrieved merged into /etc/dirsrv/ds.keytab):
ipa service-add --force ldap/hub.internal.services
ipa service-add-host ldap/hub.internal.services --hosts=ipa-01.internal.services
ipa service-allow-retrieve-keytab ldap/hub.internal.services --users=admin
ipa-getkeytab -s ipa-01.internal.services -p ldap/hub.internal.services -k /etc/dirsrv/ds.keytab \
-e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96,des3-cbc-sha1,arcfour-hmac,camellia128-cts-cmac,camellia256-cts-cmac
chown dirsrv:dirsrv /etc/dirsrv/ds.keytab
>> Next I created HTTP service - I need keytab for HTTP/hub.internal.services (after retrieved merged into /etc/httpd/conf/ipa.keytab):
ipa service-add --force HTTP/hub.internal.services
ipa service-add-host HTTP/hub.internal.services --hosts={ipa-01.internal.services,ipa-02.internal.services,ipa-03.internal.services}
ipa service-allow-retrieve-keytab HTTP/hub.internal.services --users=admin
ipa-getkeytab -s ipa-01.internal.services -p HTTP/hub.internal.services -k /etc/httpd/conf/ipa.keytab \
-e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96,des3-cbc-sha1,arcfour-hmac,camellia128-cts-cmac,camellia256-cts-cmac
chown apache:apache /etc/httpd/conf/ipa.keytab
>> Check keytabs:
klist -Kket /etc/krb5.keytab
klist -Kket /etc/dirsrv/ds.keytab
klist -Kket /etc/httpd/conf/ipa.keytab
All keytabs looks like this:
Keytab name: FILE:/etc/dirsrv/ds.keytab
KVNO Timestamp Principal
---- ------------------ -------------------------------------------------------
3 13.5.2016 22:05:14 ldap/ipa-01.internal.services at INTERNAL.SERVICES (aes256-cts-hmac-sha1-96) (0x0b8140ce7a7a521cbacecda8902e7c7a6b61fd21758997fb2f2721d9f2d3c8e5)
3 13.5.2016 22:05:14 ldap/ipa-01.internal.services at INTERNAL.SERVICES (aes128-cts-hmac-sha1-96) (0x4247b97e7b2b62a49094105b86740537)
3 13.5.2016 22:05:14 ldap/ipa-01.internal.services at INTERNAL.SERVICES (des3-cbc-sha1) (0x67851f1a16f8df45b30b1a89fe677ad03eaeae6ba2940e4a)
3 13.5.2016 22:05:14 ldap/ipa-01.internal.services at INTERNAL.SERVICES (arcfour-hmac) (0xed6d8caba385fdd8b5775e2f17303fb6)
1 13.5.2016 23:00:43 ldap/hub.internal.services at INTERNAL.SERVICES (aes256-cts-hmac-sha1-96) (0x439341b1848dc91f02f6b38f2e04446e9f7f8547d8251a708dce99d1526e961a)
1 13.5.2016 23:00:43 ldap/hub.internal.services at INTERNAL.SERVICES (aes128-cts-hmac-sha1-96) (0x11e1c820db6b49bb9290c0c9e2888914)
1 13.5.2016 23:00:43 ldap/hub.internal.services at INTERNAL.SERVICES (des3-cbc-sha1) (0xbad3cb89fbf132abbcad29bcfd79fb4532cedfe90bf1078f)
1 13.5.2016 23:00:43 ldap/hub.internal.services at INTERNAL.SERVICES (arcfour-hmac) (0xb80563d1f60ac374ffb3888c95434371)
>> Next I add 'ignore_acceptor_hostname = true' to the /etc/krb5.conf file (because I need ignore acceptor hostname):
sed -i '/^\[libdefaults\]$/a\ ignore_acceptor_hostname = true' /etc/krb5.conf
>> Last step was modify rewrite rules in /etc/httpd/conf.d/ipa-rewrite.conf file, I commented all lines except these:
RewriteEngine on
RewriteRule ^/ipa/ui/js/freeipa/plugins.js$ /ipa/wsgi/plugins.py [PT]
>> On Loadbalancer I created iRule for replacing referer when client send request to hub.internal.services and for replace cookie domain when response from IPA node:
when HTTP_REQUEST_SEND {
clientside {
# Odstani z IP route domain
scan [LB::server addr] {%[^%]} iponly
# Najde v listu s nazvem ipa-hostnames podle IP hostname a nahradi referer
HTTP::header replace Referer "https://[class match -value $iponly equals ipa-hostnames]/ipa/ui/"
# Vypise referer do logu
#log local0. "[HTTP::header Referer]"
}
}
when HTTP_RESPONSE {
set newdomain "hub.internal.services"
foreach mycookie [HTTP::cookie names] {
HTTP::cookie domain $mycookie $newdomain
}
}
>> I make SSL offloading on loadbalancer for LDAPS (636), LDAP over SSL (389 starttls extension), HTTPS, so SSL certificate CN matching everytime. Certs on LB are from the same authority as certificates for IPA nodes.
>> Now I am in state all services working fine (LDAP, HTTP web gui, NTP, DNS) with kerberos auth bud freeIPA json or xml api NOT.
david at dklima:~$ ldapsearch -H ldap://hub.internal.services -Y GSSAPI
SASL/GSSAPI authentication started
SASL username: admin at INTERNAL.SERVICES
SASL SSF: 56
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <> (default) with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# search result
search: 4
result: 32 No such object
# numResponses: 1
>> I know why kerberos auth to ONLY freeipa json api fails. It is because freeipa using S4U2Proxy/Services4User and client (ipa-client-install) not send TGT. So freeipa backend can not connect to 389DS with user identity >> If I calling API throught loadbalancer:
>> My freeipa api testing command:
rm $COOKIEJAR -f
export KRB5CCNAME=FILE:/tmp/krb5cc_1000
export COOKIE=/tmp/cookie.ipa
export IPAHOSTNAME=hub.internal.services
curl -vc $COOKIE -b $COOKIE -k --negotiate -u : -X GET https://$IPAHOSTNAME/ipa/xml
Result:
<?xml version='1.0' encoding='UTF-8'?>
<methodResponse>
<fault>
<value><struct>
<member>
<name>faultCode</name>
<value><int>2100</int></value>
</member>
<member>
<name>faultString</name>
<value><string>Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Matching credential not found)</string></value>
</member>
</struct></value>
</fault>
</methodResponse>
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
>> If i try add parameter '--delegation always' to curl command result is OK, API is working
curl --delegation always -vc $COOKIE -b $COOKIE -k --negotiate -u : -X GET https://$IPAHOSTNAME/ipa/xml
<?xml version='1.0' encoding='UTF-8'?>
<methodResponse>
<fault>
<value><struct>
<member>
<name>faultCode</name>
<value><int>905</int></value>
</member>
<member>
<name>faultString</name>
<value><string>unknown command 'xml'</string></value>
</member>
</struct></value>
</fault>
</methodResponse>
>> So I add the service constraint delegation:
[root at ipa-01 ~]# ipa servicedelegationrule-show ipa-http-delegation
Delegation name: ipa-http-delegation
Allowed Target: ipa-ldap-delegation-targets, ipa-cifs-delegation-targets
Member principals: HTTP/ipa-01.internal.services at INTERNAL.SERVICES, HTTP/hub.internal.services at INTERNAL.SERVICES
[root at ipa-01 ~]# ipa servicedelegationtarget-show ipa-ldap-delegation-targets
Delegation name: ipa-ldap-delegation-targets
Member principals: ldap/hub.internal.services at INTERNAL.SERVICES, HTTP/ipa-01.internal.services at INTERNAL.SERVICES, ldap/ipa-01.internal.services at INTERNAL.SERVICES
>> Now as you can see I am able to get ticket for ldap/ipa-01.internal.services based on ticket HTTP/hub.internal.services:
[root at ipa-01 ~]# kinit -kt /etc/httpd/conf/ipa.keytab HTTP/hub.internal.services
[root at ipa-01 ~]# kvno -k /etc/httpd/conf/ipa.keytab -U admin -P HTTP/hub.internal.services ldap/ipa-01.internal.services
HTTP/hub.internal.services at INTERNAL.SERVICES: kvno = 1, keytab entry valid
ldap/ipa-01.internal.services at INTERNAL.SERVICES: kvno = 1, keytab entry valid
>> I monitored KRB clientcache on IPA node and difference if connection failed:
>> This is direct connection to API - cache is good, as you can see - based on HTTP/ipa-01.internal.services IPA framework got ldap/ipa-01.internal.services
[root at ipa-01 caches]# klist admin at INTERNAL.SERVICES-directipa
Ticket cache: FILE:admin at INTERNAL.SERVICES-directipa
Default principal: admin at INTERNAL.SERVICES
Valid starting Expires Service principal
27.9.2016 21:51:01 28.9.2016 21:50:47 HTTP/ipa-01.internal.services at INTERNAL.SERVICES
27.9.2016 19:21:16 28.9.2016 19:21:16 krbtgt/INTERNAL.SERVICES at INTERNAL.SERVICES
for client HTTP/ipa-01.internal.services at INTERNAL.SERVICES
27.9.2016 21:51:02 28.9.2016 19:21:16 ldap/ipa-01.internal.services at INTERNAL.SERVICES
>> This is connection to API trought loadbalancer (hub.internal.services), connection ended with error, because IPA framework do not knowing that they must use HTTP/hub.internal.services for got ldap/ipa-01.internal.services ticket or client TGT (not send in this case)
[root at ipa-01 caches]# klist admin at INTERNAL.SERVICES-throught-loadbalancer
Ticket cache: FILE:admin at INTERNAL.SERVICES-throught-loadbalancer
Default principal: admin at INTERNAL.SERVICES
Valid starting Expires Service principal
27.9.2016 21:54:00 28.9.2016 21:50:47 HTTP/hub.internal.services at INTERNAL.SERVICES
27.9.2016 19:21:16 28.9.2016 19:21:16 krbtgt/INTERNAL.SERVICES at INTERNAL.SERVICES
for client HTTP/ipa-01.internal.services at INTERNAL.SERVICES
>> This is connection to API trought loadbalancer (hub.internal.services) with TGT delegation, connection ended with success, because IPA framework use TGT to obtain ticket for ldap/ipa-01.internal.services
[root at ipa-01 caches]# klist admin at INTERNAL.SERVICES-loadbalancer-delegace
Ticket cache: FILE:admin at INTERNAL.SERVICES-loadbalancer-delegace
Default principal: admin at INTERNAL.SERVICES
Valid starting Expires Service principal
27.9.2016 21:54:01 28.9.2016 21:50:47 krbtgt/INTERNAL.SERVICES at INTERNAL.SERVICES
27.9.2016 21:54:01 28.9.2016 21:50:47 ldap/ipa-01.internal.services at INTERNAL.SERVICES
>> So without delegation TGT there is missing ticket for ldap/ipa-01.internal.services and IPA framework return this message:
Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Matching credential not found)
>> In freeipa code is hardcored ipa node principal and i do not know where is this part of code. Can you help me create patch (or introduce a code point where I'm looking for) or found other solutions, please? Or I think one solution is to force the client (in my case ipa-client-instal) to send TGT. Bud I do not want to send TGT. I think better solutions is create patch for freeipa server code. Delegation is already produced by ipa servicedelegationrule.
Thank you very much and sorry for my English.
David
More information about the Freeipa-users
mailing list