From mkosek at redhat.com Thu Aug 1 08:36:18 2013 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 01 Aug 2013 10:36:18 +0200 Subject: [Freeipa-users] Providing minimal permissions to read replication status In-Reply-To: References: Message-ID: <51FA1E02.5030809@redhat.com> On 07/31/2013 01:36 PM, James Hogarth wrote: > Hi, > > We're looking to add monitoring to our IPA replicas and want to provide a > user with the minimum possible permissions to do so. > > Allowing the user to have the Replication Administrators role works but for > monitoring the ability to add/modify/remove is overkill by a long shot. Agreed. > > There's no existing permission for Read Replication Agreements - only add, > remove and modify. > > I've tried to use ipa perimssion-add with --filter to allow access to > objectClass=nsds5replicationagreement but checking the status via: > > ldapsearch -Y GSSAPI -h c6test2.c6ipa.local -b cn=config > '(objectclass=nsds5replicationagreement)' > > Does not show anything unless the account being tested with gets > replication administrator privileges... > > I've tried using subtree as well but the ipa command errors that the base > of cn=config is not $SUFFIX ... and out of scope. Yes, this won't work with FreeIPA permission-* commands by design as they operate purely on the FreeIPA suffix. > > What am I missing to set this up - or is this not possible with the > role/privilege/permission mechanism within IPA? I can see how the > replication administration permissions are added in replica-acis.ldif but > I'm concerned that if I manually add an ACI via pure LDIF commands it will > cause issues with future IPA upgrades due to schema differences - so was > hoping to remain within the IPA command side of things... > > 1) Is this even possible with the ipa command? No - as above. > 2) If I use ldapmodify to add a new permission by hand via ldif for "Read > Replication Agreements" will this likely break on IPA upgrades in future? There may be upgrade issue e.g. if you name the new permission "Read Replication Agreements" as this will be the name that FreeIPA project will use in a future to name this permission - as it makes sense that there is a permission like that. I see 2 solutions for your issue: 1) Handle the ACIs and permission yourselves, I tested that an ACI like that works: # ldapmodify -h localhost -D "cn=Directory Manager" -x -w kokos123 dn: cn="$SUFFIX",cn=mapping tree,cn=config changetype: modify add: aci aci: (targetattr=*)(targetfilter="(|(objectclass=nsds5Replica)(objectclass=nsds5replicationagreement)(objectclass=nsDSWindowsReplicationAgreement)(objectClass=nsMappingTree))")(version 3.0; acl "permission:Read Replication Agreements"; allow (read, search, compare) groupdn = "ldap:///$PERMISSION_OR_GROUP_DN";) It should be pretty safe to add ACI like this one, especially because cn=config tree is not replicated and your changes are only local. 2) Work with freeipa-devel list to add this permission to FreeIPA out of the box - this is of course the preferred option by us :-) The patch for this would do basically this: - remove the following aci: (targetattr != aci)(version 3.0; aci "replica admins read access"; allow (read, search, compare) groupdn = "ldap:///cn=Modify Replication Agreements,cn=permissions,cn=pbac,$SUFFIX";) ... from installer and from LDAP as it is too general - add new permission ACI like this: (targetattr=*)(targetfilter="(|(objectclass=nsds5Replica)(objectclass=nsds5replicationagreement)(objectclass=nsDSWindowsReplicationAgreement)(objectClass=nsMappingTree))")(version 3.0; acl "permission:Read Replication Agreements"; allow (read, search, compare) groupdn = "ldap:///cn=Read Replication Agreements,cn=permissions,cn=pbac,$SUFFIX";) - make sure that "Replication Administrators" privilege has it assigned. I created an upstream ticket to track this effort: https://fedorahosted.org/freeipa/ticket/3829 Martin From sbose at redhat.com Thu Aug 1 09:44:20 2013 From: sbose at redhat.com (Sumit Bose) Date: Thu, 1 Aug 2013 11:44:20 +0200 Subject: [Freeipa-users] authenticate with base domain name? In-Reply-To: References: <20130731115652.GA3648@localhost.localdomain> <20130731162420.GD3648@localhost.localdomain> Message-ID: <20130801094420.GH3648@localhost.localdomain> On Wed, Jul 31, 2013 at 03:03:04PM -0500, KodaK wrote: > On Wed, Jul 31, 2013 at 1:28 PM, KodaK wrote: > > On Wed, Jul 31, 2013 at 11:24 AM, Sumit Bose wrote: > >> > >> On Wed, Jul 31, 2013 at 11:12:47AM -0500, KodaK wrote: > >> > On Wed, Jul 31, 2013 at 11:09 AM, KodaK wrote: > >> > > >> > > > >> > > > >> > > On Wed, Jul 31, 2013 at 6:56 AM, Sumit Bose wrote: > >> > > > >> > > > I think that's the issue. You have to make sure that host.domain.com has > >> > > > >> > > > a DNS entry somewhere, it does not have to be the IPA DNS but the DNS > >> > > > >> > > > setup must be correct so the IPA DNS can forward the request to the > >> > > > >> > > > right server. Then you can call 'ipa host-add host.domain.com' which > >> > > > >> > > > will create a host entry with the principal > >> > > > >> > > > host/host.domain.com at UNIX.DOMAIN.COM. Now you can call ipa-getkeytab and > >> > > > >> > > > transfer the new keytab to host.domain.com. > >> > > > >> > > Ok, I'm dumbfounded (again.) > >> > > > >> > > I've removed the old host from IPA: > >> > > > >> > > xxx at slpidml01 ~]$ ipa host-show sla400q1.unix.domain.com > >> > > > >> > > ipa: INFO: trying https://slpidml01.unix.domain.com/ipa/session/xml > >> > > > >> > > ipa: INFO: Forwarding 'host_show' to server u' > >> > > https://slpidml01.unix.domain.com/ipa/session/xml' > >> > > > >> > > ipa: ERROR: sla400q1.unix.domain.com: host not found > >> > > > >> > > And I added the new host: > >> > > > >> > > [xxx at slpidml01 ~]$ ipa host-show sla400q1.domain.com > >> > > > >> > > ipa: INFO: trying https://slpidml01.unix.domain.com/ipa/xml > >> > > > >> > > ipa: INFO: Forwarding 'host_show' to server u' > >> > > https://slpidml01.unix.domain.com/ipa/xml' > >> > > > >> > > Host name: sla400q1.domain.com > >> > > > >> > > Principal name: host/sla400q1.domain.com at UNIX.DOMAIN.COM > >> > > > >> > > Password: False > >> > > > >> > > Keytab: True > >> > > > >> > > Managed by: sla400q1.domain.com > >> > > > >> > > I generated the keytab: > >> > > > >> > > [xxx at slpidml01 ~]$ ipa-getkeytab -s slpidml01.unix.domain.com -p host/ > >> > > sla400q1.domain.com -k /tmp/sla400q1.keytabKeytab successfully retrieved > >> > > and stored in: /tmp/sla400q1.keytab > >> > > > >> > > [xxx at slpidml01 ~]$ > >> > > > >> > > Then I copied that keytab to the host and put it in /etc/krb5/krb5.keytab > >> > > > >> > > But, when I list the principals in the keytab: > >> > > > >> > > sla400q1:/var/adm> /usr/krb5/bin/klist -k -e > >> > > > >> > > Keytab name: FILE:/etc/krb5/krb5.keytab > >> > > > >> > > KVNO Principal > >> > > > >> > > ---- --------- > >> > > > >> > > 1 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 1 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 1 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode > >> > > with HMAC/sha1) > >> > > > >> > > 1 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 2 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 2 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 2 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode > >> > > with HMAC/sha1) > >> > > > >> > > 2 host/sla400q1.unix.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 1 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 1 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 1 host/sla400q1.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode with > >> > > HMAC/sha1) > >> > > > >> > > 1 host/sla400q1.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 2 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 2 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 2 host/sla400q1.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode with > >> > > HMAC/sha1) > >> > > > >> > > 2 host/sla400q1.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 3 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 3 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 3 host/sla400q1.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode with > >> > > HMAC/sha1) > >> > > > >> > > 3 host/sla400q1.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 4 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 4 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 4 host/sla400q1.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode with > >> > > HMAC/sha1) > >> > > > >> > > 4 host/sla400q1.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 5 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 5 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 5 host/sla400q1.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode with > >> > > HMAC/sha1) > >> > > > >> > > 5 host/sla400q1.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > 6 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-256 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 6 host/sla400q1.domain.com at UNIX.DOMAIN.COM (AES-128 CTS mode with > >> > > 96-bit SHA-1 HMAC) > >> > > > >> > > 6 host/sla400q1.domain.com at UNIX.DOMAIN.COM (Triple DES cbc mode with > >> > > HMAC/sha1) > >> > > > >> > > 6 host/sla400q1.domain.com at UNIX.DOMAIN.COM (ArcFour with HMAC/md5) > >> > > > >> > > Where are the sla400q1.unix.domain.com coming from? I've done this over > >> > > and over, I can't find > >> > > > >> > > any reference to sla400q1.unix.domain.com in DNS in IPA, and the box > >> > > never had any > >> > > > >> > > unix.comain.com references. > >> > > > >> > > In addition, I?m still getting the error: > >> > > > >> > > Miscellaneous failure\nNo principal in keytab matches desired name\n > >> > > > >> > > in the logs, even though: > >> > > > >> > > sla400q1:/var/adm> grep sla400q1 /etc/hosts > >> > > > >> > > 192.168.42.108 sla400q1-bk > >> > > > >> > > #10.200.5.48 sla400q1.domain.com sla400q1 > >> > > > >> > > 10.200.5.48 sla400q1.domain.com sla400q1 > >> > > > >> > > sla400q1:/var/adm> hostname > >> > > > >> > > sla400q1.domain.com > >> > > > >> > > sla400q1:/var/adm> domainname > >> > > > >> > > domain.com > >> > > > >> > > sla400q1:/var/adm> > >> > > > >> > > Any clues? > >> > > > >> > > > >> > forgot to add: > >> > > >> > sla400q1:/var/adm> nslookup 10.200.5.48 > >> > Server: 10.200.2.24 > >> > Address: 10.200.2.24#53 > >> > > >> > 48.5.200.10.in-addr.arpa name = SLA400Q1.domain.com. > >> > >> hmm, DNS is case-insensitive, Kerberos is case-sensitive. If AIX Kerberos > >> does some reverse DNS lookups it might end up looking for > >> home/SLA400Q1.domain.com at UNIX.DOMAIN.COM. If you cannot change the case > >> of the DNS entry, please try to create an IPA host with the case > >> returned by DNS. > >> > >> > > > > IPA just changes SLA400Q1.domain.com to lower case when I do a host-add. > > > > I've asked the admins of "domain.com" to change the reverse entry, > > we'll see how that goes. > > > > Thanks again, > > > > --Jason > > Blew everything away regarding this host in IPA, cleared the keytab > and caches on the AIX box. > > And I have success. Finally. > > Thanks! great, thank you for the feedback. bye, Sumit > > > > -- > The government is going to read our mail anyway, might as well make it > tough for them. GPG Public key ID: B6A1A7C6 From james.hogarth at gmail.com Thu Aug 1 13:56:16 2013 From: james.hogarth at gmail.com (James Hogarth) Date: Thu, 1 Aug 2013 14:56:16 +0100 Subject: [Freeipa-users] Providing minimal permissions to read replication status In-Reply-To: <51FA1E02.5030809@redhat.com> References: <51FA1E02.5030809@redhat.com> Message-ID: On 1 August 2013 09:36, Martin Kosek wrote: > > > The patch for this would do basically this: > - remove the following aci: > (targetattr != aci)(version 3.0; aci "replica admins read access"; allow > (read, > search, compare) groupdn = "ldap:///cn=Modify Replication > Agreements,cn=permissions,cn=pbac,$SUFFIX";) > ... from installer and from LDAP as it is too general > - add new permission ACI like this: > > (targetattr=*)(targetfilter="(|(objectclass=nsds5Replica)(objectclass=nsds5replicationagreement)(objectclass=nsDSWindowsReplicationAgreement)(objectClass=nsMappingTree))")(version > 3.0; acl "permission:Read Replication Agreements"; allow (read, search, > compare) groupdn = "ldap:///cn=Read Replication > Agreements,cn=permissions,cn=pbac,$SUFFIX";) > - make sure that "Replication Administrators" privilege has it assigned. > > I created an upstream ticket to track this effort: > https://fedorahosted.org/freeipa/ticket/3829 > > Reading the upstream documentation I'm wondering if it'd be sensible to include an additional ACI in replica-acis.ldif of: dn: $SUFFIX changetype: modify add: aci aci: (targetattr=dn nsDS5ReplConflict nsUniqureID)(targetfilter="(|(objectclass=nsTombstone)(nsDS5ReplConflict=*))")((version 3.0; aci "conflict read access"; allow (read, search, compare) groupdn = "ldap:///cn=Read Replication Agreements,cn=permissions,cn=pbac,$SUFFIX";) >From the upstream documentation here: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Configuration_Command_and_File_Reference/index.html#Replication_Attributes_under_cnreplica_cnsuffixName_cnmapping_tree_cnconfig This would allow a user with Read Replication Agreements permission to be able to search for conflicts or tombstone records which would seem sane from a monitoring point of view... What do you think? Also just to confirm the only thing I need to do with ACIs like this is to update the ldif (delegation.ldif and replica-acis.ldif) with the new role/privilege/permission and acis in install/share for the new installs and add an appropriate entry (not quite ldif) in install/updates to update the default schema of those updating in future, given no new attributes - right? Cheers, James -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkosek at redhat.com Thu Aug 1 14:47:48 2013 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 01 Aug 2013 16:47:48 +0200 Subject: [Freeipa-users] Providing minimal permissions to read replication status In-Reply-To: References: <51FA1E02.5030809@redhat.com> Message-ID: <51FA7514.4010400@redhat.com> On 08/01/2013 03:56 PM, James Hogarth wrote: > On 1 August 2013 09:36, Martin Kosek wrote: >> >> >> The patch for this would do basically this: >> - remove the following aci: >> (targetattr != aci)(version 3.0; aci "replica admins read access"; allow >> (read, >> search, compare) groupdn = "ldap:///cn=Modify Replication >> Agreements,cn=permissions,cn=pbac,$SUFFIX";) >> ... from installer and from LDAP as it is too general >> - add new permission ACI like this: >> >> (targetattr=*)(targetfilter="(|(objectclass=nsds5Replica)(objectclass=nsds5replicationagreement)(objectclass=nsDSWindowsReplicationAgreement)(objectClass=nsMappingTree))")(version >> 3.0; acl "permission:Read Replication Agreements"; allow (read, search, >> compare) groupdn = "ldap:///cn=Read Replication >> Agreements,cn=permissions,cn=pbac,$SUFFIX";) >> - make sure that "Replication Administrators" privilege has it assigned. >> >> I created an upstream ticket to track this effort: >> https://fedorahosted.org/freeipa/ticket/3829 >> >> > Reading the upstream documentation I'm wondering if it'd be sensible to > include an additional ACI in replica-acis.ldif of: > dn: $SUFFIX > changetype: modify > add: aci > aci: (targetattr=dn nsDS5ReplConflict > nsUniqureID)(targetfilter="(|(objectclass=nsTombstone)(nsDS5ReplConflict=*))")((version > 3.0; aci "conflict read access"; allow (read, search, compare) groupdn = > "ldap:///cn=Read Replication Agreements,cn=permissions,cn=pbac,$SUFFIX";) > > From the upstream documentation here: > https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Configuration_Command_and_File_Reference/index.html#Replication_Attributes_under_cnreplica_cnsuffixName_cnmapping_tree_cnconfig > > This would allow a user with Read Replication Agreements permission to be > able to search for conflicts or tombstone records which would seem sane > from a monitoring point of view... > > What do you think? I think it would make sense, but IMO it should have a separate permission named "Read Replication Conflicts" - this would also need the aci to be named "permission:Read Replication Conflicts" to let IPA couple it with the actual ACI. > Also just to confirm the only thing I need to do with ACIs like this is to > update the ldif (delegation.ldif and replica-acis.ldif) with the new > role/privilege/permission and acis in install/share for the new installs > and add an appropriate entry (not quite ldif) in install/updates to update > the default schema of those updating in future, given no new attributes - > right? That's right (you also need to remove the inappropriate ACI) You also need to make sure that the appropriate privilege has these new permissions as members - I tried to capture these steps in the upstream ticket. Martin From rcritten at redhat.com Thu Aug 1 14:55:28 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 01 Aug 2013 10:55:28 -0400 Subject: [Freeipa-users] Providing minimal permissions to read replication status In-Reply-To: References: <51FA1E02.5030809@redhat.com> Message-ID: <51FA76E0.6060205@redhat.com> James Hogarth wrote: > > > > On 1 August 2013 09:36, Martin Kosek > wrote: > > > The patch for this would do basically this: > - remove the following aci: > (targetattr != aci)(version 3.0; aci "replica admins read access"; > allow (read, > search, compare) groupdn = "ldap:///cn=Modify Replication > Agreements,cn=permissions,cn=pbac,$SUFFIX";) > ... from installer and from LDAP as it is too general > - add new permission ACI like this: > (targetattr=*)(targetfilter="(|(objectclass=nsds5Replica)(objectclass=nsds5replicationagreement)(objectclass=nsDSWindowsReplicationAgreement)(objectClass=nsMappingTree))")(version > 3.0; acl "permission:Read Replication Agreements"; allow (read, search, > compare) groupdn = "ldap:///cn=Read Replication > Agreements,cn=permissions,cn=pbac,$SUFFIX";) > - make sure that "Replication Administrators" privilege has it assigned. > > I created an upstream ticket to track this effort: > https://fedorahosted.org/freeipa/ticket/3829 > > > Reading the upstream documentation I'm wondering if it'd be sensible to > include an additional ACI in replica-acis.ldif of: > dn: $SUFFIX > changetype: modify > add: aci > aci: (targetattr=dn nsDS5ReplConflict > nsUniqureID)(targetfilter="(|(objectclass=nsTombstone)(nsDS5ReplConflict=*))")((version > 3.0; aci "conflict read access"; allow (read, search, compare) groupdn = > "ldap:///cn=Read Replication Agreements,cn=permissions,cn=pbac,$SUFFIX";) > > From the upstream documentation here: > https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Configuration_Command_and_File_Reference/index.html#Replication_Attributes_under_cnreplica_cnsuffixName_cnmapping_tree_cnconfig > > This would allow a user with Read Replication Agreements permission to > be able to search for conflicts or tombstone records which would seem > sane from a monitoring point of view... > > What do you think? I think this would be a separate issue. Being able to find the conflicting issues leads directly to the question "what do I do with them?" That is ticket https://fedorahosted.org/freeipa/ticket/1025 > Also just to confirm the only thing I need to do with ACIs like this is > to update the ldif (delegation.ldif and replica-acis.ldif) with the new > role/privilege/permission and acis in install/share for the new installs > and add an appropriate entry (not quite ldif) in install/updates to > update the default schema of those updating in future, given no new > attributes - right? You'll need to create a .update file in install/updates to modify an existing installation. rob From james.hogarth at gmail.com Thu Aug 1 15:12:50 2013 From: james.hogarth at gmail.com (James Hogarth) Date: Thu, 1 Aug 2013 16:12:50 +0100 Subject: [Freeipa-users] Providing minimal permissions to read replication status In-Reply-To: <51FA76E0.6060205@redhat.com> References: <51FA1E02.5030809@redhat.com> <51FA76E0.6060205@redhat.com> Message-ID: On 1 August 2013 15:55, Rob Crittenden wrote: > James Hogarth wrote: > >> >> >> >> On 1 August 2013 09:36, Martin Kosek > > wrote: >> >> >> The patch for this would do basically this: >> - remove the following aci: >> (targetattr != aci)(version 3.0; aci "replica admins read access"; >> allow (read, >> search, compare) groupdn = "ldap:///cn=Modify Replication >> Agreements,cn=permissions,cn=**pbac,$SUFFIX";) >> ... from installer and from LDAP as it is too general >> - add new permission ACI like this: >> (targetattr=*)(targetfilter="(**|(objectclass=nsds5Replica)(** >> objectclass=**nsds5replicationagreement)(**objectclass=** >> nsDSWindowsReplicationAgreemen**t)(objectClass=nsMappingTree))** >> ")(version >> 3.0; acl "permission:Read Replication Agreements"; allow (read, >> search, >> compare) groupdn = "ldap:///cn=Read Replication >> Agreements,cn=permissions,cn=**pbac,$SUFFIX";) >> - make sure that "Replication Administrators" privilege has it >> assigned. >> >> I created an upstream ticket to track this effort: >> https://fedorahosted.org/**freeipa/ticket/3829 >> >> >> Reading the upstream documentation I'm wondering if it'd be sensible to >> include an additional ACI in replica-acis.ldif of: >> dn: $SUFFIX >> changetype: modify >> add: aci >> aci: (targetattr=dn nsDS5ReplConflict >> nsUniqureID)(targetfilter="(|(**objectclass=nsTombstone)(** >> nsDS5ReplConflict=*))")((**version >> 3.0; aci "conflict read access"; allow (read, search, compare) groupdn = >> "ldap:///cn=Read Replication Agreements,cn=permissions,cn=** >> pbac,$SUFFIX";) >> >> From the upstream documentation here: >> https://access.redhat.com/**site/documentation/en-US/Red_** >> Hat_Directory_Server/9.0/html-**single/Configuration_Command_** >> and_File_Reference/index.html#**Replication_Attributes_under_** >> cnreplica_cnsuffixName_**cnmapping_tree_cnconfig >> >> This would allow a user with Read Replication Agreements permission to >> be able to search for conflicts or tombstone records which would seem >> sane from a monitoring point of view... >> >> What do you think? >> > > I think this would be a separate issue. Being able to find the conflicting > issues leads directly to the question "what do I do with them?" That is > ticket https://fedorahosted.org/**freeipa/ticket/1025 > > Thanks Rob - I think it worthwhile adding the permissions in place to at least find them as a 'quick win' as it were ... What to do after that is an interesting question and would probably take a fair chuck of work to make it nicely visible plus show ways to resolve it. > > Also just to confirm the only thing I need to do with ACIs like this is >> to update the ldif (delegation.ldif and replica-acis.ldif) with the new >> role/privilege/permission and acis in install/share for the new installs >> and add an appropriate entry (not quite ldif) in install/updates to >> update the default schema of those updating in future, given no new >> attributes - right? >> > > You'll need to create a .update file in install/updates to modify an > existing installation. > > That's great - I had a look through the README in there and looking at other similar bits appears to be fairly simple. -------------- next part -------------- An HTML attachment was scrubbed... URL: From henry.hebert at roche.com Thu Aug 1 18:11:54 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Thu, 1 Aug 2013 14:11:54 -0400 Subject: [Freeipa-users] kinit - gui Message-ID: I have inherited an ipa system that has been running fantastic. However the gui is no longer functioning. I was wondering if this list has seen this sort of error in the past. hostname# kinit admin kinit: Clients credentials have been revoked while getting initial credentials so i then tried http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/using-the-ui.html#tab.ui-troubleshooting [hostname]# cat /tmp/moz.log 64608032[7fad03b53150]: using REQ_DELEGATE 64608032[7fad03b53150]: service = hostname 64608032[7fad03b53150]: using negotiate-gss 64608032[7fad03b53150]: entering nsAuthGSSAPI::nsAuthGSSAPI() 64608032[7fad03b53150]: Attempting to load gss functions 64608032[7fad03b53150]: entering nsAuthGSSAPI::Init() 64608032[7fad03b53150]: nsHttpNegotiateAuth::GenerateCredentials() [challenge=Negotiate] 64608032[7fad03b53150]: entering nsAuthGSSAPI::GetNextToken() 64608032[7fad03b53150]: gss_init_sec_context() failed: Unspecified GSS failure. Minor code may provide more information 64608032[7fad03b53150]: leaving nsAuthGSSAPI::GetNextToken [rv=80004005] Thanks in advance! Henry -- Henry Hebert System Administrator III -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 1 18:26:14 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 01 Aug 2013 14:26:14 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: Message-ID: <51FAA846.8050904@redhat.com> Hebert, Henry wrote: > I have inherited an ipa system that has been running fantastic. However > the gui is no longer functioning. I was wondering if this list has seen > this sort of error in the past. > > hostname# kinit admin > kinit: Clients credentials have been revoked while getting initial > credentials This is unrelated to the GUI. It appears that the admin account is disabled or locked due to too many failed logins. Using any other user, can you do ipa user-show admin? Look for: Account disabled: True If it is False then try ipa user-status admin see the number of failed logins. rob > > so i then tried > http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/using-the-ui.html#tab.ui-troubleshooting > > > [hostname]# cat /tmp/moz.log > 64608032[7fad03b53150]: using REQ_DELEGATE > 64608032[7fad03b53150]: service = hostname > 64608032[7fad03b53150]: using negotiate-gss > 64608032[7fad03b53150]: entering nsAuthGSSAPI::nsAuthGSSAPI() > 64608032[7fad03b53150]: Attempting to load gss functions > 64608032[7fad03b53150]: entering nsAuthGSSAPI::Init() > 64608032[7fad03b53150]: nsHttpNegotiateAuth::GenerateCredentials() > [challenge=Negotiate] > 64608032[7fad03b53150]: entering nsAuthGSSAPI::GetNextToken() > 64608032[7fad03b53150]: gss_init_sec_context() failed: Unspecified GSS > failure. Minor code may provide more information > 64608032[7fad03b53150]: leaving nsAuthGSSAPI::GetNextToken [rv=80004005] > > > Thanks in advance! > Henry > > -- > > Henry Hebert > System Administrator III > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From henry.hebert at roche.com Thu Aug 1 18:55:02 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Thu, 1 Aug 2013 14:55:02 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: <51FAA846.8050904@redhat.com> References: <51FAA846.8050904@redhat.com> Message-ID: Thank you for the respons Rob. [root at hostname ~]# ipa user-show admin User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash UID: #### GID: #### Account disabled: False Password: True Member of groups: admins, trust admins Indirect Member of HBAC rule: hostname Kerberos keys available: True [root at hostname ~]# [root at hostname ~]# [root at hostname ~]# [root at hostname ~]# ipa user-status admin ----------------------- Account disabled: False ----------------------- Server: hostname Failed logins: 12 Last successful authentication: 2013-07-25T13:14:27Z Last failed authentication: 2013-07-26T13:12:04Z Time now: 2013-08-01T18:52:44Z ---------------------------- Number of entries returned 1 ---------------------------- On Thu, Aug 1, 2013 at 2:26 PM, Rob Crittenden wrote: > Hebert, Henry wrote: > >> I have inherited an ipa system that has been running fantastic. However >> the gui is no longer functioning. I was wondering if this list has seen >> this sort of error in the past. >> >> hostname# kinit admin >> kinit: Clients credentials have been revoked while getting initial >> credentials >> > > This is unrelated to the GUI. It appears that the admin account is > disabled or locked due to too many failed logins. Using any other user, can > you do ipa user-show admin? > > Look for: > > Account disabled: True > > If it is False then try ipa user-status admin see the number of failed > logins. > > rob > > >> so i then tried >> http://docs.fedoraproject.org/**en-US/Fedora/17/html/FreeIPA_** >> Guide/using-the-ui.html#tab.**ui-troubleshooting >> >> >> [hostname]# cat /tmp/moz.log >> 64608032[7fad03b53150]: using REQ_DELEGATE >> 64608032[7fad03b53150]: service = hostname >> 64608032[7fad03b53150]: using negotiate-gss >> 64608032[7fad03b53150]: entering nsAuthGSSAPI::nsAuthGSSAPI() >> 64608032[7fad03b53150]: Attempting to load gss functions >> 64608032[7fad03b53150]: entering nsAuthGSSAPI::Init() >> 64608032[7fad03b53150]: nsHttpNegotiateAuth::**GenerateCredentials() >> [challenge=Negotiate] >> 64608032[7fad03b53150]: entering nsAuthGSSAPI::GetNextToken() >> 64608032[7fad03b53150]: gss_init_sec_context() failed: Unspecified GSS >> failure. Minor code may provide more information >> 64608032[7fad03b53150]: leaving nsAuthGSSAPI::GetNextToken [rv=80004005] >> >> >> Thanks in advance! >> Henry >> >> -- >> >> Henry Hebert >> System Administrator III >> >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> >> > -- Henry Hebert System Administrator III 454 Life Sciences A Roche Company 15 Commercial Street Branford, CT 06405 Phone +1 203 871 2249 Mobile +1 203 215 5904 e-mail henry.hebert at roche.com**** *Visit our new webpage, featuring the ?454 Sequencing breakthrough community webinar series? at www.454.com***** *Confidentiality Note* This message is intended only for the use of the named recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient, please contact the sender and delete the message. Any unauthorized use of the information contained in this message is prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 1 18:59:15 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 01 Aug 2013 14:59:15 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> Message-ID: <51FAB003.8000700@redhat.com> Hebert, Henry wrote: > Thank you for the respons Rob. > > > [root at hostname ~]# ipa user-show admin > User login: admin > Last name: Administrator > Home directory: /home/admin > Login shell: /bin/bash > UID: #### > GID: #### > Account disabled: False > Password: True > Member of groups: admins, trust admins > Indirect Member of HBAC rule: hostname > Kerberos keys available: True > [root at hostname ~]# > [root at hostname ~]# > [root at hostname ~]# > [root at hostname ~]# ipa user-status admin > ----------------------- > Account disabled: False > ----------------------- > Server: hostname > Failed logins: 12 > Last successful authentication: 2013-07-25T13:14:27Z > Last failed authentication: 2013-07-26T13:12:04Z > Time now: 2013-08-01T18:52:44Z > ---------------------------- > Number of entries returned 1 > ---------------------------- Sure seems like the password policy is preventing the login. You might try: ipa pwpolicy-show --user=admin Do you have any other users in the admins group? Do you know the Directory Manager password? (set during IPA install). rob > > > > > > > On Thu, Aug 1, 2013 at 2:26 PM, Rob Crittenden > wrote: > > Hebert, Henry wrote: > > I have inherited an ipa system that has been running fantastic. > However > the gui is no longer functioning. I was wondering if this list > has seen > this sort of error in the past. > > hostname# kinit admin > kinit: Clients credentials have been revoked while getting initial > credentials > > > This is unrelated to the GUI. It appears that the admin account is > disabled or locked due to too many failed logins. Using any other > user, can you do ipa user-show admin? > > Look for: > > Account disabled: True > > If it is False then try ipa user-status admin see the number of > failed logins. > > rob > > > so i then tried > http://docs.fedoraproject.org/__en-US/Fedora/17/html/FreeIPA___Guide/using-the-ui.html#tab.__ui-troubleshooting > > > > [hostname]# cat /tmp/moz.log > 64608032[7fad03b53150]: using REQ_DELEGATE > 64608032[7fad03b53150]: service = hostname > 64608032[7fad03b53150]: using negotiate-gss > 64608032[7fad03b53150]: entering nsAuthGSSAPI::nsAuthGSSAPI() > 64608032[7fad03b53150]: Attempting to load gss functions > 64608032[7fad03b53150]: entering nsAuthGSSAPI::Init() > 64608032[7fad03b53150]: nsHttpNegotiateAuth::__GenerateCredentials() > [challenge=Negotiate] > 64608032[7fad03b53150]: entering nsAuthGSSAPI::GetNextToken() > 64608032[7fad03b53150]: gss_init_sec_context() failed: > Unspecified GSS > failure. Minor code may provide more information > 64608032[7fad03b53150]: leaving nsAuthGSSAPI::GetNextToken > [rv=80004005] > > > Thanks in advance! > Henry > > -- > > Henry Hebert > System Administrator III > > > > _________________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/__mailman/listinfo/freeipa-users > > > > > > > -- > > Henry Hebert > System Administrator III > 454 Life Sciences > A Roche Company > > 15 Commercial Street > Branford, CT 06405 > Phone +1 203 871 2249 > Mobile +1 203 215 5904 > e-mail henry.hebert at roche.com ____ > > /Visit our new webpage, featuring the ?454 Sequencing breakthrough > community webinar series? at www.454.com /____ > > *Confidentiality Note* > This message is intended only for the use of the named recipient(s) and > may contain confidential and/or privileged information. If you are not > the intended recipient, please contact the sender and delete the > message. Any unauthorized use of the information contained in this > message is prohibited. > From henry.hebert at roche.com Thu Aug 1 19:11:58 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Thu, 1 Aug 2013 15:11:58 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: <51FAB003.8000700@redhat.com> References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> Message-ID: Aha! See Max failures below... [root at hostname ~]# ipa pwpolicy-show --user=admin Group: global_policy Max lifetime (days): 365 Min lifetime (hours): 1 History size: 1 Character classes: 1 Min length: 8 Max failures: 12 Failure reset interval: 0 Lockout duration: 0 is there a command like pam_tally2 for ipa to reset the number of failed logins? On Thu, Aug 1, 2013 at 2:59 PM, Rob Crittenden wrote: > Hebert, Henry wrote: > >> Thank you for the respons Rob. >> >> >> [root at hostname ~]# ipa user-show admin >> User login: admin >> Last name: Administrator >> Home directory: /home/admin >> Login shell: /bin/bash >> UID: #### >> GID: #### >> Account disabled: False >> Password: True >> Member of groups: admins, trust admins >> Indirect Member of HBAC rule: hostname >> Kerberos keys available: True >> [root at hostname ~]# >> [root at hostname ~]# >> [root at hostname ~]# >> [root at hostname ~]# ipa user-status admin >> ----------------------- >> Account disabled: False >> ----------------------- >> Server: hostname >> Failed logins: 12 >> Last successful authentication: 2013-07-25T13:14:27Z >> Last failed authentication: 2013-07-26T13:12:04Z >> Time now: 2013-08-01T18:52:44Z >> ---------------------------- >> Number of entries returned 1 >> ---------------------------- >> > > Sure seems like the password policy is preventing the login. You might > try: ipa pwpolicy-show --user=admin > > Do you have any other users in the admins group? > > Do you know the Directory Manager password? (set during IPA install). > > rob > > >> >> >> >> >> >> On Thu, Aug 1, 2013 at 2:26 PM, Rob Crittenden > > wrote: >> >> Hebert, Henry wrote: >> >> I have inherited an ipa system that has been running fantastic. >> However >> the gui is no longer functioning. I was wondering if this list >> has seen >> this sort of error in the past. >> >> hostname# kinit admin >> kinit: Clients credentials have been revoked while getting initial >> credentials >> >> >> This is unrelated to the GUI. It appears that the admin account is >> disabled or locked due to too many failed logins. Using any other >> user, can you do ipa user-show admin? >> >> Look for: >> >> Account disabled: True >> >> If it is False then try ipa user-status admin see the number of >> failed logins. >> >> rob >> >> >> so i then tried >> http://docs.fedoraproject.org/**__en-US/Fedora/17/html/** >> FreeIPA___Guide/using-the-ui.**html#tab.__ui-troubleshooting >> >> > FreeIPA_Guide/using-the-ui.**html#tab.ui-troubleshooting >> > >> >> >> [hostname]# cat /tmp/moz.log >> 64608032[7fad03b53150]: using REQ_DELEGATE >> 64608032[7fad03b53150]: service = hostname >> 64608032[7fad03b53150]: using negotiate-gss >> 64608032[7fad03b53150]: entering nsAuthGSSAPI::nsAuthGSSAPI() >> 64608032[7fad03b53150]: Attempting to load gss functions >> 64608032[7fad03b53150]: entering nsAuthGSSAPI::Init() >> 64608032[7fad03b53150]: nsHttpNegotiateAuth::__** >> GenerateCredentials() >> >> [challenge=Negotiate] >> 64608032[7fad03b53150]: entering nsAuthGSSAPI::GetNextToken() >> 64608032[7fad03b53150]: gss_init_sec_context() failed: >> Unspecified GSS >> failure. Minor code may provide more information >> 64608032[7fad03b53150]: leaving nsAuthGSSAPI::GetNextToken >> [rv=80004005] >> >> >> Thanks in advance! >> Henry >> >> -- >> >> Henry Hebert >> System Administrator III >> >> >> >> ______________________________**___________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> > >> https://www.redhat.com/__**mailman/listinfo/freeipa-users >> >> >> **> >> >> >> >> >> >> -- >> >> Henry Hebert >> System Administrator III >> 454 Life Sciences >> A Roche Company >> >> 15 Commercial Street >> Branford, CT 06405 >> Phone +1 203 871 2249 >> Mobile +1 203 215 5904 >> e-mail henry.hebert at roche.com ____ >> >> /Visit our new webpage, featuring the ?454 Sequencing breakthrough >> community webinar series? at www.454.com /____ >> >> *Confidentiality Note* >> >> This message is intended only for the use of the named recipient(s) and >> may contain confidential and/or privileged information. If you are not >> the intended recipient, please contact the sender and delete the >> message. Any unauthorized use of the information contained in this >> message is prohibited. >> >> > -- Henry Hebert System Administrator III 454 Life Sciences A Roche Company 15 Commercial Street Branford, CT 06405 Phone +1 203 871 2249 Mobile +1 203 215 5904 e-mail henry.hebert at roche.com**** *Visit our new webpage, featuring the ?454 Sequencing breakthrough community webinar series? at www.454.com***** *Confidentiality Note* This message is intended only for the use of the named recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient, please contact the sender and delete the message. Any unauthorized use of the information contained in this message is prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 1 20:24:15 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 01 Aug 2013 16:24:15 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> Message-ID: <51FAC3EF.6070907@redhat.com> Hebert, Henry wrote: > Aha! See Max failures below... > > [root at hostname ~]# ipa pwpolicy-show --user=admin > Group: global_policy > Max lifetime (days): 365 > Min lifetime (hours): 1 > History size: 1 > Character classes: 1 > Min length: 8 > Max failures: 12 > Failure reset interval: 0 > Lockout duration: 0 > > is there a command like pam_tally2 for ipa to reset the number of failed > logins? ipa user-unlock You need to be in the admins group to execute this. The account is permanently lock (until unlocked) because the lockout duration is 0, meaning forever. If you have the DM password we can use that account to unlock admin if you have no other users in the admins group. rob From henry.hebert at roche.com Thu Aug 1 20:43:04 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Thu, 1 Aug 2013 16:43:04 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: <51FAC3EF.6070907@redhat.com> References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> <51FAC3EF.6070907@redhat.com> Message-ID: My user is in the admins group however not in the "trust admins" Group name: admins Description: Account administrators group GID: 988200000 Member users: admin, XXXXXXXXX, hhebertXXX Member of HBAC rule: hostname Group name: trust admins Description: Trusts administrators group Member users: admin I ran the above command to the same results. [hhebertXXX at hostname ~]$ ipa user-unlock admin ipa: ERROR: did not receive Kerberos credentials I am asking the installer about the DM password. Again thx for all your help. Henry On Thu, Aug 1, 2013 at 4:24 PM, Rob Crittenden wrote: > Hebert, Henry wrote: > >> Aha! See Max failures below... >> >> [root at hostname ~]# ipa pwpolicy-show --user=admin >> Group: global_policy >> Max lifetime (days): 365 >> Min lifetime (hours): 1 >> History size: 1 >> Character classes: 1 >> Min length: 8 >> Max failures: 12 >> Failure reset interval: 0 >> Lockout duration: 0 >> >> is there a command like pam_tally2 for ipa to reset the number of failed >> logins? >> > > ipa user-unlock > > You need to be in the admins group to execute this. The account is > permanently lock (until unlocked) because the lockout duration is 0, > meaning forever. > > If you have the DM password we can use that account to unlock admin if you > have no other users in the admins group. > > rob > -------------- next part -------------- An HTML attachment was scrubbed... URL: From henry.hebert at roche.com Thu Aug 1 20:52:28 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Thu, 1 Aug 2013 16:52:28 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> <51FAC3EF.6070907@redhat.com> Message-ID: I have the DM password how do i unlock with it? ipa user-find doesn't show any user named Directory Manager? On Thu, Aug 1, 2013 at 4:43 PM, Henry Hebert wrote: > My user is in the admins group however not in the "trust admins" > > Group name: admins > Description: Account administrators group > GID: 988200000 > Member users: admin, XXXXXXXXX, hhebertXXX > Member of HBAC rule: hostname > > Group name: trust admins > Description: Trusts administrators group > Member users: admin > > I ran the above command to the same results. > > [hhebertXXX at hostname ~]$ ipa user-unlock admin > ipa: ERROR: did not receive Kerberos credentials > > I am asking the installer about the DM password. > > Again thx for all your help. > Henry > > > > On Thu, Aug 1, 2013 at 4:24 PM, Rob Crittenden wrote: > >> Hebert, Henry wrote: >> >>> Aha! See Max failures below... >>> >>> [root at hostname ~]# ipa pwpolicy-show --user=admin >>> Group: global_policy >>> Max lifetime (days): 365 >>> Min lifetime (hours): 1 >>> History size: 1 >>> Character classes: 1 >>> Min length: 8 >>> Max failures: 12 >>> Failure reset interval: 0 >>> Lockout duration: 0 >>> >>> is there a command like pam_tally2 for ipa to reset the number of failed >>> logins? >>> >> >> ipa user-unlock >> >> You need to be in the admins group to execute this. The account is >> permanently lock (until unlocked) because the lockout duration is 0, >> meaning forever. >> >> If you have the DM password we can use that account to unlock admin if >> you have no other users in the admins group. >> >> rob >> > > -- Henry Hebert System Administrator III 454 Life Sciences A Roche Company 15 Commercial Street Branford, CT 06405 Phone +1 203 871 2249 Mobile +1 203 215 5904 e-mail henry.hebert at roche.com**** *Visit our new webpage, featuring the ?454 Sequencing breakthrough community webinar series? at www.454.com***** *Confidentiality Note* This message is intended only for the use of the named recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient, please contact the sender and delete the message. Any unauthorized use of the information contained in this message is prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 1 21:48:54 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 01 Aug 2013 17:48:54 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> <51FAC3EF.6070907@redhat.com> Message-ID: <51FAD7C6.9010909@redhat.com> Hebert, Henry wrote: > My user is in the admins group however not in the "trust admins" > > Group name: admins > Description: Account administrators group > GID: 988200000 > Member users: admin, XXXXXXXXX, hhebertXXX > Member of HBAC rule: hostname > > Group name: trust admins > Description: Trusts administrators group > Member users: admin > > I ran the above command to the same results. admins is enough. > > [hhebertXXX at hostname ~]$ ipa user-unlock admin > ipa: ERROR: did not receive Kerberos credentials You need to kinit as yourself first. rob > > I am asking the installer about the DM password. > > Again thx for all your help. > Henry > > > > On Thu, Aug 1, 2013 at 4:24 PM, Rob Crittenden > wrote: > > Hebert, Henry wrote: > > Aha! See Max failures below... > > [root at hostname ~]# ipa pwpolicy-show --user=admin > Group: global_policy > Max lifetime (days): 365 > Min lifetime (hours): 1 > History size: 1 > Character classes: 1 > Min length: 8 > Max failures: 12 > Failure reset interval: 0 > Lockout duration: 0 > > is there a command like pam_tally2 for ipa to reset the number > of failed > logins? > > > ipa user-unlock > > You need to be in the admins group to execute this. The account is > permanently lock (until unlocked) because the lockout duration is 0, > meaning forever. > > If you have the DM password we can use that account to unlock admin > if you have no other users in the admins group. > > rob > > From henry.hebert at roche.com Fri Aug 2 14:15:23 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Fri, 2 Aug 2013 10:15:23 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> <51FAC3EF.6070907@redhat.com> Message-ID: Rob I tried the command. How do I unlock the account using the DM? [hhebertXXX at hostname ~]$ kinit hhebertXXX Password for hhebertXXX at dc.COM: [hhebertXXX at hostname ~]$* ipa user-unlock admin* ipa: ERROR: Server is unwilling to perform: Entry permanently locked. [hhebertXXX at hostname ~]$ and now my username is permanently locked. [hhebertXXX at hostname ~]$ ipa user-status hhebertXXX ipa: ERROR: Server is unwilling to perform: Entry permanently locked. On Thu, Aug 1, 2013 at 4:52 PM, Henry Hebert wrote: > I have the DM password how do i unlock with it? ipa user-find doesn't show > any user named Directory Manager? > > > On Thu, Aug 1, 2013 at 4:43 PM, Henry Hebert wrote: > >> My user is in the admins group however not in the "trust admins" >> >> Group name: admins >> Description: Account administrators group >> GID: 988200000 >> Member users: admin, XXXXXXXXX, hhebertXXX >> Member of HBAC rule: hostname >> >> Group name: trust admins >> Description: Trusts administrators group >> Member users: admin >> >> I ran the above command to the same results. >> >> [hhebertXXX at hostname ~]$ ipa user-unlock admin >> ipa: ERROR: did not receive Kerberos credentials >> >> I am asking the installer about the DM password. >> >> Again thx for all your help. >> Henry >> >> >> >> On Thu, Aug 1, 2013 at 4:24 PM, Rob Crittenden wrote: >> >>> Hebert, Henry wrote: >>> >>>> Aha! See Max failures below... >>>> >>>> [root at hostname ~]# ipa pwpolicy-show --user=admin >>>> Group: global_policy >>>> Max lifetime (days): 365 >>>> Min lifetime (hours): 1 >>>> History size: 1 >>>> Character classes: 1 >>>> Min length: 8 >>>> Max failures: 12 >>>> Failure reset interval: 0 >>>> Lockout duration: 0 >>>> >>>> is there a command like pam_tally2 for ipa to reset the number of failed >>>> logins? >>>> >>> >>> ipa user-unlock >>> >>> You need to be in the admins group to execute this. The account is >>> permanently lock (until unlocked) because the lockout duration is 0, >>> meaning forever. >>> >>> If you have the DM password we can use that account to unlock admin if >>> you have no other users in the admins group. >>> >>> rob >>> >> >> > > > -- > > Henry Hebert > System Administrator III > 454 Life Sciences > A Roche Company > > 15 Commercial Street > Branford, CT 06405 > Phone +1 203 871 2249 > Mobile +1 203 215 5904 > e-mail henry.hebert at roche.com**** > > *Visit our new webpage, featuring the ?454 Sequencing breakthrough > community webinar series? at www.454.com***** > > *Confidentiality Note* > This message is intended only for the use of the named recipient(s) and > may contain confidential and/or privileged information. If you are not the > intended recipient, please contact the sender and delete the message. Any > unauthorized use of the information contained in this message is prohibited. > -- Henry Hebert System Administrator III 454 Life Sciences A Roche Company 15 Commercial Street Branford, CT 06405 Phone +1 203 871 2249 Mobile +1 203 215 5904 e-mail henry.hebert at roche.com**** *Visit our new webpage, featuring the ?454 Sequencing breakthrough community webinar series? at www.454.com***** *Confidentiality Note* This message is intended only for the use of the named recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient, please contact the sender and delete the message. Any unauthorized use of the information contained in this message is prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From henry.hebert at roche.com Fri Aug 2 14:55:33 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Fri, 2 Aug 2013 10:55:33 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> <51FAC3EF.6070907@redhat.com> Message-ID: I found this. http://directory.fedoraproject.org/wiki/Howto:PasswordReset Still trying to get the syntax down correctly but I think this is what I am looking for. On Fri, Aug 2, 2013 at 10:15 AM, Henry Hebert wrote: > Rob I tried the command. How do I unlock the account using the DM? > > [hhebertXXX at hostname ~]$ kinit hhebertXXX > Password for hhebertXXX at dc.COM: > > [hhebertXXX at hostname ~]$* ipa user-unlock admin* > ipa: ERROR: Server is unwilling to perform: Entry permanently locked. > [hhebertXXX at hostname ~]$ > > and now my username is permanently locked. > > [hhebertXXX at hostname ~]$ ipa user-status hhebertXXX > ipa: ERROR: Server is unwilling to perform: Entry permanently locked. > > > > > On Thu, Aug 1, 2013 at 4:52 PM, Henry Hebert wrote: > >> I have the DM password how do i unlock with it? ipa user-find doesn't >> show any user named Directory Manager? >> >> >> On Thu, Aug 1, 2013 at 4:43 PM, Henry Hebert wrote: >> >>> My user is in the admins group however not in the "trust admins" >>> >>> Group name: admins >>> Description: Account administrators group >>> GID: 988200000 >>> Member users: admin, XXXXXXXXX, hhebertXXX >>> Member of HBAC rule: hostname >>> >>> Group name: trust admins >>> Description: Trusts administrators group >>> Member users: admin >>> >>> I ran the above command to the same results. >>> >>> [hhebertXXX at hostname ~]$ ipa user-unlock admin >>> ipa: ERROR: did not receive Kerberos credentials >>> >>> I am asking the installer about the DM password. >>> >>> Again thx for all your help. >>> Henry >>> >>> >>> >>> On Thu, Aug 1, 2013 at 4:24 PM, Rob Crittenden wrote: >>> >>>> Hebert, Henry wrote: >>>> >>>>> Aha! See Max failures below... >>>>> >>>>> [root at hostname ~]# ipa pwpolicy-show --user=admin >>>>> Group: global_policy >>>>> Max lifetime (days): 365 >>>>> Min lifetime (hours): 1 >>>>> History size: 1 >>>>> Character classes: 1 >>>>> Min length: 8 >>>>> Max failures: 12 >>>>> Failure reset interval: 0 >>>>> Lockout duration: 0 >>>>> >>>>> is there a command like pam_tally2 for ipa to reset the number of >>>>> failed >>>>> logins? >>>>> >>>> >>>> ipa user-unlock >>>> >>>> You need to be in the admins group to execute this. The account is >>>> permanently lock (until unlocked) because the lockout duration is 0, >>>> meaning forever. >>>> >>>> If you have the DM password we can use that account to unlock admin if >>>> you have no other users in the admins group. >>>> >>>> rob >>>> >>> >>> >> >> >> -- >> >> Henry Hebert >> System Administrator III >> 454 Life Sciences >> A Roche Company >> >> 15 Commercial Street >> Branford, CT 06405 >> Phone +1 203 871 2249 >> Mobile +1 203 215 5904 >> e-mail henry.hebert at roche.com**** >> >> *Visit our new webpage, featuring the ?454 Sequencing breakthrough >> community webinar series? at www.454.com***** >> >> *Confidentiality Note* >> This message is intended only for the use of the named recipient(s) and >> may contain confidential and/or privileged information. If you are not the >> intended recipient, please contact the sender and delete the message. Any >> unauthorized use of the information contained in this message is prohibited. >> > > > > -- > > Henry Hebert > System Administrator III > 454 Life Sciences > A Roche Company > > 15 Commercial Street > Branford, CT 06405 > Phone +1 203 871 2249 > Mobile +1 203 215 5904 > e-mail henry.hebert at roche.com**** > > *Visit our new webpage, featuring the ?454 Sequencing breakthrough > community webinar series? at www.454.com***** > > *Confidentiality Note* > This message is intended only for the use of the named recipient(s) and > may contain confidential and/or privileged information. If you are not the > intended recipient, please contact the sender and delete the message. Any > unauthorized use of the information contained in this message is prohibited. > -- Henry Hebert System Administrator III 454 Life Sciences A Roche Company 15 Commercial Street Branford, CT 06405 Phone +1 203 871 2249 Mobile +1 203 215 5904 e-mail henry.hebert at roche.com**** *Visit our new webpage, featuring the ?454 Sequencing breakthrough community webinar series? at www.454.com***** *Confidentiality Note* This message is intended only for the use of the named recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient, please contact the sender and delete the message. Any unauthorized use of the information contained in this message is prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sakodak at gmail.com Fri Aug 2 17:55:12 2013 From: sakodak at gmail.com (KodaK) Date: Fri, 2 Aug 2013 12:55:12 -0500 Subject: [Freeipa-users] Sanity check on hbac rule on "foreign" domains. Message-ID: First, before we go any further: is it supported to use sssd when the client machines domain differs from the realm name? If not, then the rest of this is moot. Client box is a RHEL 5.something. I didn't do "ipa-client-install" because I wanted to configure by hand as a test. The client box has a DNS name of stlmoracsbx01.domain.com, and the realm is UNIX.DOMAIN.COM I've configured the box with sssd, and I can log in with my personal credentials because I have a wide-open rule for admins. I've created a simple rule for a test user, and it's not working. [xxx at slpidml01 ~]$ ipa hbacrule-show stlmoracsbx01-access Rule name: stlmoracsbx01-access Source host category: all Service category: all Enabled: TRUE Users: testuser Hosts: stlmoracsbx01.domain.com However: [xxx at slpidml01 ~]$ ipa hbactest --user=testuser --host=stlmoracsbx01.domain.com --service=sshd --------------------- Access granted: False --------------------- And my access: [xxx at slpidml01 ~]$ ipa hbactest --user=xxx --host=stlmoracsbx01.domain.com --service=sshd -------------------- Access granted: True -------------------- Matched rules: admin access I also tried opening that host up to everyone: [jebalicki at slpidml01 ~]$ ipa hbacrule-show stlmoracsbx01-access Rule name: stlmoracsbx01-access User category: all Source host category: all Service category: all Enabled: TRUE Hosts: stlmoracsbx01.domain.com But the rule fails. I thought maybe there might be something with the user "testuser", so I tried another user and I still get a failure. Any ideas would be appreciated. -- The government is going to read our mail anyway, might as well make it tough for them. GPG Public key ID: B6A1A7C6 From henry.hebert at roche.com Fri Aug 2 18:18:45 2013 From: henry.hebert at roche.com (Hebert, Henry) Date: Fri, 2 Aug 2013 14:18:45 -0400 Subject: [Freeipa-users] kinit - gui In-Reply-To: References: <51FAA846.8050904@redhat.com> <51FAB003.8000700@redhat.com> <51FAC3EF.6070907@redhat.com> Message-ID: Resolution was a little different than the URL fedora project url. ldapmodify -x -D "cn=directory manager" -w *your bind password (for simple authentication)* dn: uid=admin,cn=users,cn=accounts,dc=XXX,dc=XXX,dc=com changetype: modify delete: krbLoginFailedCount (Ctrl-D) ipa user-status admin now shows zero. Thanks for all your help Rob. Henry On Fri, Aug 2, 2013 at 10:55 AM, Henry Hebert wrote: > I found this. http://directory.fedoraproject.org/wiki/Howto:PasswordReset > Still trying to get the syntax down correctly but I think this is what I > am looking for. > > > > > > > > On Fri, Aug 2, 2013 at 10:15 AM, Henry Hebert wrote: > >> Rob I tried the command. How do I unlock the account using the DM? >> >> [hhebertXXX at hostname ~]$ kinit hhebertXXX >> Password for hhebertXXX at dc.COM: >> >> [hhebertXXX at hostname ~]$* ipa user-unlock admin* >> ipa: ERROR: Server is unwilling to perform: Entry permanently locked. >> [hhebertXXX at hostname ~]$ >> >> and now my username is permanently locked. >> >> [hhebertXXX at hostname ~]$ ipa user-status hhebertXXX >> ipa: ERROR: Server is unwilling to perform: Entry permanently locked. >> >> >> >> >> On Thu, Aug 1, 2013 at 4:52 PM, Henry Hebert wrote: >> >>> I have the DM password how do i unlock with it? ipa user-find doesn't >>> show any user named Directory Manager? >>> >>> >>> On Thu, Aug 1, 2013 at 4:43 PM, Henry Hebert wrote: >>> >>>> My user is in the admins group however not in the "trust admins" >>>> >>>> Group name: admins >>>> Description: Account administrators group >>>> GID: 988200000 >>>> Member users: admin, XXXXXXXXX, hhebertXXX >>>> Member of HBAC rule: hostname >>>> >>>> Group name: trust admins >>>> Description: Trusts administrators group >>>> Member users: admin >>>> >>>> I ran the above command to the same results. >>>> >>>> [hhebertXXX at hostname ~]$ ipa user-unlock admin >>>> ipa: ERROR: did not receive Kerberos credentials >>>> >>>> I am asking the installer about the DM password. >>>> >>>> Again thx for all your help. >>>> Henry >>>> >>>> >>>> >>>> On Thu, Aug 1, 2013 at 4:24 PM, Rob Crittenden wrote: >>>> >>>>> Hebert, Henry wrote: >>>>> >>>>>> Aha! See Max failures below... >>>>>> >>>>>> [root at hostname ~]# ipa pwpolicy-show --user=admin >>>>>> Group: global_policy >>>>>> Max lifetime (days): 365 >>>>>> Min lifetime (hours): 1 >>>>>> History size: 1 >>>>>> Character classes: 1 >>>>>> Min length: 8 >>>>>> Max failures: 12 >>>>>> Failure reset interval: 0 >>>>>> Lockout duration: 0 >>>>>> >>>>>> is there a command like pam_tally2 for ipa to reset the number of >>>>>> failed >>>>>> logins? >>>>>> >>>>> >>>>> ipa user-unlock >>>>> >>>>> You need to be in the admins group to execute this. The account is >>>>> permanently lock (until unlocked) because the lockout duration is 0, >>>>> meaning forever. >>>>> >>>>> If you have the DM password we can use that account to unlock admin if >>>>> you have no other users in the admins group. >>>>> >>>>> rob >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> Henry Hebert >>> System Administrator III >>> 454 Life Sciences >>> A Roche Company >>> >>> 15 Commercial Street >>> Branford, CT 06405 >>> Phone +1 203 871 2249 >>> Mobile +1 203 215 5904 >>> e-mail henry.hebert at roche.com**** >>> >>> *Visit our new webpage, featuring the ?454 Sequencing breakthrough >>> community webinar series? at www.454.com***** >>> >>> *Confidentiality Note* >>> This message is intended only for the use of the named recipient(s) and >>> may contain confidential and/or privileged information. If you are not the >>> intended recipient, please contact the sender and delete the message. Any >>> unauthorized use of the information contained in this message is prohibited. >>> >> >> >> >> -- >> >> Henry Hebert >> System Administrator III >> 454 Life Sciences >> A Roche Company >> >> 15 Commercial Street >> Branford, CT 06405 >> Phone +1 203 871 2249 >> Mobile +1 203 215 5904 >> e-mail henry.hebert at roche.com**** >> >> *Visit our new webpage, featuring the ?454 Sequencing breakthrough >> community webinar series? at www.454.com***** >> >> *Confidentiality Note* >> This message is intended only for the use of the named recipient(s) and >> may contain confidential and/or privileged information. If you are not the >> intended recipient, please contact the sender and delete the message. Any >> unauthorized use of the information contained in this message is prohibited. >> > > > > -- > > Henry Hebert > System Administrator III > 454 Life Sciences > A Roche Company > > 15 Commercial Street > Branford, CT 06405 > Phone +1 203 871 2249 > Mobile +1 203 215 5904 > e-mail henry.hebert at roche.com**** > > *Visit our new webpage, featuring the ?454 Sequencing breakthrough > community webinar series? at www.454.com***** > > *Confidentiality Note* > This message is intended only for the use of the named recipient(s) and > may contain confidential and/or privileged information. If you are not the > intended recipient, please contact the sender and delete the message. Any > unauthorized use of the information contained in this message is prohibited. > -- Henry Hebert System Administrator III 454 Life Sciences A Roche Company 15 Commercial Street Branford, CT 06405 Phone +1 203 871 2249 Mobile +1 203 215 5904 e-mail henry.hebert at roche.com**** *Visit our new webpage, featuring the ?454 Sequencing breakthrough community webinar series? at www.454.com***** *Confidentiality Note* This message is intended only for the use of the named recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient, please contact the sender and delete the message. Any unauthorized use of the information contained in this message is prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Mon Aug 5 09:23:39 2013 From: sbose at redhat.com (Sumit Bose) Date: Mon, 5 Aug 2013 11:23:39 +0200 Subject: [Freeipa-users] Sanity check on hbac rule on "foreign" domains. In-Reply-To: References: Message-ID: <20130805092339.GO3648@localhost.localdomain> On Fri, Aug 02, 2013 at 12:55:12PM -0500, KodaK wrote: > First, before we go any further: is it supported to use > sssd when the client machines domain differs from > the realm name? If not, then the rest of this is moot. > > Client box is a RHEL 5.something. I didn't do "ipa-client-install" > because I wanted to configure by hand as a test. The client > box has a DNS name of stlmoracsbx01.domain.com, and the > realm is UNIX.DOMAIN.COM > > I've configured the box with sssd, and I can log in with my personal > credentials because I have a wide-open rule for admins. > > I've created a simple rule for a test user, and it's not working. > > [xxx at slpidml01 ~]$ ipa hbacrule-show stlmoracsbx01-access > Rule name: stlmoracsbx01-access > Source host category: all > Service category: all > Enabled: TRUE > Users: testuser > Hosts: stlmoracsbx01.domain.com > > However: > > [xxx at slpidml01 ~]$ ipa hbactest --user=testuser > --host=stlmoracsbx01.domain.com --service=sshd > --------------------- > Access granted: False > --------------------- > > And my access: > > [xxx at slpidml01 ~]$ ipa hbactest --user=xxx > --host=stlmoracsbx01.domain.com --service=sshd > -------------------- > Access granted: True > -------------------- > Matched rules: admin access > > I also tried opening that host up to everyone: > > [jebalicki at slpidml01 ~]$ ipa hbacrule-show stlmoracsbx01-access > > Rule name: stlmoracsbx01-access > User category: all > Source host category: all > Service category: all > Enabled: TRUE > Hosts: stlmoracsbx01.domain.com > > But the rule fails. > > I thought maybe there might be something with the user "testuser", so > I tried another > user and I still get a failure. > > Any ideas would be appreciated. First I think this is not a general issue. I did a quick test which worked as expected: [root at ipa18-devel ~]# ipa hbacrule-show abc-test Rule name: abc-test User category: all Service category: all Enabled: TRUE Hosts: abc.def [root at ipa18-devel ~]# ipa hbactest --user=qgwe --host=abc.def --service=wced -------------------- Access granted: True -------------------- Matched rules: abc-test [root at ipa18-devel ~]# ipa hbactest --user=qgwe --host=abc.defx --service=wced --------------------- Access granted: False --------------------- Not matched rules: abc-test Which version of FreeIPA are you using on the server? Maybe the sssd logs at a high debug level will give more details why the access is denied you you try to log in with ssh as testuser on stlmoracsbx01.domain.com. bye, Sumit > > -- > The government is going to read our mail anyway, might as well make it > tough for them. GPG Public key ID: B6A1A7C6 > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users From john.moyer at digitalreasoning.com Tue Aug 6 03:17:38 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Mon, 5 Aug 2013 23:17:38 -0400 Subject: [Freeipa-users] IPA Load Problems? Message-ID: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> Hello, So I've been preparing my infrastructure for a big change from an older openldap system to a nice new IPA server. I have a redundant secondary server and snapshots taken daily. I populated all my user data into IPA, and gave the users a week to set a password. They all did this and the big switch was this past weekend. We had done previous tests on each server and it all worked. We switched this past weekend and it worked great. This morning a light load hit it (since I've only put a small fraction of our servers on it about 15) and the primary came to it's knees. Processor spiked, and logs started to fill (didn't fill at this point). I then decided it's probably a glitch (I'm an optimist) so I restarted IPA services. They all restarted except for named which crashed (which then caused everything to stop). I looked and now the disk was full. So I trash the logs (had no easy place to put them at the time which I regret now) and I restart the services again. IPA fully crashes now (didn't even start the DIRSRV for my domain). So here are my questions: 1. Any idea what caused this? Any performance issues that have been seen? 2. Are the connection settings for IPA good out of the box? I ask because in RHDS (in the first versions I used) the default connection timeouts were a MAJOR issue, I used to run a network of 400 servers and I had to set the time-outs to >30sec which made my servers run really really well, but if I used the 60 min defaults they also would come to their knees. Is there a buried setting like this? (However, I must admit there didn't seem like there were a lot of connections like when I had the issue with the 400 servers years ago). Also is there an easy place to set log rotation settings? (If it's log rotate just let me know, I just don't want to step on an internal app rotate). Thanks, _____________________________________________________ John Moyer Director, IT Operations -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From stephane.neveu at thalesgroup.com Tue Aug 6 08:48:01 2013 From: stephane.neveu at thalesgroup.com (NEVEU Stephane) Date: Tue, 6 Aug 2013 10:48:01 +0200 Subject: [Freeipa-users] Install error pkispawn Message-ID: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> Hi guys, New & trying to install FreeIPA-server with the online documentation on a fresh fedora 19... I've got this error message : Any idea is welcome :) Thank you ... Continue to configure the system with these values? [no]: yes The following operations may take some minutes to complete. Please wait until the prompt is returned. Configuring NTP daemon (ntpd) [1/4]: stopping ntpd [2/4]: writing configuration [3/4]: configuring ntpd to start on boot [4/4]: starting ntpd Done configuring NTP daemon (ntpd). Configuring directory server (dirsrv): Estimated time 1 minute [1/38]: creating directory server user [2/38]: creating directory server instance [3/38]: adding default schema [4/38]: enabling memberof plugin [5/38]: enabling winsync plugin [6/38]: configuring replication version plugin [7/38]: enabling IPA enrollment plugin [8/38]: enabling ldapi [9/38]: configuring uniqueness plugin [10/38]: configuring uuid plugin [11/38]: configuring modrdn plugin [12/38]: configuring DNS plugin [13/38]: enabling entryUSN plugin [14/38]: configuring lockout plugin [15/38]: creating indices [16/38]: enabling referential integrity plugin [17/38]: configuring certmap.conf [18/38]: configure autobind for root [19/38]: configure new location for managed entries [20/38]: configure dirsrv ccache [21/38]: enable SASL mapping fallback [22/38]: restarting directory server [23/38]: adding default layout [24/38]: adding delegation layout [25/38]: creating container for managed entries [26/38]: configuring user private groups [27/38]: configuring netgroups from hostgroups [28/38]: creating default Sudo bind user [29/38]: creating default Auto Member layout [30/38]: adding range check plugin [31/38]: creating default HBAC rule allow_all [32/38]: initializing group membership [33/38]: adding master entry [34/38]: configuring Posix uid/gid generation [35/38]: adding replication acis [36/38]: enabling compatibility plugin [37/38]: tuning directory server [38/38]: configuring directory to start on boot Done configuring directory server (dirsrv). Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds [1/20]: creating certificate server user [2/20]: configuring certificate server instance ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpFi7bLc' returned non-zero exit status 1 Configuration of CA failed -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkosek at redhat.com Tue Aug 6 11:47:32 2013 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 06 Aug 2013 13:47:32 +0200 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> Message-ID: <5200E254.3040900@redhat.com> On 08/06/2013 10:48 AM, NEVEU Stephane wrote: > Hi guys, > > New & trying to install FreeIPA-server with the online documentation on a fresh fedora 19... I've got this error message : > Any idea is welcome :) > Thank you > ... > Continue to configure the system with these values? [no]: yes > > The following operations may take some minutes to complete. > Please wait until the prompt is returned. > > Configuring NTP daemon (ntpd) > [1/4]: stopping ntpd > [2/4]: writing configuration > [3/4]: configuring ntpd to start on boot > [4/4]: starting ntpd > Done configuring NTP daemon (ntpd). > Configuring directory server (dirsrv): Estimated time 1 minute > [1/38]: creating directory server user > [2/38]: creating directory server instance > [3/38]: adding default schema > [4/38]: enabling memberof plugin > [5/38]: enabling winsync plugin > [6/38]: configuring replication version plugin > [7/38]: enabling IPA enrollment plugin > [8/38]: enabling ldapi > [9/38]: configuring uniqueness plugin > [10/38]: configuring uuid plugin > [11/38]: configuring modrdn plugin > [12/38]: configuring DNS plugin > [13/38]: enabling entryUSN plugin > [14/38]: configuring lockout plugin > [15/38]: creating indices > [16/38]: enabling referential integrity plugin > [17/38]: configuring certmap.conf > [18/38]: configure autobind for root > [19/38]: configure new location for managed entries > [20/38]: configure dirsrv ccache > [21/38]: enable SASL mapping fallback > [22/38]: restarting directory server > [23/38]: adding default layout > [24/38]: adding delegation layout > [25/38]: creating container for managed entries > [26/38]: configuring user private groups > [27/38]: configuring netgroups from hostgroups > [28/38]: creating default Sudo bind user > [29/38]: creating default Auto Member layout > [30/38]: adding range check plugin > [31/38]: creating default HBAC rule allow_all > [32/38]: initializing group membership > [33/38]: adding master entry > [34/38]: configuring Posix uid/gid generation > [35/38]: adding replication acis > [36/38]: enabling compatibility plugin > [37/38]: tuning directory server > [38/38]: configuring directory to start on boot > Done configuring directory server (dirsrv). > Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds > [1/20]: creating certificate server user > [2/20]: configuring certificate server instance > ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpFi7bLc' returned non-zero exit status 1 > Configuration of CA failed > Hello Stephane, Thanks for contacting the list! We need to get at first more information about the failure, i.e.: 1) $ rpm -qa freeipa-server pki-ca "java-*-openjdk-*" 2) Related errors from /var/log/ipaserver-install.log 3) Related errors from /var/log/pki/pki-tomcat/catalina.out (if any) 4) # ausearch -m AVC Martin From stephane.neveu at thalesgroup.com Tue Aug 6 12:22:40 2013 From: stephane.neveu at thalesgroup.com (NEVEU Stephane) Date: Tue, 6 Aug 2013 14:22:40 +0200 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <5200E254.3040900@redhat.com> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> <5200E254.3040900@redhat.com> Message-ID: <32122_1375791763_5200EA92_32122_11510_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782D3@THSONEA01CMS03P.one.grp> Hi Martin & thank you for your reply :) I added the update-testing repositories on fedora 19 after reading this : http://www.redhat.com/archives/freeipa-users/2013-June/msg00099.html But nothing changed, I also tried with selinux disabled/enabled but same issue... Here we go : [root at omcsvcipa01d ~]# rpm -qa freeipa-server pki-ca "java-*-openjdk-*" java-1.7.0-openjdk-devel-1.7.0.25-2.3.12.3.fc19.x86_64 freeipa-server-3.2.2-1.fc19.x86_64 pki-ca-10.0.4-2.fc19.noarch [root at omcsvcipa01d ~]# ausearch -m AVC ---- time->Tue Aug 6 08:07:36 2013 type=SYSCALL msg=audit(1375776456.741:125): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7fd5080076e0 a2=90800 a3=0 items=0 ppid=1 pid=1995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC msg=audit(1375776456.741:125): avc: denied { read } for pid=1995 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir ---- time->Tue Aug 6 08:07:36 2013 type=SYSCALL msg=audit(1375776456.741:126): arch=c000003e syscall=2 success=no exit=-13 a0=7fd508007700 a1=242 a2=180 a3=0 items=0 ppid=1 pid=1995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC msg=audit(1375776456.741:126): avc: denied { write } for pid=1995 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir ---- time->Tue Aug 6 08:19:15 2013 type=SYSCALL msg=audit(1375777155.023:174): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7f33540072b0 a2=90800 a3=0 items=0 ppid=2713 pid=2734 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC msg=audit(1375777155.023:174): avc: denied { read } for pid=2734 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir ---- time->Tue Aug 6 08:19:15 2013 type=SYSCALL msg=audit(1375777155.023:175): arch=c000003e syscall=2 success=no exit=-13 a0=7f33540072d0 a1=242 a2=180 a3=0 items=0 ppid=2713 pid=2734 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC msg=audit(1375777155.023:175): avc: denied { write } for pid=2734 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir Errors on the ipaserver-install.log : ... pki_subsystem_nickname = subsystemCert cert-pki-ca pki_ocsp_signing_nickname = ocspSigningCert cert-pki-ca pki_ssl_server_nickname = Server-Cert cert-pki-ca pki_audit_signing_nickname = auditSigningCert cert-pki-ca pki_ca_signing_nickname = caSigningCert cert-pki-ca 2013-08-06T12:05:08Z DEBUG Starting external process 2013-08-06T12:05:08Z DEBUG args=/usr/sbin/pkispawn -s CA -f /tmp/tmpRlQD7m 2013-08-06T12:05:09Z DEBUG Process finished, return code=1 2013-08-06T12:05:09Z DEBUG stdout=Loading deployment configuration from /tmp/tmpRlQD7m. Installing CA into /var/lib/pki/pki-tomcat. Installation failed. 2013-08-06T12:05:09Z DEBUG stderr=pkispawn : ERROR ....... PKI subsystem 'CA' for instance 'pki-tomcat' already exists! 2013-08-06T12:05:09Z CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpRlQD7m' returned non-zero exit status 1 2013-08-06T12:05:09Z DEBUG File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 616, in run_script return_value = main_function() File "/sbin/ipa-server-install", line 1022, in main dm_password, subject_base=options.subject) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 617, in configure_instance self.start_creation(runtime=210) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 363, in start_creation method() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 736, in __spawn_instance raise RuntimeError('Configuration of CA failed') 2013-08-06T12:05:09Z DEBUG The ipa-server-install command failed, exception: RuntimeError: Configuration of CA failed And catalina.out : Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://omcsvcipa01d.dev.cloud-omc.thales:9080/ca/ocsp' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'false' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=true,ssl3=true,tls=true' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,+SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8080"] Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8443"] JSSSocketFactory init - exception thrown:java.lang.NullPointerException Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-127.0.0.1-8009"] Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1488 ms Aug 06, 2013 8:07:38 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Aug 06, 2013 8:07:38 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.40 Aug 06, 2013 8:07:39 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/pki Aug 06, 2013 8:07:41 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/ca SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback SSLAuthenticatorWithFallback: Setting container SSLAuthenticatorWithFallback: Initializing authenticators SSLAuthenticatorWithFallback: Starting authenticators 08:07:43,538 DEBUG (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to retrieve ServletContext: expandEntityReferences defaults to true 08:07:43,545 DEBUG (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to retrieve ServletContext: expandEntityReferences defaults to true CMS Warning: FAILURE: Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate|FAILURE: authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value| Server is started. Aug 06, 2013 8:07:44 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/ROOT Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8080"] Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8443"] Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"] Aug 06, 2013 8:07:45 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 6725 ms Aug 06, 2013 8:19:15 AM org.apache.catalina.core.StandardServer await INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance. Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler ["http-bio-8080"] Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler ["http-bio-8443"] Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause INFO: Pausing ProtocolHandler ["ajp-bio-127.0.0.1-8009"] Aug 06, 2013 8:19:15 AM org.apache.catalina.core.StandardService stopInternal INFO: Stopping service Catalina -----Message d'origine----- De : Martin Kosek [mailto:mkosek at redhat.com] Envoy? : mardi 6 ao?t 2013 13:48 ? : NEVEU Stephane Cc : freeipa-users at redhat.com Objet : Re: [Freeipa-users] Install error pkispawn On 08/06/2013 10:48 AM, NEVEU Stephane wrote: > Hi guys, > > New & trying to install FreeIPA-server with the online documentation on a fresh fedora 19... I've got this error message : > Any idea is welcome :) > Thank you > ... > Continue to configure the system with these values? [no]: yes > > The following operations may take some minutes to complete. > Please wait until the prompt is returned. > > Configuring NTP daemon (ntpd) > [1/4]: stopping ntpd > [2/4]: writing configuration > [3/4]: configuring ntpd to start on boot > [4/4]: starting ntpd > Done configuring NTP daemon (ntpd). > Configuring directory server (dirsrv): Estimated time 1 minute > [1/38]: creating directory server user > [2/38]: creating directory server instance > [3/38]: adding default schema > [4/38]: enabling memberof plugin > [5/38]: enabling winsync plugin > [6/38]: configuring replication version plugin > [7/38]: enabling IPA enrollment plugin > [8/38]: enabling ldapi > [9/38]: configuring uniqueness plugin > [10/38]: configuring uuid plugin > [11/38]: configuring modrdn plugin > [12/38]: configuring DNS plugin > [13/38]: enabling entryUSN plugin > [14/38]: configuring lockout plugin > [15/38]: creating indices > [16/38]: enabling referential integrity plugin > [17/38]: configuring certmap.conf > [18/38]: configure autobind for root > [19/38]: configure new location for managed entries > [20/38]: configure dirsrv ccache > [21/38]: enable SASL mapping fallback > [22/38]: restarting directory server > [23/38]: adding default layout > [24/38]: adding delegation layout > [25/38]: creating container for managed entries > [26/38]: configuring user private groups > [27/38]: configuring netgroups from hostgroups > [28/38]: creating default Sudo bind user > [29/38]: creating default Auto Member layout > [30/38]: adding range check plugin > [31/38]: creating default HBAC rule allow_all > [32/38]: initializing group membership > [33/38]: adding master entry > [34/38]: configuring Posix uid/gid generation > [35/38]: adding replication acis > [36/38]: enabling compatibility plugin > [37/38]: tuning directory server > [38/38]: configuring directory to start on boot Done configuring > directory server (dirsrv). > Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds > [1/20]: creating certificate server user > [2/20]: configuring certificate server instance > ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpFi7bLc' returned non-zero exit status 1 > Configuration of CA failed > Hello Stephane, Thanks for contacting the list! We need to get at first more information about the failure, i.e.: 1) $ rpm -qa freeipa-server pki-ca "java-*-openjdk-*" 2) Related errors from /var/log/ipaserver-install.log 3) Related errors from /var/log/pki/pki-tomcat/catalina.out (if any) 4) # ausearch -m AVC Martin From mkosek at redhat.com Tue Aug 6 12:44:57 2013 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 06 Aug 2013 14:44:57 +0200 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <32122_1375791763_5200EA92_32122_11510_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782D3@THSONEA01CMS03P.one.grp> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> <5200E254.3040900@redhat.com> <32122_1375791763_5200EA92_32122_11510_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782D3@THSONEA01CMS03P.one.grp> Message-ID: <5200EFC9.6070809@redhat.com> Thanks! I see there are some SELinux issues for accessing /tmp/hsperfdata_root, they look strange. But what seems even stranger is this error in /var/log/ipaserver_install.log: 2013-08-06T12:05:09Z DEBUG stderr=pkispawn : ERROR ....... PKI subsystem 'CA' for instance 'pki-tomcat' already exists! Did you try to install IPA server before? This procedure may help to re-install: # ipa-server-install --uninstall --unattended # pkidestroy -s CA -i pki-tomcat Second command is to make sure that PKI instance is not left configured on the system. After these 2 commads, you can try to install IPA server again. If that fails again, second thing we can try is to: 1) Run the clean up commands as above again 2) Turn SELinux to permissive with "# setenforce 0" 3) Run IPA server installation again I hope that these procedures will now lead to successful installation :-) Martin On 08/06/2013 02:22 PM, NEVEU Stephane wrote: > > Hi Martin & thank you for your reply :) > > I added the update-testing repositories on fedora 19 after reading this : http://www.redhat.com/archives/freeipa-users/2013-June/msg00099.html > But nothing changed, I also tried with selinux disabled/enabled but same issue... > > > Here we go : > > [root at omcsvcipa01d ~]# rpm -qa freeipa-server pki-ca "java-*-openjdk-*" > java-1.7.0-openjdk-devel-1.7.0.25-2.3.12.3.fc19.x86_64 > freeipa-server-3.2.2-1.fc19.x86_64 > pki-ca-10.0.4-2.fc19.noarch > > [root at omcsvcipa01d ~]# ausearch -m AVC > ---- > time->Tue Aug 6 08:07:36 2013 > type=SYSCALL msg=audit(1375776456.741:125): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7fd5080076e0 a2=90800 a3=0 items=0 ppid=1 pid=1995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) > type=AVC msg=audit(1375776456.741:125): avc: denied { read } for pid=1995 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > ---- > time->Tue Aug 6 08:07:36 2013 > type=SYSCALL msg=audit(1375776456.741:126): arch=c000003e syscall=2 success=no exit=-13 a0=7fd508007700 a1=242 a2=180 a3=0 items=0 ppid=1 pid=1995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) > type=AVC msg=audit(1375776456.741:126): avc: denied { write } for pid=1995 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > ---- > time->Tue Aug 6 08:19:15 2013 > type=SYSCALL msg=audit(1375777155.023:174): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7f33540072b0 a2=90800 a3=0 items=0 ppid=2713 pid=2734 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) > type=AVC msg=audit(1375777155.023:174): avc: denied { read } for pid=2734 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > ---- > time->Tue Aug 6 08:19:15 2013 > type=SYSCALL msg=audit(1375777155.023:175): arch=c000003e syscall=2 success=no exit=-13 a0=7f33540072d0 a1=242 a2=180 a3=0 items=0 ppid=2713 pid=2734 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) > type=AVC msg=audit(1375777155.023:175): avc: denied { write } for pid=2734 comm="java" name="hsperfdata_root" dev="vda1" ino=39527 scontext=system_u:system_r:pki_tomcat_t:s0 tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > > Errors on the ipaserver-install.log : > ... > pki_subsystem_nickname = subsystemCert cert-pki-ca > pki_ocsp_signing_nickname = ocspSigningCert cert-pki-ca > pki_ssl_server_nickname = Server-Cert cert-pki-ca > pki_audit_signing_nickname = auditSigningCert cert-pki-ca > pki_ca_signing_nickname = caSigningCert cert-pki-ca > > > 2013-08-06T12:05:08Z DEBUG Starting external process > 2013-08-06T12:05:08Z DEBUG args=/usr/sbin/pkispawn -s CA -f /tmp/tmpRlQD7m > 2013-08-06T12:05:09Z DEBUG Process finished, return code=1 > 2013-08-06T12:05:09Z DEBUG stdout=Loading deployment configuration from /tmp/tmpRlQD7m. > Installing CA into /var/lib/pki/pki-tomcat. > Installation failed. > > > 2013-08-06T12:05:09Z DEBUG stderr=pkispawn : ERROR ....... PKI subsystem 'CA' for instance 'pki-tomcat' already exists! > > 2013-08-06T12:05:09Z CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpRlQD7m' returned non-zero exit status 1 > 2013-08-06T12:05:09Z DEBUG File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 616, in run_script > return_value = main_function() > > File "/sbin/ipa-server-install", line 1022, in main > dm_password, subject_base=options.subject) > > File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 617, in configure_instance > self.start_creation(runtime=210) > > File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 363, in start_creation > method() > > File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 736, in __spawn_instance > raise RuntimeError('Configuration of CA failed') > > 2013-08-06T12:05:09Z DEBUG The ipa-server-install command failed, exception: RuntimeError: Configuration of CA failed > > And catalina.out : > > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://omcsvcipa01d.dev.cloud-omc.thales:9080/ca/ocsp' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=true,ssl3=true,tls=true' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,+SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.tomcat.util.digester.SetPropertiesRule begin > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.tomcat.util.digester.SetPropertiesRule begin > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-bio-8080"] > Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-bio-8443"] > JSSSocketFactory init - exception thrown:java.lang.NullPointerException > > Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["ajp-bio-127.0.0.1-8009"] > Aug 06, 2013 8:07:38 AM org.apache.catalina.startup.Catalina load > INFO: Initialization processed in 1488 ms > Aug 06, 2013 8:07:38 AM org.apache.catalina.core.StandardService startInternal > INFO: Starting service Catalina > Aug 06, 2013 8:07:38 AM org.apache.catalina.core.StandardEngine startInternal > INFO: Starting Servlet Engine: Apache Tomcat/7.0.40 > Aug 06, 2013 8:07:39 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/pki > Aug 06, 2013 8:07:41 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/ca > SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback > SSLAuthenticatorWithFallback: Setting container > SSLAuthenticatorWithFallback: Initializing authenticators > SSLAuthenticatorWithFallback: Starting authenticators > 08:07:43,538 DEBUG (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to retrieve ServletContext: expandEntityReferences defaults to true > 08:07:43,545 DEBUG (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to retrieve ServletContext: expandEntityReferences defaults to true > CMS Warning: FAILURE: Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate|FAILURE: authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value| > Server is started. > Aug 06, 2013 8:07:44 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/ROOT > Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8080"] > Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8443"] > Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"] > Aug 06, 2013 8:07:45 AM org.apache.catalina.startup.Catalina start > INFO: Server startup in 6725 ms > Aug 06, 2013 8:19:15 AM org.apache.catalina.core.StandardServer await > INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance. > Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["http-bio-8080"] > Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["http-bio-8443"] > Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["ajp-bio-127.0.0.1-8009"] > Aug 06, 2013 8:19:15 AM org.apache.catalina.core.StandardService stopInternal > INFO: Stopping service Catalina > > > > > > -----Message d'origine----- > De : Martin Kosek [mailto:mkosek at redhat.com] > Envoy? : mardi 6 ao?t 2013 13:48 > ? : NEVEU Stephane > Cc : freeipa-users at redhat.com > Objet : Re: [Freeipa-users] Install error pkispawn > > On 08/06/2013 10:48 AM, NEVEU Stephane wrote: >> Hi guys, >> >> New & trying to install FreeIPA-server with the online documentation on a fresh fedora 19... I've got this error message : >> Any idea is welcome :) >> Thank you >> ... >> Continue to configure the system with these values? [no]: yes >> >> The following operations may take some minutes to complete. >> Please wait until the prompt is returned. >> >> Configuring NTP daemon (ntpd) >> [1/4]: stopping ntpd >> [2/4]: writing configuration >> [3/4]: configuring ntpd to start on boot >> [4/4]: starting ntpd >> Done configuring NTP daemon (ntpd). >> Configuring directory server (dirsrv): Estimated time 1 minute >> [1/38]: creating directory server user >> [2/38]: creating directory server instance >> [3/38]: adding default schema >> [4/38]: enabling memberof plugin >> [5/38]: enabling winsync plugin >> [6/38]: configuring replication version plugin >> [7/38]: enabling IPA enrollment plugin >> [8/38]: enabling ldapi >> [9/38]: configuring uniqueness plugin >> [10/38]: configuring uuid plugin >> [11/38]: configuring modrdn plugin >> [12/38]: configuring DNS plugin >> [13/38]: enabling entryUSN plugin >> [14/38]: configuring lockout plugin >> [15/38]: creating indices >> [16/38]: enabling referential integrity plugin >> [17/38]: configuring certmap.conf >> [18/38]: configure autobind for root >> [19/38]: configure new location for managed entries >> [20/38]: configure dirsrv ccache >> [21/38]: enable SASL mapping fallback >> [22/38]: restarting directory server >> [23/38]: adding default layout >> [24/38]: adding delegation layout >> [25/38]: creating container for managed entries >> [26/38]: configuring user private groups >> [27/38]: configuring netgroups from hostgroups >> [28/38]: creating default Sudo bind user >> [29/38]: creating default Auto Member layout >> [30/38]: adding range check plugin >> [31/38]: creating default HBAC rule allow_all >> [32/38]: initializing group membership >> [33/38]: adding master entry >> [34/38]: configuring Posix uid/gid generation >> [35/38]: adding replication acis >> [36/38]: enabling compatibility plugin >> [37/38]: tuning directory server >> [38/38]: configuring directory to start on boot Done configuring >> directory server (dirsrv). >> Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds >> [1/20]: creating certificate server user >> [2/20]: configuring certificate server instance >> ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpFi7bLc' returned non-zero exit status 1 >> Configuration of CA failed >> > > Hello Stephane, > > Thanks for contacting the list! We need to get at first more information about the failure, i.e.: > > 1) $ rpm -qa freeipa-server pki-ca "java-*-openjdk-*" > 2) Related errors from /var/log/ipaserver-install.log > 3) Related errors from /var/log/pki/pki-tomcat/catalina.out (if any) > 4) # ausearch -m AVC > > Martin > From stephane.neveu at thalesgroup.com Tue Aug 6 13:03:14 2013 From: stephane.neveu at thalesgroup.com (NEVEU Stephane) Date: Tue, 6 Aug 2013 15:03:14 +0200 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <5200EFC9.6070809@redhat.com> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> <5200E254.3040900@redhat.com> <32122_1375791763_5200EA92_32122_11510_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782D3@THSONEA01CMS03P.one.grp> <5200EFC9.6070809@redhat.com> Message-ID: <32122_1375794196_5200F414_32122_12967_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782FE@THSONEA01CMS03P.one.grp> Yes I've tried to make the installation twice ... Setenforce 0 & the pkidestroy -s CA -i pki-tomcat solved my issue ! Setup completed... Many thanks Martin for your help :) -----Message d'origine----- De : Martin Kosek [mailto:mkosek at redhat.com] Envoy? : mardi 6 ao?t 2013 14:45 ? : NEVEU Stephane Cc : freeipa-users at redhat.com Objet : Re: [Freeipa-users] Install error pkispawn Thanks! I see there are some SELinux issues for accessing /tmp/hsperfdata_root, they look strange. But what seems even stranger is this error in /var/log/ipaserver_install.log: 2013-08-06T12:05:09Z DEBUG stderr=pkispawn : ERROR ....... PKI subsystem 'CA' for instance 'pki-tomcat' already exists! Did you try to install IPA server before? This procedure may help to re-install: # ipa-server-install --uninstall --unattended # pkidestroy -s CA -i pki-tomcat Second command is to make sure that PKI instance is not left configured on the system. After these 2 commads, you can try to install IPA server again. If that fails again, second thing we can try is to: 1) Run the clean up commands as above again 2) Turn SELinux to permissive with "# setenforce 0" 3) Run IPA server installation again I hope that these procedures will now lead to successful installation :-) Martin On 08/06/2013 02:22 PM, NEVEU Stephane wrote: > > Hi Martin & thank you for your reply :) > > I added the update-testing repositories on fedora 19 after reading > this : > http://www.redhat.com/archives/freeipa-users/2013-June/msg00099.html > But nothing changed, I also tried with selinux disabled/enabled but same issue... > > > Here we go : > > [root at omcsvcipa01d ~]# rpm -qa freeipa-server pki-ca "java-*-openjdk-*" > java-1.7.0-openjdk-devel-1.7.0.25-2.3.12.3.fc19.x86_64 > freeipa-server-3.2.2-1.fc19.x86_64 > pki-ca-10.0.4-2.fc19.noarch > > [root at omcsvcipa01d ~]# ausearch -m AVC > ---- > time->Tue Aug 6 08:07:36 2013 > type=SYSCALL msg=audit(1375776456.741:125): arch=c000003e syscall=257 > success=no exit=-13 a0=ffffffffffffff9c a1=7fd5080076e0 a2=90800 a3=0 > items=0 ppid=1 pid=1995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 > fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" > exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre > /bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC > msg=audit(1375776456.741:125): avc: denied { read } for pid=1995 > comm="java" name="hsperfdata_root" dev="vda1" ino=39527 > scontext=system_u:system_r:pki_tomcat_t:s0 > tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > ---- > time->Tue Aug 6 08:07:36 2013 > type=SYSCALL msg=audit(1375776456.741:126): arch=c000003e syscall=2 > success=no exit=-13 a0=7fd508007700 a1=242 a2=180 a3=0 items=0 ppid=1 > pid=1995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 > sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" > exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre > /bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC > msg=audit(1375776456.741:126): avc: denied { write } for pid=1995 > comm="java" name="hsperfdata_root" dev="vda1" ino=39527 > scontext=system_u:system_r:pki_tomcat_t:s0 > tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > ---- > time->Tue Aug 6 08:19:15 2013 > type=SYSCALL msg=audit(1375777155.023:174): arch=c000003e syscall=257 > success=no exit=-13 a0=ffffffffffffff9c a1=7f33540072b0 a2=90800 a3=0 > items=0 ppid=2713 pid=2734 auid=4294967295 uid=0 gid=0 euid=0 suid=0 > fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" > exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre > /bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC > msg=audit(1375777155.023:174): avc: denied { read } for pid=2734 > comm="java" name="hsperfdata_root" dev="vda1" ino=39527 > scontext=system_u:system_r:pki_tomcat_t:s0 > tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > ---- > time->Tue Aug 6 08:19:15 2013 > type=SYSCALL msg=audit(1375777155.023:175): arch=c000003e syscall=2 > success=no exit=-13 a0=7f33540072d0 a1=242 a2=180 a3=0 items=0 > ppid=2713 pid=2734 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 > egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="java" > exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre > /bin/java" subj=system_u:system_r:pki_tomcat_t:s0 key=(null) type=AVC > msg=audit(1375777155.023:175): avc: denied { write } for pid=2734 > comm="java" name="hsperfdata_root" dev="vda1" ino=39527 > scontext=system_u:system_r:pki_tomcat_t:s0 > tcontext=unconfined_u:object_r:rpm_script_tmp_t:s0 tclass=dir > > Errors on the ipaserver-install.log : > ... > pki_subsystem_nickname = subsystemCert cert-pki-ca > pki_ocsp_signing_nickname = ocspSigningCert cert-pki-ca > pki_ssl_server_nickname = Server-Cert cert-pki-ca > pki_audit_signing_nickname = auditSigningCert cert-pki-ca > pki_ca_signing_nickname = caSigningCert cert-pki-ca > > > 2013-08-06T12:05:08Z DEBUG Starting external process > 2013-08-06T12:05:08Z DEBUG args=/usr/sbin/pkispawn -s CA -f > /tmp/tmpRlQD7m 2013-08-06T12:05:09Z DEBUG Process finished, return > code=1 2013-08-06T12:05:09Z DEBUG stdout=Loading deployment configuration from /tmp/tmpRlQD7m. > Installing CA into /var/lib/pki/pki-tomcat. > Installation failed. > > > 2013-08-06T12:05:09Z DEBUG stderr=pkispawn : ERROR ....... PKI subsystem 'CA' for instance 'pki-tomcat' already exists! > > 2013-08-06T12:05:09Z CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpRlQD7m' returned non-zero exit status 1 > 2013-08-06T12:05:09Z DEBUG File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 616, in run_script > return_value = main_function() > > File "/sbin/ipa-server-install", line 1022, in main > dm_password, subject_base=options.subject) > > File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 617, in configure_instance > self.start_creation(runtime=210) > > File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 363, in start_creation > method() > > File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 736, in __spawn_instance > raise RuntimeError('Configuration of CA failed') > > 2013-08-06T12:05:09Z DEBUG The ipa-server-install command failed, > exception: RuntimeError: Configuration of CA failed > > And catalina.out : > > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'enableOCSP' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderURL' to 'http://omcsvcipa01d.dev.cloud-omc.thales:9080/ca/ocsp' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspResponderCertNickname' to 'ocspSigningCert cert-pki-ca' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspCacheSize' to '1000' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMinCacheEntryDuration' to '60' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspMaxCacheEntryDuration' to '120' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ocspTimeout' to '10' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'strictCiphers' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'sslOptions' to 'ssl2=true,ssl3=true,tls=true' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl2Ciphers' to '-SSL2_RC4_128_WITH_MD5,-SSL2_RC4_128_EXPORT40_WITH_MD5,-SSL2_RC2_128_CBC_WITH_MD5,-SSL2_RC2_128_CBC_EXPORT40_WITH_MD5,-SSL2_DES_64_CBC_WITH_MD5,-SSL2_DES_192_EDE3_CBC_WITH_MD5' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'ssl3Ciphers' to '-SSL3_FORTEZZA_DMS_WITH_NULL_SHA,-SSL3_FORTEZZA_DMS_WITH_RC4_128_SHA,+SSL3_RSA_WITH_RC4_128_SHA,-SSL3_RSA_EXPORT_WITH_RC4_40_MD5,+SSL3_RSA_WITH_3DES_EDE_CBC_SHA,+SSL3_RSA_WITH_DES_CBC_SHA,-SSL3_RSA_EXPORT_WITH_RC2_CBC_40_MD5,-SSL3_FORTEZZA_DMS_WITH_FORTEZZA_CBC_SHA,-SSL_RSA_FIPS_WITH_DES_CBC_SHA,+SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA,-SSL3_RSA_WITH_NULL_MD5,-TLS_RSA_EXPORT1024_WITH_RC4_56_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'tlsCiphers' to '-TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,-TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,-TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_DSS_WITH_AES_128_CBC_SHA,+TLS_DHE_DSS_WITH_AES_256_CBC_SHA,+TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,+TLS_DHE_RSA_WITH_AES_128_CBC_SHA,+TLS_DHE_RSA_WITH_AES_256_CBC_SHA' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'serverCertNickFile' to '/var/lib/pki/pki-tomcat/conf/serverCertNick.conf' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordFile' to '/var/lib/pki/pki-tomcat/conf/password.conf' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'passwordClass' to 'org.apache.tomcat.util.net.jss.PlainPasswordFile' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.catalina.startup.SetAllPropertiesRule begin > WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'certdbDir' to '/var/lib/pki/pki-tomcat/alias' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.tomcat.util.digester.SetPropertiesRule begin > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlValidation' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM > org.apache.tomcat.util.digester.SetPropertiesRule begin > WARNING: [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'xmlNamespaceAware' to 'false' did not find a matching property. > Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-bio-8080"] Aug 06, 2013 > 8:07:38 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-bio-8443"] JSSSocketFactory > init - exception thrown:java.lang.NullPointerException > > Aug 06, 2013 8:07:38 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["ajp-bio-127.0.0.1-8009"] Aug 06, > 2013 8:07:38 AM org.apache.catalina.startup.Catalina load > INFO: Initialization processed in 1488 ms Aug 06, 2013 8:07:38 AM > org.apache.catalina.core.StandardService startInternal > INFO: Starting service Catalina > Aug 06, 2013 8:07:38 AM org.apache.catalina.core.StandardEngine > startInternal > INFO: Starting Servlet Engine: Apache Tomcat/7.0.40 Aug 06, 2013 > 8:07:39 AM org.apache.catalina.startup.HostConfig deployDirectory > INFO: Deploying web application directory > /var/lib/pki/pki-tomcat/webapps/pki > Aug 06, 2013 8:07:41 AM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deploying web application directory > /var/lib/pki/pki-tomcat/webapps/ca > SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback > SSLAuthenticatorWithFallback: Setting container > SSLAuthenticatorWithFallback: Initializing authenticators > SSLAuthenticatorWithFallback: Starting authenticators > 08:07:43,538 DEBUG > (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to > retrieve ServletContext: expandEntityReferences defaults to true > 08:07:43,545 DEBUG > (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to > retrieve ServletContext: expandEntityReferences defaults to true CMS Warning: FAILURE: Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate|FAILURE: authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value| Server is started. > Aug 06, 2013 8:07:44 AM org.apache.catalina.startup.HostConfig > deployDirectory > INFO: Deploying web application directory > /var/lib/pki/pki-tomcat/webapps/ROOT > Aug 06, 2013 8:07:45 AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8080"] Aug 06, 2013 8:07:45 > AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["http-bio-8443"] Aug 06, 2013 8:07:45 > AM org.apache.coyote.AbstractProtocol start > INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"] Aug 06, 2013 > 8:07:45 AM org.apache.catalina.startup.Catalina start > INFO: Server startup in 6725 ms > Aug 06, 2013 8:19:15 AM org.apache.catalina.core.StandardServer await > INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance. > Aug 06, 2013 8:19:15 AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["http-bio-8080"] Aug 06, 2013 8:19:15 > AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["http-bio-8443"] Aug 06, 2013 8:19:15 > AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["ajp-bio-127.0.0.1-8009"] Aug 06, 2013 > 8:19:15 AM org.apache.catalina.core.StandardService stopInternal > INFO: Stopping service Catalina > > > > > > -----Message d'origine----- > De : Martin Kosek [mailto:mkosek at redhat.com] Envoy? : mardi 6 ao?t > 2013 13:48 ? : NEVEU Stephane Cc : freeipa-users at redhat.com Objet : > Re: [Freeipa-users] Install error pkispawn > > On 08/06/2013 10:48 AM, NEVEU Stephane wrote: >> Hi guys, >> >> New & trying to install FreeIPA-server with the online documentation on a fresh fedora 19... I've got this error message : >> Any idea is welcome :) >> Thank you >> ... >> Continue to configure the system with these values? [no]: yes >> >> The following operations may take some minutes to complete. >> Please wait until the prompt is returned. >> >> Configuring NTP daemon (ntpd) >> [1/4]: stopping ntpd >> [2/4]: writing configuration >> [3/4]: configuring ntpd to start on boot >> [4/4]: starting ntpd >> Done configuring NTP daemon (ntpd). >> Configuring directory server (dirsrv): Estimated time 1 minute >> [1/38]: creating directory server user >> [2/38]: creating directory server instance >> [3/38]: adding default schema >> [4/38]: enabling memberof plugin >> [5/38]: enabling winsync plugin >> [6/38]: configuring replication version plugin >> [7/38]: enabling IPA enrollment plugin >> [8/38]: enabling ldapi >> [9/38]: configuring uniqueness plugin >> [10/38]: configuring uuid plugin >> [11/38]: configuring modrdn plugin >> [12/38]: configuring DNS plugin >> [13/38]: enabling entryUSN plugin >> [14/38]: configuring lockout plugin >> [15/38]: creating indices >> [16/38]: enabling referential integrity plugin >> [17/38]: configuring certmap.conf >> [18/38]: configure autobind for root >> [19/38]: configure new location for managed entries >> [20/38]: configure dirsrv ccache >> [21/38]: enable SASL mapping fallback >> [22/38]: restarting directory server >> [23/38]: adding default layout >> [24/38]: adding delegation layout >> [25/38]: creating container for managed entries >> [26/38]: configuring user private groups >> [27/38]: configuring netgroups from hostgroups >> [28/38]: creating default Sudo bind user >> [29/38]: creating default Auto Member layout >> [30/38]: adding range check plugin >> [31/38]: creating default HBAC rule allow_all >> [32/38]: initializing group membership >> [33/38]: adding master entry >> [34/38]: configuring Posix uid/gid generation >> [35/38]: adding replication acis >> [36/38]: enabling compatibility plugin >> [37/38]: tuning directory server >> [38/38]: configuring directory to start on boot Done configuring >> directory server (dirsrv). >> Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds >> [1/20]: creating certificate server user >> [2/20]: configuring certificate server instance >> ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpFi7bLc' returned non-zero exit status 1 >> Configuration of CA failed >> > > Hello Stephane, > > Thanks for contacting the list! We need to get at first more information about the failure, i.e.: > > 1) $ rpm -qa freeipa-server pki-ca "java-*-openjdk-*" > 2) Related errors from /var/log/ipaserver-install.log > 3) Related errors from /var/log/pki/pki-tomcat/catalina.out (if any) > 4) # ausearch -m AVC > > Martin > From sakodak at gmail.com Tue Aug 6 13:34:23 2013 From: sakodak at gmail.com (KodaK) Date: Tue, 6 Aug 2013 08:34:23 -0500 Subject: [Freeipa-users] Sanity check on hbac rule on "foreign" domains. In-Reply-To: <20130805092339.GO3648@localhost.localdomain> References: <20130805092339.GO3648@localhost.localdomain> Message-ID: On Mon, Aug 5, 2013 at 4:23 AM, Sumit Bose wrote: > Which version of FreeIPA are you using on the server? Maybe the sssd > logs at a high debug level will give more details why the access is > denied you you try to log in with ssh as testuser on > stlmoracsbx01.domain.com. Something must have been cached, somewhere. (Even though I cleared every cache I could think of.) I haven't had time until now; I just tried again and allowed users work and disallowed users don't. I have no idea. Thanks, --Jason From rmeggins at redhat.com Tue Aug 6 14:57:43 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Tue, 06 Aug 2013 08:57:43 -0600 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> Message-ID: <52010EE7.8090402@redhat.com> On 08/05/2013 09:17 PM, John Moyer wrote: > Hello, > > So I've been preparing my infrastructure for a big change from an > older openldap system to a nice new IPA server. I have a redundant > secondary server and snapshots taken daily. I populated all my user > data into IPA, and gave the users a week to set a password. They all > did this and the big switch was this past weekend. We had done > previous tests on each server and it all worked. We switched this > past weekend and it worked great. > > This morning a light load hit it (since I've only put a small fraction > of our servers on it about 15) and the primary came to it's knees. What platform? What version of ipa? What version of 389-ds-base? What was the nature of the load? Search requests? Update requests? Updates from replication? The logconv.pl tool can be used to analyze the 389-ds-base access logs. During this time of the load, are there any errors in the errors log? > Processor spiked, and logs started to fill (didn't fill at this point). I'm not sure what you mean by "logs started to fill (didn't fill at this point)." > I then decided it's probably a glitch (I'm an optimist) so I > restarted IPA services. They all restarted except for named which > crashed (which then caused everything to stop). I looked and now the > disk was full. Which directory contained the files that caused the disk to become full? /var/log? /var/lib? Somewhere else? > So I trash the logs (had no easy place to put them at the time which I > regret now) and I restart the services again. What do you mean by "trash the logs"? > IPA fully crashes now (didn't even start the DIRSRV for my domain). Which component of IPA is crashing? If it is dirsrv that is refusing to start, is it crashing? What's in /var/log/dirsrv/slapd-*/errors? If it is crashing, we will need a core file and/or stack trace - see http://port389.org/wiki/FAQ#Debugging_Crashes > > So here are my questions: > > 1. Any idea what caused this? Any performance issues that have been > seen? It could be almost anything given the above information. > > 2. Are the connection settings for IPA good out of the box? I ask > because in RHDS (in the first versions I used) the default connection > timeouts were a MAJOR issue, How so? Details? > I used to run a network of 400 servers and I had to set the time-outs > to >30sec which made my servers run really really well, Exactly which timeout settings are you talking about? > but if I used the 60 min defaults they also would come to their knees. > Is there a buried setting like this? (However, I must admit there > didn't seem like there were a lot of connections like when I had the > issue with the 400 servers years ago). > > Also is there an easy place to set log rotation settings? (If it's > log rotate just let me know, I just don't want to step on an internal > app rotate). IPA does not provide a GUI nor a command line utility for managing 389 logging settings. This document gives an example of how logs are managed using the RHDS GUI (not available when using IPA). https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Monitoring_Server_and_Database_Activity.html#types-of-log-files However, all of these correspond to settings which you can set via ldapmodify: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnconfig-nsslapd_accesslog_logexpirationtime_Access_Log_Expiration_Time There are several attributes which control access log rotation parameters. > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Tue Aug 6 15:15:21 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 06 Aug 2013 11:15:21 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> Message-ID: <52011309.1070401@redhat.com> John Moyer wrote: > Hello, > > So I've been preparing my infrastructure for a big change from an older > openldap system to a nice new IPA server. I have a redundant secondary > server and snapshots taken daily. I populated all my user data into > IPA, and gave the users a week to set a password. They all did this and > the big switch was this past weekend. We had done previous tests on > each server and it all worked. We switched this past weekend and it > worked great. > > This morning a light load hit it (since I've only put a small fraction > of our servers on it about 15) and the primary came to it's knees. > Processor spiked, and logs started to fill (didn't fill at this > point). I then decided it's probably a glitch (I'm an optimist) so I > restarted IPA services. They all restarted except for named which > crashed (which then caused everything to stop). I looked and now the > disk was full. So I trash the logs (had no easy place to put them at > the time which I regret now) and I restart the services again. IPA > fully crashes now (didn't even start the DIRSRV for my domain). What error do you see in the 389-ds error log when the server fails to start? > So here are my questions: > > 1. Any idea what caused this? Any performance issues that have been seen? No, the logs would have really helped here. I don't recall any other reports like this. > 2. Are the connection settings for IPA good out of the box? I ask > because in RHDS (in the first versions I used) the default connection > timeouts were a MAJOR issue, I used to run a network of 400 servers and > I had to set the time-outs to >30sec which made my servers run really > really well, but if I used the 60 min defaults they also would come to > their knees. Is there a buried setting like this? (However, I must > admit there didn't seem like there were a lot of connections like when I > had the issue with the 400 servers years ago). What does your IPA topology look like? How many clients are we talking about? > > Also is there an easy place to set log rotation settings? (If it's log > rotate just let me know, I just don't want to step on an internal app > rotate). It uses internal log rotation, https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Configuring_Logs.html#Manual_Log_File_Rotation rob From davis.goodman at digital-district.ca Tue Aug 6 21:31:34 2013 From: davis.goodman at digital-district.ca (Davis Goodman) Date: Tue, 6 Aug 2013 17:31:34 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login Message-ID: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> Hi, I have an FreeIPA server configured, managed to configure a Mountain Lion Client for automounts and user logins. My issue is that whenever I first login with a user the "New Password" box shows up and even if I try to change the password the box keeps reappearing without any success. If I log onto the machine with the local admin user and try to get a ticket for this user I get a "New Password" prompt. From there I can change the password and I get a ticket without an issue. After that I can login through the GUI without being asked for a new password. Anyone has seen this behaviour before? -- Davis Goodman Directeur Informatique | IT Manager 5605 Avenue de Gasp?, Suite 408 | Montr?al, QC H2T 2A4 T?l: +1 (514) 360-3253 x104 Cell: +1 (514) 994-7360 From sakodak at gmail.com Tue Aug 6 23:14:01 2013 From: sakodak at gmail.com (KodaK) Date: Tue, 6 Aug 2013 18:14:01 -0500 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> Message-ID: On Tue, Aug 6, 2013 at 4:31 PM, Davis Goodman wrote: > Hi, > > I have an FreeIPA server configured, managed to configure a Mountain Lion Client for automounts and user logins. > > My issue is that whenever I first login with a user the "New Password" box shows up and even if I try to change the password the box keeps reappearing without any success. > > If I log onto the machine with the local admin user and try to get a ticket for this user I get a "New Password" prompt. From there I can change the password and I get a ticket without an issue. After that I can login through the GUI without being asked for a new password. > > Anyone has seen this behaviour before? That's the expected behavior. When you set the user's password as an admin, it sets the "force a password change" flag. I don't know anything aobut OSX, but there may be a way to configure the login GUI to deal with the password change correctly. Failing that, you can use a web based password change utility and let users do self service, or if you don't want that you can set up a special password administrator you can use that when it sets passwords it doesn't force a change (bad idea.) For setting up either, you need to do this: http://www.freeipa.org/page/PasswordSynchronization for the password change user. This is the web based password change utility I chose to use, but there are others -- or you can roll your own: http://ltb-project.org/wiki/documentation/self-service-password --Jason -- The government is going to read our mail anyway, might as well make it tough for them. GPG Public key ID: B6A1A7C6 From lroot at redhat.com Wed Aug 7 00:25:44 2013 From: lroot at redhat.com (Lynn Root) Date: Tue, 6 Aug 2013 17:25:44 -0700 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> Message-ID: On Aug 6, 2013, at 4:14 PM, KodaK wrote: > On Tue, Aug 6, 2013 at 4:31 PM, Davis Goodman > wrote: >> Hi, >> >> I have an FreeIPA server configured, managed to configure a Mountain Lion Client for automounts and user logins. >> >> My issue is that whenever I first login with a user the "New Password" box shows up and even if I try to change the password the box keeps reappearing without any success. >> >> If I log onto the machine with the local admin user and try to get a ticket for this user I get a "New Password" prompt. From there I can change the password and I get a ticket without an issue. After that I can login through the GUI without being asked for a new password. >> >> Anyone has seen this behaviour before? > > That's the expected behavior. When you set the user's password as an > admin, it sets the "force a password change" flag. Correct me if I'm wrong, but it's not expect to *not* be able to change the password on an IPA client after the initial setup, and be forced to use the IPA Server to re-set the password. Granted, the client is OSX. However, I personally have experience the inability to change a new user's password on an IPA client, and only on the IPA Server. Unfortunately, I've been trying to reproduce this and I can not. I've tried on Fedora 19, and will try on RHEL next. Davis - Can you let me know your IPA Server and IPA Client versions? As well as the OS that the IPA Server is on? Also, out of curiosity, do you have directions on how you set up the client on Mac OSX? Thanks! Lynn Root Lynn Root @roguelynn Associate Software Engineer From brian_lee1 at jabil.com Wed Aug 7 13:29:29 2013 From: brian_lee1 at jabil.com (Brian Lee) Date: Wed, 7 Aug 2013 09:29:29 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> Message-ID: Hi Lynn, I just checked this in my lab setup: - Set up a new user on the FreeIPA server as 'ipatest'. - Logged in to a Linux client configured for FreeIPA, it prompted me to change my password. - Successfully changed my password for ipatest. Verified this on another machine. - Furthermore, I reset the "Password Policy" min lifetime to 0 and typed passwd on one of the ipa clients while logged in as ipatest. This worked without issue. I also have FreeIPA set up in the lab with a domain trust to a 2008 R2 AD server, so I checked to see if the results would be the same. - Logged in to FreeIPA client machine as the AD user. - Typed passwd, and successfully reset my password. Verified the change in Windows as well as another IPA client. All Linux systems in this test are running CentOS 6.4 x86_64 FreeIPA server is running ipa-server-3.0.0-26.el6_4.4.x86_64 FreeIPA clients are running ipa-client-3.0.0-26.el6_4.4.x86_64 AD Server is running Windows 2008 R2 This won't necessarily help with the OS X problem, but maybe it assists with how it's working on Linux. Thanks, Brian On Tue, Aug 6, 2013 at 8:25 PM, Lynn Root wrote: > > On Aug 6, 2013, at 4:14 PM, KodaK wrote: > > > On Tue, Aug 6, 2013 at 4:31 PM, Davis Goodman > > wrote: > >> Hi, > >> > >> I have an FreeIPA server configured, managed to configure a Mountain > Lion Client for automounts and user logins. > >> > >> My issue is that whenever I first login with a user the "New Password" > box shows up and even if I try to change the password the box keeps > reappearing without any success. > >> > >> If I log onto the machine with the local admin user and try to get a > ticket for this user I get a "New Password" prompt. From there I can change > the password and I get a ticket without an issue. After that I can login > through the GUI without being asked for a new password. > >> > >> Anyone has seen this behaviour before? > > > > That's the expected behavior. When you set the user's password as an > > admin, it sets the "force a password change" flag. > > Correct me if I'm wrong, but it's not expect to *not* be able to change > the password on an IPA client after the initial setup, and be forced to use > the IPA Server to re-set the password. Granted, the client is OSX. > > However, I personally have experience the inability to change a new user's > password on an IPA client, and only on the IPA Server. Unfortunately, I've > been trying to reproduce this and I can not. I've tried on Fedora 19, and > will try on RHEL next. > > Davis - Can you let me know your IPA Server and IPA Client versions? As > well as the OS that the IPA Server is on? > > Also, out of curiosity, do you have directions on how you set up the > client on Mac OSX? > > Thanks! > > Lynn Root > > > > Lynn Root > @roguelynn > Associate Software Engineer > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davis.goodman at digital-district.ca Wed Aug 7 14:01:48 2013 From: davis.goodman at digital-district.ca (Davis Goodman) Date: Wed, 7 Aug 2013 10:01:48 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> Message-ID: Hi Brian, Lynn, As far as Linux client, this is not my issue for now, I believe the Linux setup is quite straight forward and the password change at first login seems to work without an issue. My main concern is on Mountain Lion 10.8.x, At this point I've managed to bind the OSX machine to the IPA server without any issue following this guide: http://linsec.ca/Using_FreeIPA_for_User_Authentication#Mac_OS_X_10.7.2F10.8 I also have all the autmounts configured via LDAP using this: https://ssl.apple.com/business/docs/Autofs.pdf on page 16. My main issue right now seems to be at the GUI login. The applet shows up for password change but doesn't seem to do anything. When I press continue the applet comes back and this goes in a loop until I hit "Cancel". My IPA versions are as follows: ipa-admintools.x86_64 3.0.0-26.el6_4.4 ipa-client.x86_64 3.0.0-26.el6_4.4 ipa-gothic-fonts.noarch 003.02-4.2.el6 ipa-mincho-fonts.noarch 003.02-3.1.el6 ipa-pgothic-fonts.noarch 003.02-4.1.el6 ipa-pmincho-fonts.noarch 003.02-3.1.el6 ipa-python.x86_64 3.0.0-26.el6_4.4 ipa-server.x86_64 3.0.0-26.el6_4.4 ipa-server-selinux.x86_64 3.0.0-26.el6_4.4 ipa-server-trust-ad.x86_64 3.0.0-26.el6_4.4 As mentioned in my first post, if I make the password change at the terminal prompt, I am then able to login without a password change prompt. Not sure if I'll be able to go through this issue unless someone as already experienced this. Davis -- Davis Goodman Directeur Informatique | IT Manager 5605 Avenue de Gasp?, Suite 408 | Montr?al, QC H2T 2A4 T?l: +1 (514) 360-3253 x104 Cell: +1 (514) 994-7360 On 2013-08-07, at 9:29 , Brian Lee wrote: > Hi Lynn, > > > I just checked this in my lab setup: > > - Set up a new user on the FreeIPA server as 'ipatest'. > > - Logged in to a Linux client configured for FreeIPA, it prompted me to change my password. > > - Successfully changed my password for ipatest. Verified this on another machine. > > - Furthermore, I reset the "Password Policy" min lifetime to 0 and typed passwd on one of the ipa clients while logged in as ipatest. This worked without issue. > > I also have FreeIPA set up in the lab with a domain trust to a 2008 R2 AD server, so I checked to see if the results would be the same. > > - Logged in to FreeIPA client machine as the AD user. > > - Typed passwd, and successfully reset my password. Verified the change in Windows as well as another IPA client. > > All Linux systems in this test are running CentOS 6.4 x86_64 > FreeIPA server is running ipa-server-3.0.0-26.el6_4.4.x86_64 > FreeIPA clients are running ipa-client-3.0.0-26.el6_4.4.x86_64 > AD Server is running Windows 2008 R2 > > This won't necessarily help with the OS X problem, but maybe it assists with how it's working on Linux. > > Thanks, > Brian > > > > On Tue, Aug 6, 2013 at 8:25 PM, Lynn Root wrote: > > On Aug 6, 2013, at 4:14 PM, KodaK wrote: > > > On Tue, Aug 6, 2013 at 4:31 PM, Davis Goodman > > wrote: > >> Hi, > >> > >> I have an FreeIPA server configured, managed to configure a Mountain Lion Client for automounts and user logins. > >> > >> My issue is that whenever I first login with a user the "New Password" box shows up and even if I try to change the password the box keeps reappearing without any success. > >> > >> If I log onto the machine with the local admin user and try to get a ticket for this user I get a "New Password" prompt. From there I can change the password and I get a ticket without an issue. After that I can login through the GUI without being asked for a new password. > >> > >> Anyone has seen this behaviour before? > > > > That's the expected behavior. When you set the user's password as an > > admin, it sets the "force a password change" flag. > > Correct me if I'm wrong, but it's not expect to *not* be able to change the password on an IPA client after the initial setup, and be forced to use the IPA Server to re-set the password. Granted, the client is OSX. > > However, I personally have experience the inability to change a new user's password on an IPA client, and only on the IPA Server. Unfortunately, I've been trying to reproduce this and I can not. I've tried on Fedora 19, and will try on RHEL next. > > Davis - Can you let me know your IPA Server and IPA Client versions? As well as the OS that the IPA Server is on? > > Also, out of curiosity, do you have directions on how you set up the client on Mac OSX? > > Thanks! > > Lynn Root > > > > Lynn Root > @roguelynn > Associate Software Engineer > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users From rcritten at redhat.com Wed Aug 7 14:07:05 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 07 Aug 2013 10:07:05 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> Message-ID: <52025489.9010805@redhat.com> Davis Goodman wrote: > Hi Brian, Lynn, > > As far as Linux client, this is not my issue for now, I believe the Linux setup is quite straight forward and the password change at first login seems to work without an issue. > > My main concern is on Mountain Lion 10.8.x, > > At this point I've managed to bind the OSX machine to the IPA server without any issue following this guide: > > http://linsec.ca/Using_FreeIPA_for_User_Authentication#Mac_OS_X_10.7.2F10.8 > > I also have all the autmounts configured via LDAP using this: https://ssl.apple.com/business/docs/Autofs.pdf on page 16. > > My main issue right now seems to be at the GUI login. The applet shows up for password change but doesn't seem to do anything. When I press continue the applet comes back and this goes in a loop until I hit "Cancel". > > My IPA versions are as follows: > ipa-admintools.x86_64 3.0.0-26.el6_4.4 > ipa-client.x86_64 3.0.0-26.el6_4.4 > ipa-gothic-fonts.noarch 003.02-4.2.el6 > ipa-mincho-fonts.noarch 003.02-3.1.el6 > ipa-pgothic-fonts.noarch 003.02-4.1.el6 > ipa-pmincho-fonts.noarch 003.02-3.1.el6 > ipa-python.x86_64 3.0.0-26.el6_4.4 > ipa-server.x86_64 3.0.0-26.el6_4.4 > ipa-server-selinux.x86_64 3.0.0-26.el6_4.4 > ipa-server-trust-ad.x86_64 3.0.0-26.el6_4.4 > > As mentioned in my first post, if I make the password change at the terminal prompt, I am then able to login without a password change prompt. > > Not sure if I'll be able to go through this issue unless someone as already experienced this. > > Davis What browser are you using? Have you tried the GUI with a new user from a Linux client? I'm thinking this is a browser issue rather than something with OSX as the majority of the work is done on the server. rob From davis.goodman at digital-district.ca Wed Aug 7 14:27:41 2013 From: davis.goodman at digital-district.ca (Davis Goodman) Date: Wed, 7 Aug 2013 10:27:41 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: <52025489.9010805@redhat.com> References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> <52025489.9010805@redhat.com> Message-ID: <213C6A6D-F238-4A56-A3F3-21D8620AB92C@digital-district.ca> When I mention GUI I'm talking about the Mac OSX Login screen not through a browser -- Davis Goodman Directeur Informatique | IT Manager 5605 Avenue de Gasp?, Suite 408 | Montr?al, QC H2T 2A4 T?l: +1 (514) 360-3253 x104 Cell: +1 (514) 994-7360 On 2013-08-07, at 10:07 , Rob Crittenden wrote: > Davis Goodman wrote: >> Hi Brian, Lynn, >> >> As far as Linux client, this is not my issue for now, I believe the Linux setup is quite straight forward and the password change at first login seems to work without an issue. >> >> My main concern is on Mountain Lion 10.8.x, >> >> At this point I've managed to bind the OSX machine to the IPA server without any issue following this guide: >> >> http://linsec.ca/Using_FreeIPA_for_User_Authentication#Mac_OS_X_10.7.2F10.8 >> >> I also have all the autmounts configured via LDAP using this: https://ssl.apple.com/business/docs/Autofs.pdf on page 16. >> >> My main issue right now seems to be at the GUI login. The applet shows up for password change but doesn't seem to do anything. When I press continue the applet comes back and this goes in a loop until I hit "Cancel". >> >> My IPA versions are as follows: >> ipa-admintools.x86_64 3.0.0-26.el6_4.4 >> ipa-client.x86_64 3.0.0-26.el6_4.4 >> ipa-gothic-fonts.noarch 003.02-4.2.el6 >> ipa-mincho-fonts.noarch 003.02-3.1.el6 >> ipa-pgothic-fonts.noarch 003.02-4.1.el6 >> ipa-pmincho-fonts.noarch 003.02-3.1.el6 >> ipa-python.x86_64 3.0.0-26.el6_4.4 >> ipa-server.x86_64 3.0.0-26.el6_4.4 >> ipa-server-selinux.x86_64 3.0.0-26.el6_4.4 >> ipa-server-trust-ad.x86_64 3.0.0-26.el6_4.4 >> >> As mentioned in my first post, if I make the password change at the terminal prompt, I am then able to login without a password change prompt. >> >> Not sure if I'll be able to go through this issue unless someone as already experienced this. >> >> Davis > > What browser are you using? > > Have you tried the GUI with a new user from a Linux client? > > I'm thinking this is a browser issue rather than something with OSX as the majority of the work is done on the server. > > rob > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amessina at messinet.com Wed Aug 7 14:28:07 2013 From: amessina at messinet.com (Anthony Messina) Date: Wed, 07 Aug 2013 09:28:07 -0500 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <5200EFC9.6070809@redhat.com> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> <32122_1375791763_5200EA92_32122_11510_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782D3@THSONEA01CMS03P.one.grp> <5200EFC9.6070809@redhat.com> Message-ID: <2311429.oxqREpoW9R@linux-ws1.messinet.com> On Tuesday, August 06, 2013 02:44:57 PM Martin Kosek wrote: > I see there are some SELinux issues for accessing /tmp/hsperfdata_root, they > look strange. I was running into the same SELinux issue when installing two FreeIPA servers in virtual machines yesterday: SELinux is preventing /usr/lib/jvm/java-1.7.0- openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java from read access on the directory hsperfdata_root. For me, the problem was two-fold: 1. When creating a new VM, I typically issue 'systemctl mask tmp.mount' and reboot as a first rule, since I don't have the availability to have a huge chunk of the VM's allocated RAM used up for /tmp. 2. Beccause of 1., the /tmp directory persists across reboots, and after initial FreeIPA installation, the /tmp/hsperfdata_root diretctory and files have the system_u:object_r:rpm_script_tmp_t:s0 SELinux label, when they should have system_u:object_r:pki_tomcat_tmp_t:s0. I resolved this issue by stopping IPA, removing /tmp/hsperfdata_root, and rebooting the machine, where I was able to observe the directory and files created with the proper context. Without knowing the proper context beforehand, there was no way to issue a restorecon, since there is no default label for /tmp/hsperfdata_root. -A -- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From klarmstrong2 at liberty.edu Wed Aug 7 18:46:48 2013 From: klarmstrong2 at liberty.edu (Armstrong, Kenneth Lawrence) Date: Wed, 7 Aug 2013 18:46:48 +0000 Subject: [Freeipa-users] AD user log in Message-ID: I have a test environment set up where we have a trust between the IdM domain and the AD domain. When we go to log into an IdM client with an AD user, we have to use the format of: ADDOMAIN\\username at idm.client.example.com Is there a way to prepend the domain part so that we won't have to type that in every time? Thanks! -Kenny -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Wed Aug 7 19:01:06 2013 From: jhrozek at redhat.com (Jakub Hrozek) Date: Wed, 7 Aug 2013 21:01:06 +0200 Subject: [Freeipa-users] AD user log in In-Reply-To: References: Message-ID: <20130807190106.GN26424@hendrix.redhat.com> On Wed, Aug 07, 2013 at 06:46:48PM +0000, Armstrong, Kenneth Lawrence wrote: > I have a test environment set up where we have a trust between the IdM domain and the AD domain. When we go to log into an IdM client with an AD user, we have to use the format of: > > ADDOMAIN\\username at idm.client.example.com > > Is there a way to prepend the domain part so that we won't have to type that in every time? > > Thanks! > > -Kenny Hi Kenny, I think that you're looking for the "default_domain_suffix" parameter. >From man sssd.conf: default_domain_suffix (string) This string will be used as a default domain name for all names without a domain name component. The main use case is environments where the primary domain is intended for managing host policies and all users are located in a trusted domain. The option allows those users to log in just with their user name without giving a domain name as well. Please note that if this option is set all users from the primary domain have to use their fully qualified name, e.g. user at domain.name, to log in. Default: not set The parameter should be set in the [sssd] section, not in the domain section. From dpal at redhat.com Wed Aug 7 19:41:43 2013 From: dpal at redhat.com (Dmitri Pal) Date: Wed, 07 Aug 2013 15:41:43 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: <213C6A6D-F238-4A56-A3F3-21D8620AB92C@digital-district.ca> References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> <52025489.9010805@redhat.com> <213C6A6D-F238-4A56-A3F3-21D8620AB92C@digital-district.ca> Message-ID: <5202A2F7.6030708@redhat.com> On 08/07/2013 10:27 AM, Davis Goodman wrote: > When I mention GUI I'm talking about the Mac OSX Login screen not > through a browser > > > -- > > > Davis Goodman > Directeur Informatique | IT Manager > > Digital-District > > 5605 Avenue de Gasp?, Suite 408 | Montr?al, QC H2T 2A4 > T?l: +1 (514) 360-3253 x104 Cell: +1 (514) 994-7360 > > > On 2013-08-07, at 10:07 , Rob Crittenden > wrote: > >> Davis Goodman wrote: >>> Hi Brian, Lynn, >>> >>> As far as Linux client, this is not my issue for now, I believe the >>> Linux setup is quite straight forward and the password change at >>> first login seems to work without an issue. >>> >>> My main concern is on Mountain Lion 10.8.x, >>> >>> At this point I've managed to bind the OSX machine to the IPA server >>> without any issue following this guide: >>> >>> http://linsec.ca/Using_FreeIPA_for_User_Authentication#Mac_OS_X_10.7.2F10.8 >>> >>> I also have all the autmounts configured via LDAP using this: >>> https://ssl.apple.com/business/docs/Autofs.pdf on page 16. >>> >>> My main issue right now seems to be at the GUI login. The applet >>> shows up for password change but doesn't seem to do anything. When I >>> press continue the applet comes back and this goes in a loop until I >>> hit "Cancel". >>> >>> My IPA versions are as follows: >>> ipa-admintools.x86_64 3.0.0-26.el6_4.4 >>> ipa-client.x86_64 3.0.0-26.el6_4.4 >>> ipa-gothic-fonts.noarch 003.02-4.2.el6 >>> ipa-mincho-fonts.noarch 003.02-3.1.el6 >>> ipa-pgothic-fonts.noarch 003.02-4.1.el6 >>> ipa-pmincho-fonts.noarch 003.02-3.1.el6 >>> ipa-python.x86_64 3.0.0-26.el6_4.4 >>> ipa-server.x86_64 3.0.0-26.el6_4.4 >>> ipa-server-selinux.x86_64 3.0.0-26.el6_4.4 >>> ipa-server-trust-ad.x86_64 3.0.0-26.el6_4.4 >>> >>> As mentioned in my first post, if I make the password change at the >>> terminal prompt, I am then able to login without a password change >>> prompt. >>> >>> Not sure if I'll be able to go through this issue unless someone as >>> already experienced this. >>> >>> Davis >> >> What browser are you using? >> >> Have you tried the GUI with a new user from a Linux client? >> >> I'm thinking this is a browser issue rather than something with OSX >> as the majority of the work is done on the server. >> >> rob >> > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users Not an expert on OSX. I wonder whether the UI prompt supports password change workflow. May be it does but needs to be explicitly enabled? There should be some logs on the OSX that would indicate what is going on when the server responds with the password change prompt. I would suggest starting troubleshooting efforts there. -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From davis.goodman at digital-district.ca Wed Aug 7 21:33:55 2013 From: davis.goodman at digital-district.ca (Davis Goodman) Date: Wed, 7 Aug 2013 17:33:55 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: <5202A2F7.6030708@redhat.com> References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> <52025489.9010805@redhat.com> <213C6A6D-F238-4A56-A3F3-21D8620AB92C@digital-district.ca> <5202A2F7.6030708@redhat.com> Message-ID: <8430B498-9727-413E-89CD-9123217313DC@digital-district.ca> This is basically the log when I attempt to change the password: Aug 7 16:59:19 mactestvm.mtl.dd.net SecurityAgent[271]: *** WARNING: -[NSImage compositeToPoint:operation:fraction:] is deprecated in MacOSX 10.8 and later. Please use -[NSImage drawAtPoint:fromRect:operation:fraction:] instead. Aug 7 16:59:19 mactestvm.mtl.dd.net SecurityAgent[271]: *** WARNING: -[NSImage compositeToPoint:fromRect:operation:fraction:] is deprecated in MacOSX 10.8 and later. Please use -[NSImage drawAtPoint:fromRect:operation:fraction:] instead. Aug 7 16:59:26 mactestvm.mtl.dd.net SecurityAgent[271]: User info context values set for testuser2 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got user: testuser2 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got ruser: (null) Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got service: authorization Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Context initialised Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Stashing kcm credentials in enviroment for kcminit: testuser2 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got user: testuser2 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got ruser: (null) Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got service: authorization Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Context initialised Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Created principal: testuser2 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done krb5_parse_name() Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got principal: testuser2 at DD.NET Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got password Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done getpwnam() Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Attempting to get forwardable TGT. Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: krb5_sendto_context is called on main thread, its a blocking api Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Attempting to get non-forwardable TGT. Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Kerberos 5 error Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Error krb5_get_init_creds_password(): Password has expired Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done cleanup2 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done cleanup3 Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Kerberos 5 refuses you Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): pam_sm_authenticate: ntlm Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): OpenDirectory - The authtok is expired or requires updating. Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_acct_mgmt(): OpenDirectory - Membership cache TTL set to 1800. Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_acct_mgmt(): OpenDirectory - Password expired. Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: Failed to authenticate user (error: 10). Aug 7 16:59:43 mactestvm.mtl.dd.net WindowServer[97]: 3891612: App SecurityAgent cannot order in untagged windows before login. Aug 7 16:59:43 mactestvm.mtl.dd.net SecurityAgent[271]: CGSOrderWindowList Does this rings a bell? -- Davis Goodman Directeur Informatique | IT Manager 5605 Avenue de Gasp?, Suite 408 | Montr?al, QC H2T 2A4 T?l: +1 (514) 360-3253 x104 Cell: +1 (514) 994-7360 On 2013-08-07, at 15:41 , Dmitri Pal wrote: > On 08/07/2013 10:27 AM, Davis Goodman wrote: >> When I mention GUI I'm talking about the Mac OSX Login screen not through a browser >> >> >> -- >> >> >> Davis Goodman >> Directeur Informatique | IT Manager >> >> 5605 Avenue de Gasp?, Suite 408 | Montr?al, QC H2T 2A4 >> T?l: +1 (514) 360-3253 x104 Cell: +1 (514) 994-7360 >> >> >> On 2013-08-07, at 10:07 , Rob Crittenden wrote: >> >>> Davis Goodman wrote: >>>> Hi Brian, Lynn, >>>> >>>> As far as Linux client, this is not my issue for now, I believe the Linux setup is quite straight forward and the password change at first login seems to work without an issue. >>>> >>>> My main concern is on Mountain Lion 10.8.x, >>>> >>>> At this point I've managed to bind the OSX machine to the IPA server without any issue following this guide: >>>> >>>> http://linsec.ca/Using_FreeIPA_for_User_Authentication#Mac_OS_X_10.7.2F10.8 >>>> >>>> I also have all the autmounts configured via LDAP using this: https://ssl.apple.com/business/docs/Autofs.pdf on page 16. >>>> >>>> My main issue right now seems to be at the GUI login. The applet shows up for password change but doesn't seem to do anything. When I press continue the applet comes back and this goes in a loop until I hit "Cancel". >>>> >>>> My IPA versions are as follows: >>>> ipa-admintools.x86_64 3.0.0-26.el6_4.4 >>>> ipa-client.x86_64 3.0.0-26.el6_4.4 >>>> ipa-gothic-fonts.noarch 003.02-4.2.el6 >>>> ipa-mincho-fonts.noarch 003.02-3.1.el6 >>>> ipa-pgothic-fonts.noarch 003.02-4.1.el6 >>>> ipa-pmincho-fonts.noarch 003.02-3.1.el6 >>>> ipa-python.x86_64 3.0.0-26.el6_4.4 >>>> ipa-server.x86_64 3.0.0-26.el6_4.4 >>>> ipa-server-selinux.x86_64 3.0.0-26.el6_4.4 >>>> ipa-server-trust-ad.x86_64 3.0.0-26.el6_4.4 >>>> >>>> As mentioned in my first post, if I make the password change at the terminal prompt, I am then able to login without a password change prompt. >>>> >>>> Not sure if I'll be able to go through this issue unless someone as already experienced this. >>>> >>>> Davis >>> >>> What browser are you using? >>> >>> Have you tried the GUI with a new user from a Linux client? >>> >>> I'm thinking this is a browser issue rather than something with OSX as the majority of the work is done on the server. >>> >>> rob >>> >> >> >> >> _______________________________________________ >> Freeipa-users mailing list >> >> Freeipa-users at redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-users > > Not an expert on OSX. > I wonder whether the UI prompt supports password change workflow. May be it does but needs to be explicitly enabled? > There should be some logs on the OSX that would indicate what is going on when the server responds with the password change prompt. > I would suggest starting troubleshooting efforts there. > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager for IdM portfolio > Red Hat Inc. > > > ------------------------------- > Looking to carve out IT costs? > > www.redhat.com/carveoutcosts/ > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users From rcritten at redhat.com Wed Aug 7 21:38:22 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 07 Aug 2013 17:38:22 -0400 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <2311429.oxqREpoW9R@linux-ws1.messinet.com> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> <32122_1375791763_5200EA92_32122_11510_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C5782D3@THSONEA01CMS03P.one.grp> <5200EFC9.6070809@redhat.com> <2311429.oxqREpoW9R@linux-ws1.messinet.com> Message-ID: <5202BE4E.9090702@redhat.com> Anthony Messina wrote: > On Tuesday, August 06, 2013 02:44:57 PM Martin Kosek wrote: >> I see there are some SELinux issues for accessing /tmp/hsperfdata_root, they >> look strange. > > I was running into the same SELinux issue when installing two FreeIPA servers > in virtual machines yesterday: > > SELinux is preventing /usr/lib/jvm/java-1.7.0- > openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java from read access on the > directory hsperfdata_root. > > For me, the problem was two-fold: > 1. When creating a new VM, I typically issue 'systemctl mask tmp.mount' and > reboot as a first rule, since I don't have the availability to have a huge > chunk of the VM's allocated RAM used up for /tmp. > > 2. Beccause of 1., the /tmp directory persists across reboots, and after > initial FreeIPA installation, the /tmp/hsperfdata_root diretctory and files > have the system_u:object_r:rpm_script_tmp_t:s0 SELinux label, when they should > have system_u:object_r:pki_tomcat_tmp_t:s0. > > I resolved this issue by stopping IPA, removing /tmp/hsperfdata_root, and > rebooting the machine, where I was able to observe the directory and files > created with the proper context. > > Without knowing the proper context beforehand, there was no way to issue a > restorecon, since there is no default label for /tmp/hsperfdata_root. > There is a bug open against selinux-policy on this from F-18 using a standard configuration, https://bugzilla.redhat.com/show_bug.cgi?id=917843 You may want to either add your own use-case here, or open a new bug and reference this one. rob From amessina at messinet.com Wed Aug 7 21:54:50 2013 From: amessina at messinet.com (Anthony Messina) Date: Wed, 07 Aug 2013 16:54:50 -0500 Subject: [Freeipa-users] Install error pkispawn In-Reply-To: <5202BE4E.9090702@redhat.com> References: <32122_1375778883_5200B842_32122_5677_1_EEF700FCD1A35041BF2C750B4EDDCA3101392C57820F@THSONEA01CMS03P.one.grp> <2311429.oxqREpoW9R@linux-ws1.messinet.com> <5202BE4E.9090702@redhat.com> Message-ID: <1944873.MeD6lChBDM@linux-ws1.messinet.com> On Wednesday, August 07, 2013 05:38:22 PM Rob Crittenden wrote: > Anthony Messina wrote: > > On Tuesday, August 06, 2013 02:44:57 PM Martin Kosek wrote: > >> I see there are some SELinux issues for accessing /tmp/hsperfdata_root, > >> they look strange. > > > > I was running into the same SELinux issue when installing two FreeIPA > > servers in virtual machines yesterday: > > > > SELinux is preventing /usr/lib/jvm/java-1.7.0- > > openjdk-1.7.0.25-2.3.12.3.fc19.x86_64/jre/bin/java from read access on the > > directory hsperfdata_root. > > > > For me, the problem was two-fold: > > 1. When creating a new VM, I typically issue 'systemctl mask tmp.mount' > > and > > reboot as a first rule, since I don't have the availability to have a huge > > chunk of the VM's allocated RAM used up for /tmp. > > > > 2. Beccause of 1., the /tmp directory persists across reboots, and after > > initial FreeIPA installation, the /tmp/hsperfdata_root diretctory and > > files > > have the system_u:object_r:rpm_script_tmp_t:s0 SELinux label, when they > > should have system_u:object_r:pki_tomcat_tmp_t:s0. > > > > I resolved this issue by stopping IPA, removing /tmp/hsperfdata_root, and > > rebooting the machine, where I was able to observe the directory and files > > created with the proper context. > > > > Without knowing the proper context beforehand, there was no way to issue a > > restorecon, since there is no default label for /tmp/hsperfdata_root. > > There is a bug open against selinux-policy on this from F-18 using a > standard configuration, > https://bugzilla.redhat.com/show_bug.cgi?id=917843 > > You may want to either add your own use-case here, or open a new bug and > reference this one. Thanks, Rob. I've added this information there. -A -- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From dpal at redhat.com Wed Aug 7 22:42:57 2013 From: dpal at redhat.com (Dmitri Pal) Date: Wed, 07 Aug 2013 18:42:57 -0400 Subject: [Freeipa-users] Mountain Lion GUI Login In-Reply-To: <8430B498-9727-413E-89CD-9123217313DC@digital-district.ca> References: <1F3C2E36-23C7-4B02-A712-BA4CC621BA5A@digital-district.ca> <52025489.9010805@redhat.com> <213C6A6D-F238-4A56-A3F3-21D8620AB92C@digital-district.ca> <5202A2F7.6030708@redhat.com> <8430B498-9727-413E-89CD-9123217313DC@digital-district.ca> Message-ID: <5202CD71.2080804@redhat.com> On 08/07/2013 05:33 PM, Davis Goodman wrote: > This is basically the log when I attempt to change the password: > > Aug 7 16:59:19 mactestvm.mtl.dd.net SecurityAgent[271]: *** WARNING: -[NSImage compositeToPoint:operation:fraction:] is deprecated in MacOSX 10.8 and later. Please use -[NSImage drawAtPoint:fromRect:operation:fraction:] instead. > Aug 7 16:59:19 mactestvm.mtl.dd.net SecurityAgent[271]: *** WARNING: -[NSImage compositeToPoint:fromRect:operation:fraction:] is deprecated in MacOSX 10.8 and later. Please use -[NSImage drawAtPoint:fromRect:operation:fraction:] instead. > Aug 7 16:59:26 mactestvm.mtl.dd.net SecurityAgent[271]: User info context values set for testuser2 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got user: testuser2 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got ruser: (null) > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got service: authorization > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Context initialised > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Stashing kcm credentials in enviroment for kcminit: testuser2 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got user: testuser2 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got ruser: (null) > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got service: authorization > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Context initialised > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Created principal: testuser2 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done krb5_parse_name() > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got principal: testuser2 at DD.NET > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Got password > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done getpwnam() > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Attempting to get forwardable TGT. > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: krb5_sendto_context is called on main thread, its a blocking api > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Attempting to get non-forwardable TGT. > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Kerberos 5 error > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Error krb5_get_init_creds_password(): Password has expired > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done cleanup2 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Done cleanup3 > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): Kerberos 5 refuses you This is where it should behave differently. It should treat this not as a failure but prompt for password change when such error is returned. I would check OSX forums on how to enable password change in UI > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): pam_sm_authenticate: ntlm > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_authenticate(): OpenDirectory - The authtok is expired or requires updating. > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_acct_mgmt(): OpenDirectory - Membership cache TTL set to 1800. > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: in pam_sm_acct_mgmt(): OpenDirectory - Password expired. > Aug 7 16:59:26 mactestvm.mtl.dd.net authorizationhost[283]: Failed to authenticate user (error: 10). > Aug 7 16:59:43 mactestvm.mtl.dd.net WindowServer[97]: 3891612: App SecurityAgent cannot order in untagged windows before login. > Aug 7 16:59:43 mactestvm.mtl.dd.net SecurityAgent[271]: CGSOrderWindowList > > Does this rings a bell? > > -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From mkosek at redhat.com Thu Aug 8 07:22:44 2013 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 08 Aug 2013 09:22:44 +0200 Subject: [Freeipa-users] Planning FreeIPA Upstream Doc changes In-Reply-To: <52034679.2030306@redhat.com> References: <52034679.2030306@redhat.com> Message-ID: <52034744.8000505@redhat.com> FYI. I originally posted the planing to the freeipa-devel list as it affects mostly developers, but it may be interesting for some of our users too. Martin -------- Original Message -------- Subject: [Freeipa-devel] Planning FreeIPA Upstream Doc changes Date: Thu, 08 Aug 2013 09:19:21 +0200 From: Martin Kosek To: freeipa-devel at redhat.com Hello all, This is a follow up for upstream doc maintenance questions I had on freeipa-users in June: http://www.redhat.com/archives/freeipa-users/2013-June/msg00202.html As Content Writer taking care of the User Guide (on docs.fedoraproject.org) no longer has resources to maintain it and the guide become partially outdated. FreeIPA development team and community will need to take over as it was agreed this is of a great benefit to the project. I would like to propose a plan to revive the guide: 1) Move FreeIPA Guide out of current Deon's git tree (hosted on https://fedorahosted.org/freeipa-guide/) to git owned by FreeIPA. There are 2 options: A) Host and package it in current FreeIPA git along with a code. Handling the doc patches would be much simpler, however it would mean, that the docs for next version would need to be prepared at the time when code is ready for release which could either postpone the release and release with incomplete doc (this introduces question how should the git be tagged/branched). B) Add a new git tree to FreeIPA fedorahosted account (freeipa/docs.git) and release the docs asynchronously to the code (there could be of course a preliminary version on FreeIPA.org site). I was already thinking about options to seamlessly integrate an RPM with docs to FreeIPA Web UI which could then provide relevant help links from the UI dialogs to the right spots of documentation. - so far, it seems that option B) will get us more flexibility and would avoid people contributing to the code only downloading also the doc. WHEN: this step would be right after the plan is acknowledged 2) Update the guide to match FreeIPA 3.3 - currently, the guide is approximately on FreeIPA 3.1 level - I filed Trac tickets for gaps I identified in the guide: https://fedorahosted.org/freeipa/milestone/Revive%20FreeIPA%20guide - I would keep the guide in Docbook format unless there are strong reasons to use other format and avoid loosing information. It is very easy to generate all sorts of formats (html, pdf, epub) out of Docbook with publican utility which is packaged in Fedora -> easy build in koji - ideally, editorial would be done by a potential contractor we identified WHEN: most should be done in August, some can be done in September 3) Host the result on FreeIPA.org - we used to release to Fedora documentation portal, but I am thinking it would be more natural to host a project site on project portal instead of Fedora Documentation which rather holds general Fedora infrastructure books. It may be also more intuitive for users to find it on FreeIPA.org than on Fedora docs. - we can take advantage of publican and build a doc site like "docs.freeipa.org" which would held publican-managed doc site with our guides. I tried to play with publican and this is a first PoC result: http://mkosek.fedorapeople.org/publican_site/ - I would prefer if our generated user guides (including current Extending guide hosted on abbra.fedorapeople.org site) use Docbook + be integrated in the site + wiki to have them all under the same roof with consistent look WHEN: Before FreeIPA 3.4 is released 4) Start maintaining and releasing User guide with FreeIPA releases - revive "Affects doc" Trac ticket flag and make sure that Doc is updated along with RFEs and bugfixes - the process of tracking and doing the doc updates and changes is essential in order to keep the doc updated and useful for our users - ideally, editorial would be done by a contractor WHEN: Before 3.4 is release Any ideas or comments very welcome. Thanks, Martin -- Martin Kosek Supervisor, Software Engineering - Identity Management Team Red Hat Inc. _______________________________________________ Freeipa-devel mailing list Freeipa-devel at redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel From mkosek at redhat.com Thu Aug 8 14:03:16 2013 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 08 Aug 2013 16:03:16 +0200 Subject: [Freeipa-users] Announcing FreeIPA 3.3.0 Message-ID: <5203A524.7060808@redhat.com> The FreeIPA team is proud to announce FreeIPA v3.3.0! It can be downloaded from http://www.freeipa.org/page/Downloads. Fedora 19 builds are already on their way to updates-testing repo. == Highlights in 3.3.0 == === New features for 3.3 === * Active Directory integration: ** Support of externally defined POSIX attributes for Active Directory trusted domains ** Automatic discovery of Active Directory identity mapping configuration ** Support of trusted domain users for legacy clients ** Identity mapping for AD users can now be delegated * Performance improvements in processing large number of users and groups * Automated integration testing infrastructure * ipa-advise utility is added to generate client setup advice based on an IPA master configuration * FreeIPA-specific SELinux policies has been merged to the main SELinux policy in Fedora 19 * SSSD 1.11 is required === Active Directory integration === Starting with FreeIPA 3.3, it is possible to define identity ranges for a trusted Active Directory domain that rely on POSIX attributes provided by AD DC instead of generating them out of corresponding security identifiers. This functionality requires Services for Unix (SFU) or Identity Management for UNIX enabled on Active Directory side and is provided mostly to aid with migration to SID-based mapping. In order to support externally defined POSIX attributes, identity ranges have been extended to support new range types: * AD trust with SID-based mapping: 'ipa-ad-trust' (default) * SFU support: 'ipa-ad-trust-posix' 'ipa-ad-trust-posix' range type is activated when range discovery finds out SFU is in use by Active Directory domain. To override automatic detection, --range-type=ipa-ad-trust can be specified to 'ipa trust-add' command. FreeIPA 3.3 requires SSSD 1.11 on the IPA master in order to support externally defined POSIX attributes in AD. More details: http://www.freeipa.org/page/V3/Use_posix_attributes_defined_in_AD FreeIPA 3.3 provides a new way to enable legacy clients to support trusted domain users. A compatibility tree, provided by slapi-nis, can now be configured to look up trusted domain users and handle authentication for them. This functionality relies on SSSD 1.11 and release 0.47.7 of slapi-nis. One can enable legacy clients support by running ipa-adtrust-install and answering positively to the corresponding question. More details: http://www.freeipa.org/page/V3/Serving_legacy_clients_for_trusts Finally, SSSD 1.11 is used to query identity information about trusted domains' users from within IPA framework, including SID to name and name to SID resolution. In addition to speed improvements, FreeIPA 3.3 allows to manage mappings for trusted domains' users without requiring elevated privileges of 'trust admins'. === Performance improvements === When acting on large datasets, FreeIPA now reduces number of potential read roundtrips required to update user and group information. When scaled to thousands of users and groups, this shortens the time required by certain operations tenfold. === Automated testing infrastructure === The FreeIPA team has been providing self-testing code for a long time. The FreeIPA 3.3 test suite includes a framework for integration tests that verify functionality such as replication across several machines. Tests can be run manually, or by test automation servers such as Jenkins or Beaker. Development builds now create a freeipa-tests RPM containing the test suite and related tools. However, as the focus is on testing development code, this package will not be released to Fedora yet. More details: http://www.freeipa.org/page/V3/Integration_testing Additionally, it is now possible to run Web UI tests through the test suite. More details: http://www.freeipa.org/page/Web_UI_Integration_Tests === IPA advise tool === FreeIPA 3.3 introduces new framework to generate recipes of configuration based on how IPA master is configured. These recipes can be taken to the target client systems and used there to configure them for a specific task. We expect to expand use of 'ipa-advise' tool to cover at least configuration of legacy systems in subsequent releases. Three advices are provided with FreeIPA 3.3.0 release: * configuring a generic Fedora release with authconfig tool * configuring RHEL-based systems with SSSD 1.5-1.8 * configuring Debian-based systems with SSSD 1.5-1.8 Contributions are always welcome to grow capabilities of 'ipa-advise' tool to other areas. More details: http://www.freeipa.org/page/V3/Serving_legacy_clients_for_trusts#Major_configuration_options_and_enablement === SELinux policy === SELinux policies specific to FreeIPA have been merged back to the main SELinux policy package in Fedora 19. Starting with FreeIPA 3.2.2 (available in Fedora 19 updates) SELinux policy is no londer provided by freeipa-selinux package and the package is removed in favor of selinux-policy package. === SSSD 1.11 is required === FreeIPA 3.3 depends on SSSD 1.11 for cross-realm trusts with Active Directory. In particular, FreeIPA 3.3 depends on a new operational mode of SSSD called 'ipa_server_mode'. Thus, SSSD 1.11 is required for FreeIPA 3.3. More details: https://fedorahosted.org/sssd/wiki/DesignDocs/IPAServerMode == Upgrading == === FreeIPA servers with CA installed prior to version 3.1 === Manual upgrade procedure is required for FreeIPA servers installed with version prior to 3.1. === Other FreeIPA servers and clients === An IPA server can be upgraded simply by installing updated rpms. The server does not need to be shut down in advance. Please note that if you are doing the upgrade in special environment (e.g. FedUp) which does not allow running LDAP server during upgrade process, upgrade scripts need to be run manually after the first boot: # ipa-upgradeconfig # ipa-ldap-updater --upgrade Please note, that the performance improvements requires an extended set of indexes to be configured. RPM update for an IPA server with a excessive number of users may require several minutes to finish. If you have multiple servers you may upgrade them one at a time. It is expected that all servers will be upgraded in a relatively short period (days or weeks not months). They should be able to co-exist peacefully but new features will not be available on old servers and enrolling a new client against an old server will result in the SSH keys not being uploaded. Downgrading a server once upgraded is not supported. Upgrading from 2.2.0 and later versions is supported. Upgrading from previous versions is not supported and has not been tested. An enrolled client does not need the new packages installed unless you want to re-enroll it. SSH keys for already installed clients are not uploaded, you will have to re-enroll the client or manually upload the keys. == Feedback == Please provide comments, bugs and other feedback via the freeipa-users mailing list (http://www.redhat.com/mailman/listinfo/freeipa-users) or #freeipa channel on Freenode. == Detailed Changelog since 3.2.0 == === Alexander Bokovoy (9): === * Fix cldap parser to work with a single equality filter (NtVer=...) * Make sure domain_name is also set when processing INP_NAME requests * Fix extdom plugin to provide unqualified name in response as sssd expects * Generate syntethic MS-PAC for all services running on IPA master * ipa-adtrust-install: configure compatibility tree to serve trusted domain users * ipa-kdb: cache KDC hostname on startup * ipa-kdb: reinit mspac on HTTP TGT acquisition to aid trust-add case * ipaserver/dcerpc: attempt to resolve SIDs through SSSD first * Rename slapi-nis configuration variable === Ana Krivokapic (26): === * Prompt for nameserver IP address in dnszone-add * Do not display success message on failure in web UI * Ignore files generated by build * Deprecate options --dom-sid and --dom-name in idrange-mod * Prevent error when running IPA commands with su/sudo * Fix displaying of success message * Fix location of service.crt in .gitignore * Improve handling of options in ipa-client-install * Fail when adding a trust with a different range * Do not display traceback to user * Require rid-base and secondary-rid-base in idrange-add after ipa-adtrust-install * Fix bug in adtrustinstance * Use correct DS instance in ipactl status * Avoid systemd service deadlock during shutdown * Make sure replication works after DM password is changed * Use --ignore-dependencies only when necessary * Properly handle non-existent cert files * Add 'ipa_server_mode' option to SSSD configuration * Bump version of sssd in spec file * Use admin at REALM when testing if SSSD is ready * Fix internal error in idrange-add * Honor 'enabled' option for widgets. * Expose ipaRangeType in Web UI * Add ipa-advise plugins for legacy clients * Enable running API commands in ipa-advise plugins * Add new command compat-is-enabled === Diane Trout (1): === * Fix log format not a string literal. === Jakub Hrozek (3): === * Remove unused variable * IPA KDB MS-PAC: return ENOMEM if allocation fails * IPA KDB MS-PAC: remove unused variable === Jan Cholasta (21): === * Use the correct PKCS#12 file for HTTP server. * Remove stray error condition in ipa-server-install. * Handle exceptions gracefully when verifying PKCS#12 files. * Skip empty lines when parsing pk12util output. * Do not allow installing CA replicas in CA-less setup. * Do not track DS certificate in CA-less setup. * Fix CA-less check in ipa-replica-install and ipa-ca-install. * Do not skip SSSD known hosts in ipa-client-install --ssh-trust-dns. * Enable SASL mapping fallback. * Skip cert issuer validation in service and host commands in CA-less install. * Check trust chain length in CA-less install. * Use LDAP search instead of *group_show to check if a group exists. * Use LDAP search instead of *group_show to check for a group objectclass. * Use LDAP modify operation directly to add/remove group members. * Add missing substring indices for attributes managed by the referint plugin. * Add missing equality index for ipaUniqueId. * Run gpg-agent explicitly when encrypting/decrypting files. * Add new hidden command option to suppress processing of membership attributes. * Ask for PKCS#12 password interactively in ipa-server-install. * Ask for PKCS#12 password interactively in ipa-replica-prepare. * Print newline after receiving EOF in installutils.read_password. === Lukas Slebodnik (4): === * Use pkg-config to detect cmocka * Use right function prototype for thread function * Remove unused variable * Remove unused variable === Martin Kosek (17): === * Set KRB5CCNAME so that dirsrv can work with newer krb5-server * Handle DIR type CCACHEs in test_cmdline properly * Avoid exporting KRB5_KTNAME in dirsrv env * Remove redundant u'' character * Drop SELinux subpackage * Drop redundant directory /var/cache/ipa/sessions * Remove entitlement support * Run server upgrade and restart in posttrans * Require new selinux-policy replacing old server-selinux subpackage * Bump minimum SSSD version * Become 3.3.0 Beta 1 * Free NSS objects in --external-ca scenario * Use valid LDAP search base in migration plugin * Increase default SASL buffer size * Become 3.3.0 Beta 2 * Add requires for slapi-nis and SSSD * Become 3.3.0 === Nathaniel McCallum (10): === * Add ipaUserAuthType and ipaUserAuthTypeClass * Add IPA OTP schema and ACLs * ipa-kdb: Add OTP support * Add the krb5/FreeIPA RADIUS companion daemon * Remove unnecessary prefixes from ipa-pwd-extop files * Add OTP support to ipa-pwd-extop * Fix client install exception if /etc/ssh is missing * Permit reads to ipatokenRadiusProxyUser objects * Fix for small syntax error in OTP schema * Use libunistring ulc_casecmp() on unicode strings === Petr Spacek (1): === * ipa-client-install: Add 'debug' and 'show' statements to nsupdate commands === Petr Viktorin (33): === * Remove leading zero from IPA_NUM_VERSION * Relax getkeytab test to allow additional messages on stderr * Remove code to install Dogtag 9 * Flush stream after writing service messages * Make an ipa-tests package * Add ipa-run-tests command * Add Nose plugin for BeakerLib integration * Add a plugin for test ordering * Add a framework for integration test configuration * Add a framework for integration testing * Introduce a class for remote commands * Collect logs from tests * Show logs in failed tests * tests: Allow public keys for authentication to the remote machines * tests: Configure/unconfigure remote hosts * Host class improvements * Use dosctrings in BeakerLib phase descriptions * Make BeakerLib logging less verbose * BeakerLib plugin: Log http links in test docstrings * Integration test config: Make it possible to specify host IP * ipa-client: Use "ipa" as the package name for i18n * Move BeakerLibProcess out of BeakerLibPlugin * test_integration: Add log collection to Host * test_integration: Set up CA on replicas by default * Add more test tasks * Add install_topo to test tasks * Add the ipa-test-task tool * Add tar and xz dependencies to the freeipa-tests package * Correct default value of LDAPClient.get_entries scope argument * test_simple_replication: Wait for replication to finish before checking * Add the new no_member option to CLI tests * Update translations * Fix installutils.get_password without a TTY === Petr Vobornik (24): === * Fix: HBAC Test tab is missing * Move spec modifications from facet factories to pre_ops * Unite and move facet pre_ops to related modules * Web UI: move ./_base/metadata_provider.js to ./metadata.js * Regression fix: missing control buttons in nested search facets * Make ssbrowser.html work in IE 10 * Fix regression: missing facet tab group labels * Regression fix: rule table with ext. member support doesn't offer any items * Fix default value selection in radio widget * Do not redirect to https in /ipa/ui on non-HTML files * Create Firefox configuration extension on CA-less install * Disable checkboxes and radios for readonly attributes * Better automated test support * Fix container element in adder dialogs * Upstream Web UI tests * Web UI search optimization * Break long words in notification area * Remove word 'field' from GECOS param label * Web UI integration tests: Add trust tests * Web UI integration tests: Add ui_driver method descriptions * Web UI integration tests: Verify data after add and mod * Web UI integration tests: Compute range sizes to avoid overlaps * Web UI integration tests: PEP8 fixes * Web UI integration tests: Code quality fixes === Rob Crittenden (4): === * Bump version for development branch to 3.2.99 * Return the correct Content-type on negotiated XML-RPC requests. * Add Camellia ciphers to allowed list. * Hide sensitive attributes in LDAP updater logging and output === Simo Sorce (2): === * CLDAP: Fix domain handling in netlogon requests * CLDAP: Return empty reply on non-fatal errors === Sumit Bose (5): === * Fix format string typo * Fix type of printf argument * Add PAC to master host TGTs * extdom: replace winbind calls with POSIX/SSSD calls * Remove winbind client configure check === Tomas Babej (32): === * Remove redundancy from hbactest help text * Do not translate trust type and direction with --raw in trust_show and trust-find * Support multiple local domain ranges with RID base set * Do not allow removal of ID range of an active trust * Use private ccache in ipa install tools * Remove redundant check for env.interactive * Add prompt_param method to avoid code duplication * Incorporate interactive prompts in idrange-add * Do not check userPassword with 7-bit plugin * Manage ipa-otpd.socket by IPA * Add ipaRangeType attribute to LDAP Schema * Add update plugin to fill in ipaRangeType attribute * Extend idrange commands to support new range origin types * PEP8 fixes in idrange.py * Remove hardcoded values from idrange plugin tests * Return ipaRangeType as a list in idrange commands * Do not redirect ipa/crl to HTTPS * Add --range-type option that forces range type of the trusted domain * Add libsss_nss_idmap-devel to BuildRequires * Change group ownership of CRL publish directory * Provide ipa-advise tool * Use AD LDAP probing to create trusted domain ID range * Move requirement for keyutils to freeipa-python package * Change shebang to absolute path in ipa-client-automount * Skip referrals when converting LDAP result to LDAPEntry * Refactor the interactive prompt logic in idrange_add * Limit pwpolicy maxlife to 20000 days * Use case-insensitive dict for trusted domain info * Improve help entry for ipa host * Remove overlapping use-cases of the same result variable * Add a word wrapping for comment log messages to AdviceLogger * Wrap lines in the list of available advices From klarmstrong2 at liberty.edu Fri Aug 9 13:29:18 2013 From: klarmstrong2 at liberty.edu (Armstrong, Kenneth Lawrence) Date: Fri, 9 Aug 2013 13:29:18 +0000 Subject: [Freeipa-users] tough one on DNS Message-ID: Hi all. We have IdM set up in a test environment as a subdomain of our Windows domain (so, linux.example.com) with integrated DNS on the IdM server and forwarders going to the AD servers on example.com. We have on the Windows DNS server the IdM server set up as a conditional forwarder for linux.example.com and an A record in its DNS for the IdM server. However, on example.com, we have other subdomains that need to be able to communicate with the linux.example.com domain, such as tier1.example.com, tier2.example.com, etc. What we are trying to figure out is the whole PTR reverse zone bit, mainly due to the fact that the linux.example.com and the example.com reside on the same subnets (and unfortunately, this is a very large network and we can't change that). For instance, say we have a system at test1.tier2.example.com that needs to make an SSL connection to another system on testA.linux.example.com. The SSL handshake will fail since the PTR record will resolve to a different domain than what the client is expecting, again due to the fact that the reverse zone is on the same subnet. The reverse zone on the Windows DNS server would only contain PTR records for all of the domains except linux.example.com, and the IdM server would only contain PTR records for just the linux.example.com systems. So, obviously, we would not want the IdM server to have a reverse zone, and have it rely on the reverse zone on the Windows DNS server. Other than manually entering PTR records for all Linux systems under linux.example.com to the reverse zone file on the Windows DNS server, is there another way that we can properly accomplish this? Would replicating the reverse zone file from the Windows server to the IdM server take care of that? Thanks. -Kenny -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Fri Aug 9 15:31:46 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Fri, 9 Aug 2013 11:31:46 -0400 Subject: [Freeipa-users] Can't update ssh keys Message-ID: Any time I try to use the command-line utilities to add a host (this includes ipa-client-install): #ipa host-mod host.damascusgrp.com--updatedns --sshpubkey="`cat /etc/ssh/ssh_host_rsa_key.pub`" ipa: ERROR: invliad 'sshpubkey': must be binary data I know I can use the GUI, but as we could be rolling out a large number of systems in coming months, that's not a good long-term option. So does anyone know a way to make the CLI tools work? Second question: is there a way to update the SSHFP records apart from using the CLI tools as above? Thanks! * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Aug 9 17:22:31 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 09 Aug 2013 13:22:31 -0400 Subject: [Freeipa-users] Can't update ssh keys In-Reply-To: References: Message-ID: <52052557.6070400@redhat.com> Bret Wortman wrote: > Any time I try to use the command-line utilities to add a host (this > includes ipa-client-install): > > #ipa host-mod host.damascusgrp.com > --updatedns > --sshpubkey="`cat /etc/ssh/ssh_host_rsa_key.pub`" > ipa: ERROR: invliad 'sshpubkey': must be binary data > > I know I can use the GUI, but as we could be rolling out a large number > of systems in coming months, that's not a good long-term option. So does > anyone know a way to make the CLI tools work? > > Second question: is there a way to update the SSHFP records apart from > using the CLI tools as above? A pub key consists of 3 pieces of data: the key type, the key and a comment. What version of IPA? IIRC in v2 only the key material itself was supported. This cli command should work with a v3 server which requires all 3 pieces. I imagine you could use dnsrecord-mod/add to manage the SSHFP record but that could lead to different values in the DNS and host entry if not done carefully. rob From bret.wortman at damascusgrp.com Fri Aug 9 19:46:28 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Fri, 09 Aug 2013 12:46:28 -0700 (PDT) Subject: [Freeipa-users] Can't update ssh keys In-Reply-To: <52052557.6070400@redhat.com> References: <52052557.6070400@redhat.com> Message-ID: <1376077587144.e0f2df44@Nodemailer> V3.1.something. I'm not at the office again till Monday. On Fri, Aug 9, 2013 at 1:22 PM, Rob Crittenden wrote: > Bret Wortman wrote: >> Any time I try to use the command-line utilities to add a host (this >> includes ipa-client-install): >> >> #ipa host-mod host.damascusgrp.com >> --updatedns >> --sshpubkey="`cat /etc/ssh/ssh_host_rsa_key.pub`" >> ipa: ERROR: invliad 'sshpubkey': must be binary data >> >> I know I can use the GUI, but as we could be rolling out a large number >> of systems in coming months, that's not a good long-term option. So does >> anyone know a way to make the CLI tools work? >> >> Second question: is there a way to update the SSHFP records apart from >> using the CLI tools as above? > A pub key consists of 3 pieces of data: the key type, the key and a comment. > What version of IPA? IIRC in v2 only the key material itself was > supported. This cli command should work with a v3 server which requires > all 3 pieces. > I imagine you could use dnsrecord-mod/add to manage the SSHFP record but > that could lead to different values in the DNS and host entry if not > done carefully. > rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From enewland at redhat.com Fri Aug 9 17:46:20 2013 From: enewland at redhat.com (Ellen Newlands) Date: Fri, 9 Aug 2013 13:46:20 -0400 Subject: [Freeipa-users] [Freeipa-interest] Announcing FreeIPA 3.3.0 In-Reply-To: <5203A524.7060808@redhat.com> References: <5203A524.7060808@redhat.com> Message-ID: Congrats team, this is a very nice list of new features. On Aug 8, 2013, at 10:03 AM, Martin Kosek wrote: > The FreeIPA team is proud to announce FreeIPA v3.3.0! > > It can be downloaded from http://www.freeipa.org/page/Downloads. Fedora 19 > builds are already on their way to updates-testing repo. > > == Highlights in 3.3.0 == > === New features for 3.3 === > * Active Directory integration: > ** Support of externally defined POSIX attributes for Active Directory trusted > domains > ** Automatic discovery of Active Directory identity mapping configuration > ** Support of trusted domain users for legacy clients > ** Identity mapping for AD users can now be delegated > * Performance improvements in processing large number of users and groups > * Automated integration testing infrastructure > * ipa-advise utility is added to generate client setup advice based on an IPA > master configuration > * FreeIPA-specific SELinux policies has been merged to the main SELinux policy > in Fedora 19 > * SSSD 1.11 is required > > === Active Directory integration === > Starting with FreeIPA 3.3, it is possible to define identity ranges for a > trusted Active Directory domain that rely on POSIX attributes provided by AD DC > instead of generating them out of corresponding security identifiers. This > functionality requires Services for Unix (SFU) or Identity Management for UNIX > enabled on Active Directory side and is provided mostly to aid with migration > to SID-based mapping. > > In order to support externally defined POSIX attributes, identity ranges have > been extended to support new range types: > * AD trust with SID-based mapping: 'ipa-ad-trust' (default) > * SFU support: 'ipa-ad-trust-posix' > > 'ipa-ad-trust-posix' range type is activated when range discovery finds out SFU > is in use by Active Directory domain. To override automatic detection, > --range-type=ipa-ad-trust can be specified to 'ipa trust-add' command. > > FreeIPA 3.3 requires SSSD 1.11 on the IPA master in order to support externally > defined POSIX attributes in AD. > > More details: http://www.freeipa.org/page/V3/Use_posix_attributes_defined_in_AD > > FreeIPA 3.3 provides a new way to enable legacy clients to support trusted > domain users. A compatibility tree, provided by slapi-nis, can now be > configured to look up trusted domain users and handle authentication for them. > This functionality relies on SSSD 1.11 and release 0.47.7 of slapi-nis. One can > enable legacy clients support by running ipa-adtrust-install and answering > positively to the corresponding question. > > More details: http://www.freeipa.org/page/V3/Serving_legacy_clients_for_trusts > > Finally, SSSD 1.11 is used to query identity information about trusted domains' > users from within IPA framework, including SID to name and name to SID > resolution. In addition to speed improvements, FreeIPA 3.3 allows to manage > mappings for trusted domains' users without requiring elevated privileges of > 'trust admins'. > > === Performance improvements === > When acting on large datasets, FreeIPA now reduces number of potential read > roundtrips required to update user and group information. When scaled to > thousands of users and groups, this shortens the time required by certain > operations tenfold. > > === Automated testing infrastructure === > The FreeIPA team has been providing self-testing code for a long time. > > The FreeIPA 3.3 test suite includes a framework for integration tests that > verify functionality such as replication across several machines. Tests can be > run manually, or by test automation servers such as Jenkins or Beaker. > > Development builds now create a freeipa-tests RPM containing the test suite and > related tools. However, as the focus is on testing development code, this > package will not be released to Fedora yet. > > More details: http://www.freeipa.org/page/V3/Integration_testing > > Additionally, it is now possible to run Web UI tests through the test suite. > > More details: http://www.freeipa.org/page/Web_UI_Integration_Tests > > === IPA advise tool === > FreeIPA 3.3 introduces new framework to generate recipes of configuration based > on how IPA master is configured. These recipes can be taken to the target > client systems and used there to configure them for a specific task. > > We expect to expand use of 'ipa-advise' tool to cover at least configuration of > legacy systems in subsequent releases. Three advices are provided with FreeIPA > 3.3.0 release: > * configuring a generic Fedora release with authconfig tool > * configuring RHEL-based systems with SSSD 1.5-1.8 > * configuring Debian-based systems with SSSD 1.5-1.8 > > Contributions are always welcome to grow capabilities of 'ipa-advise' tool to > other areas. > > More details: > http://www.freeipa.org/page/V3/Serving_legacy_clients_for_trusts#Major_configuration_options_and_enablement > > === SELinux policy === > SELinux policies specific to FreeIPA have been merged back to the main SELinux > policy package in Fedora 19. Starting with FreeIPA 3.2.2 (available in Fedora > 19 updates) SELinux policy is no londer provided by freeipa-selinux package and > the package is removed in favor of selinux-policy package. > > === SSSD 1.11 is required === > FreeIPA 3.3 depends on SSSD 1.11 for cross-realm trusts with Active Directory. > In particular, FreeIPA 3.3 depends on a new operational mode of SSSD called > 'ipa_server_mode'. Thus, SSSD 1.11 is required for FreeIPA 3.3. > > More details: https://fedorahosted.org/sssd/wiki/DesignDocs/IPAServerMode > > == Upgrading == > === FreeIPA servers with CA installed prior to version 3.1 === > Manual upgrade procedure is required for FreeIPA servers installed with version > prior to 3.1. > > === Other FreeIPA servers and clients === > An IPA server can be upgraded simply by installing updated rpms. The server > does not need to be shut down in advance. > > Please note that if you are doing the upgrade in special environment (e.g. > FedUp) which does not allow running LDAP server during upgrade process, upgrade > scripts need to be run manually after the first boot: > # ipa-upgradeconfig > # ipa-ldap-updater --upgrade > > Please note, that the performance improvements requires an extended set of > indexes to be configured. RPM update for an IPA server with a excessive number > of users may require several minutes to finish. > > If you have multiple servers you may upgrade them one at a time. It is expected > that all servers will be upgraded in a relatively short period (days or weeks > not months). They should be able to co-exist peacefully but new features will > not be available on old servers and enrolling a new client against an old > server will result in the SSH keys not being uploaded. > > Downgrading a server once upgraded is not supported. > > Upgrading from 2.2.0 and later versions is supported. Upgrading from previous > versions is not supported and has not been tested. > > An enrolled client does not need the new packages installed unless you want to > re-enroll it. SSH keys for already installed clients are not uploaded, you will > have to re-enroll the client or manually upload the keys. > > == Feedback == > Please provide comments, bugs and other feedback via the freeipa-users mailing > list (http://www.redhat.com/mailman/listinfo/freeipa-users) or #freeipa channel > on Freenode. > > == Detailed Changelog since 3.2.0 == > === Alexander Bokovoy (9): === > * Fix cldap parser to work with a single equality filter (NtVer=...) > * Make sure domain_name is also set when processing INP_NAME requests > * Fix extdom plugin to provide unqualified name in response as sssd expects > * Generate syntethic MS-PAC for all services running on IPA master > * ipa-adtrust-install: configure compatibility tree to serve trusted domain users > * ipa-kdb: cache KDC hostname on startup > * ipa-kdb: reinit mspac on HTTP TGT acquisition to aid trust-add case > * ipaserver/dcerpc: attempt to resolve SIDs through SSSD first > * Rename slapi-nis configuration variable > > === Ana Krivokapic (26): === > * Prompt for nameserver IP address in dnszone-add > * Do not display success message on failure in web UI > * Ignore files generated by build > * Deprecate options --dom-sid and --dom-name in idrange-mod > * Prevent error when running IPA commands with su/sudo > * Fix displaying of success message > * Fix location of service.crt in .gitignore > * Improve handling of options in ipa-client-install > * Fail when adding a trust with a different range > * Do not display traceback to user > * Require rid-base and secondary-rid-base in idrange-add after ipa-adtrust-install > * Fix bug in adtrustinstance > * Use correct DS instance in ipactl status > * Avoid systemd service deadlock during shutdown > * Make sure replication works after DM password is changed > * Use --ignore-dependencies only when necessary > * Properly handle non-existent cert files > * Add 'ipa_server_mode' option to SSSD configuration > * Bump version of sssd in spec file > * Use admin at REALM when testing if SSSD is ready > * Fix internal error in idrange-add > * Honor 'enabled' option for widgets. > * Expose ipaRangeType in Web UI > * Add ipa-advise plugins for legacy clients > * Enable running API commands in ipa-advise plugins > * Add new command compat-is-enabled > > === Diane Trout (1): === > * Fix log format not a string literal. > > === Jakub Hrozek (3): === > * Remove unused variable > * IPA KDB MS-PAC: return ENOMEM if allocation fails > * IPA KDB MS-PAC: remove unused variable > > === Jan Cholasta (21): === > * Use the correct PKCS#12 file for HTTP server. > * Remove stray error condition in ipa-server-install. > * Handle exceptions gracefully when verifying PKCS#12 files. > * Skip empty lines when parsing pk12util output. > * Do not allow installing CA replicas in CA-less setup. > * Do not track DS certificate in CA-less setup. > * Fix CA-less check in ipa-replica-install and ipa-ca-install. > * Do not skip SSSD known hosts in ipa-client-install --ssh-trust-dns. > * Enable SASL mapping fallback. > * Skip cert issuer validation in service and host commands in CA-less install. > * Check trust chain length in CA-less install. > * Use LDAP search instead of *group_show to check if a group exists. > * Use LDAP search instead of *group_show to check for a group objectclass. > * Use LDAP modify operation directly to add/remove group members. > * Add missing substring indices for attributes managed by the referint plugin. > * Add missing equality index for ipaUniqueId. > * Run gpg-agent explicitly when encrypting/decrypting files. > * Add new hidden command option to suppress processing of membership attributes. > * Ask for PKCS#12 password interactively in ipa-server-install. > * Ask for PKCS#12 password interactively in ipa-replica-prepare. > * Print newline after receiving EOF in installutils.read_password. > > === Lukas Slebodnik (4): === > * Use pkg-config to detect cmocka > * Use right function prototype for thread function > * Remove unused variable > * Remove unused variable > > === Martin Kosek (17): === > * Set KRB5CCNAME so that dirsrv can work with newer krb5-server > * Handle DIR type CCACHEs in test_cmdline properly > * Avoid exporting KRB5_KTNAME in dirsrv env > * Remove redundant u'' character > * Drop SELinux subpackage > * Drop redundant directory /var/cache/ipa/sessions > * Remove entitlement support > * Run server upgrade and restart in posttrans > * Require new selinux-policy replacing old server-selinux subpackage > * Bump minimum SSSD version > * Become 3.3.0 Beta 1 > * Free NSS objects in --external-ca scenario > * Use valid LDAP search base in migration plugin > * Increase default SASL buffer size > * Become 3.3.0 Beta 2 > * Add requires for slapi-nis and SSSD > * Become 3.3.0 > > === Nathaniel McCallum (10): === > * Add ipaUserAuthType and ipaUserAuthTypeClass > * Add IPA OTP schema and ACLs > * ipa-kdb: Add OTP support > * Add the krb5/FreeIPA RADIUS companion daemon > * Remove unnecessary prefixes from ipa-pwd-extop files > * Add OTP support to ipa-pwd-extop > * Fix client install exception if /etc/ssh is missing > * Permit reads to ipatokenRadiusProxyUser objects > * Fix for small syntax error in OTP schema > * Use libunistring ulc_casecmp() on unicode strings > > === Petr Spacek (1): === > * ipa-client-install: Add 'debug' and 'show' statements to nsupdate commands > > === Petr Viktorin (33): === > * Remove leading zero from IPA_NUM_VERSION > * Relax getkeytab test to allow additional messages on stderr > * Remove code to install Dogtag 9 > * Flush stream after writing service messages > * Make an ipa-tests package > * Add ipa-run-tests command > * Add Nose plugin for BeakerLib integration > * Add a plugin for test ordering > * Add a framework for integration test configuration > * Add a framework for integration testing > * Introduce a class for remote commands > * Collect logs from tests > * Show logs in failed tests > * tests: Allow public keys for authentication to the remote machines > * tests: Configure/unconfigure remote hosts > * Host class improvements > * Use dosctrings in BeakerLib phase descriptions > * Make BeakerLib logging less verbose > * BeakerLib plugin: Log http links in test docstrings > * Integration test config: Make it possible to specify host IP > * ipa-client: Use "ipa" as the package name for i18n > * Move BeakerLibProcess out of BeakerLibPlugin > * test_integration: Add log collection to Host > * test_integration: Set up CA on replicas by default > * Add more test tasks > * Add install_topo to test tasks > * Add the ipa-test-task tool > * Add tar and xz dependencies to the freeipa-tests package > * Correct default value of LDAPClient.get_entries scope argument > * test_simple_replication: Wait for replication to finish before checking > * Add the new no_member option to CLI tests > * Update translations > * Fix installutils.get_password without a TTY > > === Petr Vobornik (24): === > * Fix: HBAC Test tab is missing > * Move spec modifications from facet factories to pre_ops > * Unite and move facet pre_ops to related modules > * Web UI: move ./_base/metadata_provider.js to ./metadata.js > * Regression fix: missing control buttons in nested search facets > * Make ssbrowser.html work in IE 10 > * Fix regression: missing facet tab group labels > * Regression fix: rule table with ext. member support doesn't offer any items > * Fix default value selection in radio widget > * Do not redirect to https in /ipa/ui on non-HTML files > * Create Firefox configuration extension on CA-less install > * Disable checkboxes and radios for readonly attributes > * Better automated test support > * Fix container element in adder dialogs > * Upstream Web UI tests > * Web UI search optimization > * Break long words in notification area > * Remove word 'field' from GECOS param label > * Web UI integration tests: Add trust tests > * Web UI integration tests: Add ui_driver method descriptions > * Web UI integration tests: Verify data after add and mod > * Web UI integration tests: Compute range sizes to avoid overlaps > * Web UI integration tests: PEP8 fixes > * Web UI integration tests: Code quality fixes > > === Rob Crittenden (4): === > * Bump version for development branch to 3.2.99 > * Return the correct Content-type on negotiated XML-RPC requests. > * Add Camellia ciphers to allowed list. > * Hide sensitive attributes in LDAP updater logging and output > > === Simo Sorce (2): === > * CLDAP: Fix domain handling in netlogon requests > * CLDAP: Return empty reply on non-fatal errors > > === Sumit Bose (5): === > * Fix format string typo > * Fix type of printf argument > * Add PAC to master host TGTs > * extdom: replace winbind calls with POSIX/SSSD calls > * Remove winbind client configure check > > === Tomas Babej (32): === > * Remove redundancy from hbactest help text > * Do not translate trust type and direction with --raw in trust_show and trust-find > * Support multiple local domain ranges with RID base set > * Do not allow removal of ID range of an active trust > * Use private ccache in ipa install tools > * Remove redundant check for env.interactive > * Add prompt_param method to avoid code duplication > * Incorporate interactive prompts in idrange-add > * Do not check userPassword with 7-bit plugin > * Manage ipa-otpd.socket by IPA > * Add ipaRangeType attribute to LDAP Schema > * Add update plugin to fill in ipaRangeType attribute > * Extend idrange commands to support new range origin types > * PEP8 fixes in idrange.py > * Remove hardcoded values from idrange plugin tests > * Return ipaRangeType as a list in idrange commands > * Do not redirect ipa/crl to HTTPS > * Add --range-type option that forces range type of the trusted domain > * Add libsss_nss_idmap-devel to BuildRequires > * Change group ownership of CRL publish directory > * Provide ipa-advise tool > * Use AD LDAP probing to create trusted domain ID range > * Move requirement for keyutils to freeipa-python package > * Change shebang to absolute path in ipa-client-automount > * Skip referrals when converting LDAP result to LDAPEntry > * Refactor the interactive prompt logic in idrange_add > * Limit pwpolicy maxlife to 20000 days > * Use case-insensitive dict for trusted domain info > * Improve help entry for ipa host > * Remove overlapping use-cases of the same result variable > * Add a word wrapping for comment log messages to AdviceLogger > * Wrap lines in the list of available advices > > _______________________________________________ > Freeipa-interest mailing list > Freeipa-interest at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-interest From pspacek at redhat.com Mon Aug 12 10:47:43 2013 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 12 Aug 2013 12:47:43 +0200 Subject: [Freeipa-users] tough one on DNS In-Reply-To: References: Message-ID: <5208BD4F.1020000@redhat.com> Hello, I wonder if your problems with SSL are really caused by problems with reverse name resolution ... I think that SSL libraries usually don't care about PTR records. Which SSL libraries do you use? Do you use server's IP address in certificate subject field? Regarding the DNS: First of all, I would recommend to replace conditional forwarder on Windows side by normal DNS delegation from Windows to FreeIPA: Create NS+A record on Windows side, do not configure any forwarding from Windows to FreeIPA. Conditional forwarding usually creates more problems than it solves. Regarding the reverse zone: As you said, the correct approach is to have only one authoritative source for reverse zone - i.e. serve reverse zone only from Windows OR FreeIPA. Do I understand correctly that you want to allow Windows & non-Windows clients to update own records in the same reverse zone? It could be a tricky ... What are your security requirements? Do you require Kerberos authentication for each update? Do you have a FreeIPA-Windows trust in place? Obviously, the simplest thing is to not require update signing :-) Now seriously: BIND9 could allow you to specify that authenticated clients from REALM1 and REALM2 can do any update to PTR records in particular reverse zone, but it could be tricky to configure. (See match type 'wildcard' mentioned at ftp://ftp.isc.org/isc/bind9/9.9.3-P1/doc/arm/Bv9ARM.ch06.html#dynamic_update_policies.) If you need more granular control over each update then match type 'external' with your own ACL-daemon could be the way to go. I have no idea how DNS updates from trusted realms are processed on Windows side, it is possible that you can configure Windows to trust updates from FreeIPA realm (if there is a trust between Windows and FreeIPA). Please let us know if you need any further assistance. Petr^2 Spacek On 9.8.2013 15:29, Armstrong, Kenneth Lawrence wrote: > Hi all. > > We have IdM set up in a test environment as a subdomain of our Windows domain (so, linux.example.com) with integrated DNS on the IdM server and forwarders going to the AD servers on example.com. We have on the Windows DNS server the IdM server set up as a conditional forwarder for linux.example.com and an A record in its DNS for the IdM server. > > However, on example.com, we have other subdomains that need to be able to communicate with the linux.example.com domain, such as tier1.example.com, tier2.example.com, etc. > > What we are trying to figure out is the whole PTR reverse zone bit, mainly due to the fact that the linux.example.com and the example.com reside on the same subnets (and unfortunately, this is a very large network and we can't change that). > > For instance, say we have a system at test1.tier2.example.com that needs to make an SSL connection to another system on testA.linux.example.com. The SSL handshake will fail since the PTR record will resolve to a different domain than what the client is expecting, again due to the fact that the reverse zone is on the same subnet. The reverse zone on the Windows DNS server would only contain PTR records for all of the domains except linux.example.com, and the IdM server would only contain PTR records for just the linux.example.com systems. > > So, obviously, we would not want the IdM server to have a reverse zone, and have it rely on the reverse zone on the Windows DNS server. Other than manually entering PTR records for all Linux systems under linux.example.com to the reverse zone file on the Windows DNS server, is there another way that we can properly accomplish this? Would replicating the reverse zone file from the Windows server to the IdM server take care of that? > > Thanks. > > -Kenny -- Petr^2 Spacek From brian_lee1 at jabil.com Mon Aug 12 15:24:03 2013 From: brian_lee1 at jabil.com (Brian Lee) Date: Mon, 12 Aug 2013 11:24:03 -0400 Subject: [Freeipa-users] Blocking 389 and 636 for AD trusts Message-ID: Hello everyone, I understand this is well documented that we need to block AD from establishing communication to the LDAP ports, but I've never heard an explanation on why this is needed. Additionally, In our environment, we have a 100+ AD servers. Do I need to add an iptables rule for each AD server, on each IPA server or only the ones configured for DNS forwarding? Thanks as always -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Mon Aug 12 15:30:45 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 12 Aug 2013 11:30:45 -0400 Subject: [Freeipa-users] Can't update ssh keys In-Reply-To: <5208FAF7.7010001@redhat.com> References: <52052557.6070400@redhat.com> <5208FAF7.7010001@redhat.com> Message-ID: I can get the host keys in okay, it's the user keys that are giving me fits. No combination of fields seems to work. Before we troubleshoot very far, I will update to a newer release and try again. Every now and again, I just need the right motivation to upgrade. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 12, 2013 at 11:10 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> Rob, >> >> I'm running 2.2.1. Sorry about that, I got confused by my Cobbler >> version on a different server. I guess we're looking at an upgrade! >> > > For 2.x you might try: > > # ipa host-mod host.example.com --sshpubkey=`awk '{ print $2 '} > /etc/ssh/ssh_host_rsa_key.pub` > > I'm not 100% sure that the pub key format is space-delimited so YMMV, but > I think this is right. > > rob > > >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Fri, Aug 9, 2013 at 1:22 PM, Rob Crittenden > > wrote: >> >> Bret Wortman wrote: >> >> Any time I try to use the command-line utilities to add a host >> (this >> includes ipa-client-install): >> >> #ipa host-mod host.damascusgrp.com >> >> >> >> --updatedns >> >> --sshpubkey="`cat /etc/ssh/ssh_host_rsa_key.pub`**__" >> >> ipa: ERROR: invliad 'sshpubkey': must be binary data >> >> I know I can use the GUI, but as we could be rolling out a large >> number >> of systems in coming months, that's not a good long-term option. >> So does >> anyone know a way to make the CLI tools work? >> >> Second question: is there a way to update the SSHFP records >> apart from >> using the CLI tools as above? >> >> >> A pub key consists of 3 pieces of data: the key type, the key and a >> comment. >> >> What version of IPA? IIRC in v2 only the key material itself was >> supported. This cli command should work with a v3 server which >> requires all 3 pieces. >> >> I imagine you could use dnsrecord-mod/add to manage the SSHFP record >> but that could lead to different values in the DNS and host entry if >> not done carefully. >> >> rob >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From luis_lugo74 at hotmail.com Mon Aug 12 17:37:05 2013 From: luis_lugo74 at hotmail.com (luis lugo) Date: Mon, 12 Aug 2013 17:37:05 +0000 Subject: [Freeipa-users] Freeipa Active Directory Sync problems Message-ID: Hi, I have the following error when I try to sync Freeipa 3.2.2 with Active Directory. reports: Update failed! Status: [-1 Total update abortedLDAP error: Can't contact LDAP server] Failed to start replication All current users sync with freeipa, but new users cannot. I have differents OU and I need to sync all users in my active directories. I use the following ipa-replica-manage switches to created the sync. ipa-replica-manage connect --winsync --binddn='cn=Administrator,cn=Users,dc=domain,dc=com' --bindpw='' --cacert=/root/ADCA.cer --passsync='' --win-subtree='OU=test,OU=users,DC=domain,DC=com' windows-server-hostname In the dirsrv logs I have the following error. [12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication agreement for agmt="cn=meTo" (nigua:389) could not be updated. For replication to take place, please enable the suffix and restart the server[12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication agreement for agmt="cn=meTo" (nigua:389) could not be updated. For replication to take place, please enable the suffix and restart the server[12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication agreement for agmt="cn=me" (nigua:389) could not be updated. For replication to take place, please enable the suffix and restart the server[12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication agreement for agmt="cn=meTo" (nigua:389) could not be updated. For replication to take place, please enable the suffix and restart the server[12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - agmt="cn=meTo" (nigua:389): Replica has no update vector. It has never been initialized.[12/Aug/2013:10:45:18 -0400] - Entry -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmeggins at redhat.com Mon Aug 12 17:41:51 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Mon, 12 Aug 2013 11:41:51 -0600 Subject: [Freeipa-users] Freeipa Active Directory Sync problems In-Reply-To: References: Message-ID: <52091E5F.106@redhat.com> On 08/12/2013 11:37 AM, luis lugo wrote: > Hi, > > I have the following error when I try to sync Freeipa 3.2.2 with > Active Directory. > > reports: Update failed! Status: [-1 Total update abortedLDAP error: > Can't contact LDAP server] > > Failed to start replication > > > All current users sync with freeipa, but new users cannot. I have > differents OU and I need to sync all users in my active directories. I > use the following ipa-replica-manage switches to created the sync. > > ipa-replica-manage connect --winsync > --binddn='cn=Administrator,cn=Users,dc=domain,dc=com' --bindpw='' > --cacert=/root/ADCA.cer --passsync='' > --win-subtree='OU=test,OU=users,DC=domain,DC=com' windows-server-hostname > > In the dirsrv logs I have the following error. > > [12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication > agreement for agmt="cn=meTo" (nigua:389) could not be updated. For > replication to take place, please enable the suffix and restart the server > [12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication > agreement for agmt="cn=meTo" (nigua:389) could not be updated. For > replication to take place, please enable the suffix and restart the server > [12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication > agreement for agmt="cn=me" (nigua:389) could not be updated. For > replication to take place, please enable the suffix and restart the server > [12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - Replication > agreement for agmt="cn=meTo" (nigua:389) could not be updated. For > replication to take place, please enable the suffix and restart the server > [12/Aug/2013:10:45:18 -0400] NSMMReplicationPlugin - agmt="cn=meTo" > (nigua:389): Replica has no update vector. It has never been initialized. > [12/Aug/2013:10:45:18 -0400] - Entry This is truncated. Please provide more. > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From aellert at numeezy.com Tue Aug 13 09:54:53 2013 From: aellert at numeezy.com (Alexandre Ellert) Date: Tue, 13 Aug 2013 11:54:53 +0200 Subject: [Freeipa-users] sudo rule applied to a host group Message-ID: <390D567C-86C3-49CE-9FF2-505131BE63D0@numeezy.com> Hi, I'm trying to get working a sudo rule for a group of user, basically if want to allow all the developers (dev-users) to become root on developers servers (dev-servers). When this rule is applied to a single host or all hosts or severals named host, it works fine : dev-users can sudo without prompting for a password (I have sudo option !authenticate) But if I apply the rule to the dev-servers group, it doesn't work : when a member of dev-users try to sudo, it prompt for a password and even the password is correct, password is asked again. I use ipa-server-3.0.0-26.el6_4.4 and RHEL 6 and a custom Debian package for clients (based on freeipa 3.0.2). I checked /etc/sudo-ldap.conf, /etc/nsswitch.conf and /etc/rc.local on clients and everything seems correct. Do i missed something ? Thanks for your help. Alexandre. From mindaugas.deveikis at itreegroup.eu Tue Aug 13 10:45:50 2013 From: mindaugas.deveikis at itreegroup.eu (Mindaugas Deveikis) Date: Tue, 13 Aug 2013 13:45:50 +0300 Subject: [Freeipa-users] ldap connection over tcp/ip Message-ID: <520A0E5E.2080203@itreegroup.eu> Hi We try to do a very simple situation. Our ldap server is on different machine than IPA server. So we try to use tcp/ip connection. All that we do is just edit default.conf file on IPA server and change ldap_uri line value from socket to IP address. But after that change, an error received when trying to restart ipa server. The error is: Failed to data from service file: Unknown error when retrieving list of services from LDAP: Unknown authentication meth... available (it comes from systemctl status ipa.service). We use fedora 19. Firewalld works fine. Could anybody show me a way how to solve this problem. Thank's Mindaugas From rcritten at redhat.com Tue Aug 13 12:25:27 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 13 Aug 2013 08:25:27 -0400 Subject: [Freeipa-users] ldap connection over tcp/ip In-Reply-To: <520A0E5E.2080203@itreegroup.eu> References: <520A0E5E.2080203@itreegroup.eu> Message-ID: <520A25B7.60408@redhat.com> Mindaugas Deveikis wrote: > Hi > > We try to do a very simple situation. Our ldap server is on > different machine than IPA server. So we try to use tcp/ip connection. > All that we do is just edit default.conf file on IPA server and change > ldap_uri line value from socket to IP address. But after that change, an > error received when trying to restart ipa server. The error is: Failed > to data from service file: Unknown error when retrieving list of > services from LDAP: Unknown authentication meth... available (it comes > from systemctl status ipa.service). We use fedora 19. Firewalld works > fine. Could anybody show me a way how to solve this problem. Thank's So you want to run the IPA LDAP server on another machine? This is not a supported configuration. It is assumed that all IPA services run on the same machine. There may be some places hardcoded to use ldapi. Even if you solved that you'd likely have future problems with upgrades, keeping versions in sync and almost certainly with replication. rob From rcritten at redhat.com Tue Aug 13 12:42:08 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 13 Aug 2013 08:42:08 -0400 Subject: [Freeipa-users] sudo rule applied to a host group In-Reply-To: <390D567C-86C3-49CE-9FF2-505131BE63D0@numeezy.com> References: <390D567C-86C3-49CE-9FF2-505131BE63D0@numeezy.com> Message-ID: <520A29A0.102@redhat.com> Alexandre Ellert wrote: > Hi, > > I'm trying to get working a sudo rule for a group of user, basically if want to allow all the developers (dev-users) to become root on developers servers (dev-servers). > When this rule is applied to a single host or all hosts or severals named host, it works fine : dev-users can sudo without prompting for a password (I have sudo option !authenticate) > But if I apply the rule to the dev-servers group, it doesn't work : when a member of dev-users try to sudo, it prompt for a password and even the password is correct, password is asked again. > > I use ipa-server-3.0.0-26.el6_4.4 and RHEL 6 and a custom Debian package for clients (based on freeipa 3.0.2). > I checked /etc/sudo-ldap.conf, /etc/nsswitch.conf and /etc/rc.local on clients and everything seems correct. > > Do i missed something ? > > Thanks for your help. hostgroups are visible as netgroups on client machines, so you need a working netgroups configuration. You should have sss as a provider for netgroup in /etc/nsswitch.conf and you need to set the NIS domain name via nisdomainname (to match your domain name). You can test fetching a hostgroup as a netgroup with: getent netgroup dev-users. It should look something like: dev-users (host1.example.com,-,example.com) (host2.example.com,-,example.com) rob From aellert at numeezy.com Tue Aug 13 12:53:50 2013 From: aellert at numeezy.com (Alexandre Ellert) Date: Tue, 13 Aug 2013 14:53:50 +0200 Subject: [Freeipa-users] sudo rule applied to a host group In-Reply-To: <520A29A0.102@redhat.com> References: <390D567C-86C3-49CE-9FF2-505131BE63D0@numeezy.com> <520A29A0.102@redhat.com> Message-ID: Thank you so much Rob ! It works juste fine :) Alexandre Le 13 ao?t 2013 ? 14:42, Rob Crittenden a ?crit : > Alexandre Ellert wrote: >> Hi, >> >> I'm trying to get working a sudo rule for a group of user, basically if want to allow all the developers (dev-users) to become root on developers servers (dev-servers). >> When this rule is applied to a single host or all hosts or severals named host, it works fine : dev-users can sudo without prompting for a password (I have sudo option !authenticate) >> But if I apply the rule to the dev-servers group, it doesn't work : when a member of dev-users try to sudo, it prompt for a password and even the password is correct, password is asked again. >> >> I use ipa-server-3.0.0-26.el6_4.4 and RHEL 6 and a custom Debian package for clients (based on freeipa 3.0.2). >> I checked /etc/sudo-ldap.conf, /etc/nsswitch.conf and /etc/rc.local on clients and everything seems correct. >> >> Do i missed something ? >> >> Thanks for your help. > > hostgroups are visible as netgroups on client machines, so you need a working netgroups configuration. You should have sss as a provider for netgroup in /etc/nsswitch.conf and you need to set the NIS domain name via nisdomainname (to match your domain name). > > You can test fetching a hostgroup as a netgroup with: getent netgroup dev-users. It should look something like: > > dev-users (host1.example.com,-,example.com) (host2.example.com,-,example.com) > > rob From bret.wortman at damascusgrp.com Tue Aug 13 14:29:34 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 13 Aug 2013 10:29:34 -0400 Subject: [Freeipa-users] Upgrade failed -- how to recover? Message-ID: I just upgraded my IPA master from F17 to F18 and, in the process, updated IPA to 3.1.5-1. Apparently, though, all is not well, because there are a number of errors in /var/log/ipaupgrade.log, mostly related to things like (samples here; the server is on a private network so I'm having to transcribe, if it looks like a typo, it probably is): ERROR Cannot connect to LDAP to add DNS records: cannot connect to u'ldapi://%2fvar%2run%2fslapd-FOO-NET.socket': LDAP Server Down ERROR certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-retrieve-agent-submit -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca" -P XXXXXXXX -T auditSigningCert cert-pki-ca' returned non-zero exit status 1 and numerous certmonger errors similar to this one. Finally, there's a stacktrace from ipapython/admintool.py, line 171 which ends the whole thing. What's my best plan for re-attempting this upgrade? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Tue Aug 13 17:12:10 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 13 Aug 2013 13:12:10 -0400 Subject: [Freeipa-users] Upgrade failed -- how to recover? In-Reply-To: References: Message-ID: I tried this, but no joy: # /usr/sbin/ipa-upgradeconfig --debug : : DEBUG: caSignedLogCert.cfgprofile validity range is 720 INFO: [Certificate renewal should stop the CA] ERROR: Unable to find certmonger request ID for auditSigning Cert INFO: The ipa-upgradeconfig command was successful # But I still can't connect to http://ipamaster/ipa/ui/; I get a 903 error every time, and /var/log/httpd/error_log shows, in part: [Tue Aug 13 13:07:20.786566 2013] [:error] [pid 5890] KeyError: 'ipadnszone' [Tue Aug 13 13:07:20.786717 2013] [:error] [pid 5890] ipa: INFO: bretw at FOO.NET: json_metadata(None, None, object=u'all'): KeyError [Tue Aug 13 13:07:21.001525 2013] [:error] [pid 5890] ipa: INFO: bretw at FOO.NET: json_metadata(None, None, command=u'all'): SUCCESS DNS resolution, authentication and authorization all *appear* to be working fine. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Tue, Aug 13, 2013 at 10:29 AM, Bret Wortman wrote: > I just upgraded my IPA master from F17 to F18 and, in the process, updated > IPA to 3.1.5-1. Apparently, though, all is not well, because there are a > number of errors in /var/log/ipaupgrade.log, > mostly related to things like (samples here; the server is on a private > network so I'm having to transcribe, if it looks like a typo, it probably > is): > > ERROR Cannot connect to LDAP to add DNS records: cannot connect to > u'ldapi://%2fvar%2run%2fslapd-FOO-NET.socket': LDAP Server Down > > ERROR certmonger failed to start tracking certificate: Command > '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n > auditSigningCert cert-pki-ca -c dogtag-ipa-retrieve-agent-submit -B > /usr/lib64/ipa/certmonger/stop_pkicad -C > /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca" -P > XXXXXXXX -T auditSigningCert cert-pki-ca' returned non-zero exit status 1 > > and numerous certmonger errors similar to this one. Finally, there's a > stacktrace from ipapython/admintool.py, > line 171 which ends the whole thing. > > What's my best plan for re-attempting this upgrade? > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Tue Aug 13 19:39:19 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 13 Aug 2013 15:39:19 -0400 Subject: [Freeipa-users] Upgrade failed -- how to recover? In-Reply-To: References: Message-ID: <520A8B67.7040301@redhat.com> Bret Wortman wrote: > I tried this, but no joy: > > # /usr/sbin/ipa-upgradeconfig --debug > : > : > DEBUG: caSignedLogCert.cfg > profile > validity range is 720 > INFO: [Certificate renewal should stop the CA] > ERROR: Unable to find certmonger request ID for auditSigning Cert > INFO: The ipa-upgradeconfig command was successful > # Run getcert list and sift through the output and see if you have a request tracking for nickname auditSigningCert cert-pki-ca (or similar). > But I still can't connect to http://ipamaster/ipa/ui/; I get a 903 error > every time, and /var/log/httpd/error_log shows, in part: > > [Tue Aug 13 13:07:20.786566 2013] [:error] [pid 5890] KeyError: 'ipadnszone' > [Tue Aug 13 13:07:20.786717 2013] [:error] [pid 5890] ipa: INFO: > bretw at FOO.NET : json_metadata(None, None, > object=u'all'): KeyError > [Tue Aug 13 13:07:21.001525 2013] [:error] [pid 5890] ipa: INFO: > bretw at FOO.NET : json_metadata(None, None, > command=u'all'): SUCCESS > > DNS resolution, authentication and authorization all /appear/ to be > working fine. The DNS schema was not updated properly. I'd run: # ipa-ldap-updater --upgrade rob From aissa.brahimi at itsoninc.com Tue Aug 13 23:27:56 2013 From: aissa.brahimi at itsoninc.com (Aissa Brahimi) Date: Tue, 13 Aug 2013 16:27:56 -0700 Subject: [Freeipa-users] Freeipa-users Digest, Vol 61, Issue 21 In-Reply-To: References: Message-ID: Hi, I am having this issue: IPA server: CentOS6.x Host CentOS 5.x 2 different host and cannot join the IPA server: Here the 2 different output I got: There was a problem importing one of the required Python modules. The error was: No module named OpenSSL and .. bash: ipa-client-install: command not found On Tue, Aug 13, 2013 at 9:00 AM, wrote: > Send Freeipa-users mailing list submissions to > freeipa-users at redhat.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.redhat.com/mailman/listinfo/freeipa-users > or, via email, send a message with subject or body 'help' to > freeipa-users-request at redhat.com > > You can reach the person managing the list at > freeipa-users-owner at redhat.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Freeipa-users digest..." > > > Today's Topics: > > 1. Upgrade failed -- how to recover? (Bret Wortman) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 13 Aug 2013 10:29:34 -0400 > From: Bret Wortman > To: freeipa-users at redhat.com > Subject: [Freeipa-users] Upgrade failed -- how to recover? > Message-ID: > N1g at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > I just upgraded my IPA master from F17 to F18 and, in the process, updated > IPA to 3.1.5-1. Apparently, though, all is not well, because there are a > number of errors in > /var/log/ipaupgrade.log< > http://bl-1.com/click/load/BzZcbVU2VmpTOwFsCD4-b0231>, > mostly related to things like (samples here; the server is on a private > network so I'm having to transcribe, if it looks like a typo, it probably > is): > > ERROR Cannot connect to LDAP to add DNS records: cannot connect to > u'ldapi://%2fvar%2run%2fslapd-FOO-NET.socket': LDAP Server Down > > ERROR certmonger failed to start tracking certificate: Command > '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n > auditSigningCert cert-pki-ca -c dogtag-ipa-retrieve-agent-submit -B > /usr/lib64/ipa/certmonger/stop_pkicad -C > /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca" -P > XXXXXXXX -T auditSigningCert cert-pki-ca' returned non-zero exit status 1 > > and numerous certmonger errors similar to this one. Finally, there's a > stacktrace from > ipapython/admintool.py< > http://bl-1.com/click/load/BzYIOV0-b0221AT1QOFc6BjE-b0231>, > line 171 which ends the whole thing. > > What's my best plan for re-attempting this upgrade? > > * > * > *Bret Wortman* > > http://damascusgrp.com/< > http://bl-1.com/click/load/VWQAMVQ3UGxVPQBtADQ-b0231> > http://about.me/wortmanbret< > http://bl-1.com/click/load/XWwMPV0-b0221UW0CagZrBjM-b0231> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://www.redhat.com/archives/freeipa-users/attachments/20130813/f32400c2/attachment.html > > > > ------------------------------ > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > > End of Freeipa-users Digest, Vol 61, Issue 21 > ********************************************* > -- Aissa Brahimi IT Admin - Support ItsOn, Inc. itsoninc.com mobile: 408.858.0304 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew at andrewklau.com Wed Aug 14 06:00:06 2013 From: andrew at andrewklau.com (Andrew Lau) Date: Wed, 14 Aug 2013 16:00:06 +1000 Subject: [Freeipa-users] IPA Server UI Behind Proxy Message-ID: Hi, I've got my FreeIPA setup in an internal infrastructure, but I want to be able to have users access the web UI externally. I tweaked the ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a nginx reverse proxy and port forwarding, both works if the client manually sets the host name of the IPA server eg. ipa01.internaldomain.local in their /etc/hosts file. However if the client tries to to use eg. ipa.externaldomain.com with the same port forwarding or nginx proxy config, it'll silently error. The docs briefly touches on this - but doesn't really give much to go on. Any suggestions? Andrew . -------------- next part -------------- An HTML attachment was scrubbed... URL: From abokovoy at redhat.com Wed Aug 14 06:23:18 2013 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 14 Aug 2013 09:23:18 +0300 Subject: [Freeipa-users] IPA Server UI Behind Proxy In-Reply-To: References: Message-ID: <20130814062318.GH27194@redhat.com> On Wed, 14 Aug 2013, Andrew Lau wrote: >Hi, > >I've got my FreeIPA setup in an internal infrastructure, but I want to be >able to have users access the web UI externally. I tweaked the >ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a >nginx reverse proxy and port forwarding, both works if the client manually >sets the host name of the IPA server eg. ipa01.internaldomain.local in >their /etc/hosts file. However if the client tries to to use eg. >ipa.externaldomain.com with the same port forwarding or nginx proxy config, >it'll silently error. The docs briefly touches on this - but doesn't really >give much to go on. > >Any suggestions? Couple considerations here. First, you may need to play with debug level to see what principal mod_auth_kerb picks up when negotiation happens. Second, using Kerberos authentication requires both sides to own Kerberos principals for authentication process to go. Browsers select HTTP/server.fqdn as their target service principal when connection to the server.fqdn. Your IPA server only has HTTP/ipa01.internaldomain.local in its keytab in /etc/httpd/conf/ipa.keytab, which means it would only be able to respond on ipa01.internaldomain.local to Kerberos auth requests. A way to fix this is by making HTTP/ipa.externaldomain.com service in IPA (ipa service-add HTTP/ipa.externaldomain.com) and then use 'ipa-getkeytab' to fetch the keytab with the key for the service. Next step would be to merge content of this keytab with /etc/httpd/conf/ipa.keytab: (echo rkt /tmp/external.keytab; echo wkt /etc/httpd/conf/ipa.keytab) |ktutil Then restart httpd -- I'm not sure mod_auth_kerb re-reads the keytab on its change. -- / Alexander Bokovoy From andrew at andrewklau.com Wed Aug 14 06:49:05 2013 From: andrew at andrewklau.com (Andrew Lau) Date: Wed, 14 Aug 2013 16:49:05 +1000 Subject: [Freeipa-users] IPA Server UI Behind Proxy In-Reply-To: <20130814062318.GH27194@redhat.com> References: <20130814062318.GH27194@redhat.com> Message-ID: I followed your suggestions without much luck. Adding the kerberos keytab didn't change anything, when I try login through the UI it just redirects me again with the same notice: Your session has expired. Please re-login. However if I login with the incorrect details logs will show INFO: 401 Unauthorized: kinit: Client 'asd at DOMAIN.COM' not found in Kerberos database while getting initial credentials and the UI will give me an error message. It seems when it's logged in with the correct credentials it's still finding itself lost. I have a feeling I'm overlooking something so simple.. Andrew On Wed, Aug 14, 2013 at 4:23 PM, Alexander Bokovoy wrote: > On Wed, 14 Aug 2013, Andrew Lau wrote: > >> Hi, >> >> I've got my FreeIPA setup in an internal infrastructure, but I want to be >> able to have users access the web UI externally. I tweaked the >> ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a >> nginx reverse proxy and port forwarding, both works if the client manually >> sets the host name of the IPA server eg. ipa01.internaldomain.local in >> their /etc/hosts file. However if the client tries to to use eg. >> ipa.externaldomain.com with the same port forwarding or nginx proxy >> config, >> it'll silently error. The docs briefly touches on this - but doesn't >> really >> give much to go on. >> >> Any suggestions? >> > Couple considerations here. > > First, you may need to play with debug level to see what principal > mod_auth_kerb picks up when negotiation happens. > > Second, using Kerberos authentication requires both sides to own > Kerberos principals for authentication process to go. Browsers select > HTTP/server.fqdn as their target service principal when connection to > the server.fqdn. Your IPA server only has HTTP/ipa01.internaldomain.** > local > in its keytab in /etc/httpd/conf/ipa.keytab, which means it would only > be able to respond on ipa01.internaldomain.local to Kerberos auth requests. > > A way to fix this is by making HTTP/ipa.externaldomain.com service in > IPA (ipa service-add HTTP/ipa.externaldomain.com) and then use > 'ipa-getkeytab' to fetch the keytab with the key for the service. Next > step would be to merge content of this keytab with > /etc/httpd/conf/ipa.keytab: > > (echo rkt /tmp/external.keytab; echo wkt /etc/httpd/conf/ipa.keytab) > |ktutil > > Then restart httpd -- I'm not sure mod_auth_kerb re-reads the keytab on > its change. > > -- > / Alexander Bokovoy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pvoborni at redhat.com Wed Aug 14 07:36:42 2013 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 14 Aug 2013 09:36:42 +0200 Subject: [Freeipa-users] IPA Server UI Behind Proxy In-Reply-To: References: Message-ID: <520B338A.2070605@redhat.com> On 08/14/2013 08:00 AM, Andrew Lau wrote: > Hi, > > I've got my FreeIPA setup in an internal infrastructure, but I want to be > able to have users access the web UI externally. I tweaked the > ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a > nginx reverse proxy and port forwarding, both works if the client manually > sets the host name of the IPA server eg. ipa01.internaldomain.local in > their /etc/hosts file. However if the client tries to to use eg. > ipa.externaldomain.com with the same port forwarding or nginx proxy config, > it'll silently error. The docs briefly touches on this - but doesn't really > give much to go on. > > Any suggestions? > > Andrew > . > Hi, FreeIPA RPC API, which Web UI uses, requires http referer header to start with 'https:///ipa'. Given that you are using proxy, I assume that the referer is different and might be a cause of the issue. HTH -- Petr Vobornik From andrew at andrewklau.com Wed Aug 14 08:36:10 2013 From: andrew at andrewklau.com (Andrew Lau) Date: Wed, 14 Aug 2013 18:36:10 +1000 Subject: [Freeipa-users] IPA Server UI Behind Proxy In-Reply-To: <520B338A.2070605@redhat.com> References: <520B338A.2070605@redhat.com> Message-ID: Any suggestions or workaround, short of having to switch the IPA's hostname to use a public domain? Andrew On Wed, Aug 14, 2013 at 5:36 PM, Petr Vobornik wrote: > On 08/14/2013 08:00 AM, Andrew Lau wrote: > >> Hi, >> >> I've got my FreeIPA setup in an internal infrastructure, but I want to be >> able to have users access the web UI externally. I tweaked the >> ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a >> nginx reverse proxy and port forwarding, both works if the client manually >> sets the host name of the IPA server eg. ipa01.internaldomain.local in >> their /etc/hosts file. However if the client tries to to use eg. >> ipa.externaldomain.com with the same port forwarding or nginx proxy >> config, >> it'll silently error. The docs briefly touches on this - but doesn't >> really >> give much to go on. >> >> Any suggestions? >> >> Andrew >> . >> >> Hi, > > FreeIPA RPC API, which Web UI uses, requires http referer header to start > with 'https://**/ipa'. Given that you are using > proxy, I assume that the referer is different and might be a cause of the > issue. > > HTH > -- > Petr Vobornik > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Wed Aug 14 08:50:04 2013 From: sbose at redhat.com (Sumit Bose) Date: Wed, 14 Aug 2013 10:50:04 +0200 Subject: [Freeipa-users] Blocking 389 and 636 for AD trusts In-Reply-To: References: Message-ID: <20130814085004.GX22990@localhost.localdomain> On Mon, Aug 12, 2013 at 11:24:03AM -0400, Brian Lee wrote: > Hello everyone, > > I understand this is well documented that we need to block AD from > establishing communication to the LDAP ports, but I've never heard an > explanation on why this is needed. > > Additionally, In our environment, we have a 100+ AD servers. Do I need to > add an iptables rule for each AD server, on each IPA server or only the > ones configured for DNS forwarding? > > Thanks as always Thank you for bringing up this topic. I've discussed this with Alexander and we think that this recommendation can be dropped. I have updated http://www.freeipa.org/page/Howto/IPAv3_AD_trust_setup. The new version now says: """ Previously we recommended that you should make sure that IPA LDAP server is not reachable by AD DC by closing down TCP ports 389 and 636 for AD DC. Our current tests lead to the assumption that this is not necessary anymore. During the early development stage we tried to create a trust between IPA and AD with both IPA and AD tools. It turned out that the AD tools expect an AD like LDAP schema and layout to create a trust. Since the IPA LDAP server does not meet those requirements it is not possible to create a trust between IPA and AD with AD tools only with the 'ipa trust-add' command. By blocking the LDAP ports for the AD DC we tried to force the AD tools to fall back to other means to get the needed information with no success. But we kept the recommendation to block those ports because it was not clear at this time if AD will check the LDAP layout of a trust partner during normal operation as well. Since we have not observed those request the recommendation can be dropped. """ HTH bye, Sumit > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users From pvoborni at redhat.com Wed Aug 14 10:35:53 2013 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 14 Aug 2013 12:35:53 +0200 Subject: [Freeipa-users] IPA Server UI Behind Proxy In-Reply-To: References: <520B338A.2070605@redhat.com> Message-ID: <520B5D89.2050800@redhat.com> On 08/14/2013 10:36 AM, Andrew Lau wrote: > Any suggestions or workaround, short of having to switch the IPA's hostname > to use a public domain? IDK if anyone did that, but you can try to change the header by the proxy. You should change it only for the request with referer: 'https://ipa.externaldomain.com/ipa' to keep to original logic [1] in place. That means the proxy will work as man-in-the-middle and should have access to certificate for the external domain. Also, external users should not use FreeIPA browser configuration pages because of the different domain. [1] https://bugzilla.redhat.com/show_bug.cgi?id=747710 > > Andrew > > > On Wed, Aug 14, 2013 at 5:36 PM, Petr Vobornik wrote: > >> On 08/14/2013 08:00 AM, Andrew Lau wrote: >> >>> Hi, >>> >>> I've got my FreeIPA setup in an internal infrastructure, but I want to be >>> able to have users access the web UI externally. I tweaked the >>> ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a >>> nginx reverse proxy and port forwarding, both works if the client manually >>> sets the host name of the IPA server eg. ipa01.internaldomain.local in >>> their /etc/hosts file. However if the client tries to to use eg. >>> ipa.externaldomain.com with the same port forwarding or nginx proxy >>> config, >>> it'll silently error. The docs briefly touches on this - but doesn't >>> really >>> give much to go on. >>> >>> Any suggestions? >>> >>> Andrew >>> . >>> >>> Hi, >> >> FreeIPA RPC API, which Web UI uses, requires http referer header to start >> with 'https://**/ipa'. Given that you are using >> proxy, I assume that the referer is different and might be a cause of the >> issue. >> >> HTH >> -- >> Petr Vobornik >> > -- Petr Vobornik From brian_lee1 at jabil.com Wed Aug 14 10:47:38 2013 From: brian_lee1 at jabil.com (Brian Lee) Date: Wed, 14 Aug 2013 06:47:38 -0400 Subject: [Freeipa-users] Blocking 389 and 636 for AD trusts In-Reply-To: <20130814085004.GX22990@localhost.localdomain> References: <20130814085004.GX22990@localhost.localdomain> Message-ID: Great news! Thanks for the update. On Wed, Aug 14, 2013 at 4:50 AM, Sumit Bose wrote: > On Mon, Aug 12, 2013 at 11:24:03AM -0400, Brian Lee wrote: > > Hello everyone, > > > > I understand this is well documented that we need to block AD from > > establishing communication to the LDAP ports, but I've never heard an > > explanation on why this is needed. > > > > Additionally, In our environment, we have a 100+ AD servers. Do I need to > > add an iptables rule for each AD server, on each IPA server or only the > > ones configured for DNS forwarding? > > > > Thanks as always > > Thank you for bringing up this topic. I've discussed this with > Alexander and we think that this recommendation can be dropped. > > I have updated http://www.freeipa.org/page/Howto/IPAv3_AD_trust_setup. > The new version now says: > > """ > Previously we recommended that you should make sure that IPA LDAP server > is not reachable by AD DC by closing down TCP ports 389 and 636 for AD > DC. Our current tests lead to the assumption that this is not necessary > anymore. During the early development stage we tried to create a trust > between IPA and AD with both IPA and AD tools. It turned out that the AD > tools expect an AD like LDAP schema and layout to create a trust. Since > the IPA LDAP server does not meet those requirements it is not possible > to create a trust between IPA and AD with AD tools only with the 'ipa > trust-add' command. By blocking the LDAP ports for the AD DC we tried to > force the AD tools to fall back to other means to get the needed > information with no success. But we kept the recommendation to block > those ports because it was not clear at this time if AD will check the > LDAP layout of a trust partner during normal operation as well. Since we > have not observed those request the recommendation can be dropped. > """ > > HTH > > bye, > Sumit > > > _______________________________________________ > > Freeipa-users mailing list > > Freeipa-users at redhat.com > > https://www.redhat.com/mailman/listinfo/freeipa-users > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Wed Aug 14 11:58:06 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 14 Aug 2013 07:58:06 -0400 Subject: [Freeipa-users] Upgrade failed -- how to recover? In-Reply-To: <520A8B67.7040301@redhat.com> References: <520A8B67.7040301@redhat.com> Message-ID: Rob, I got past this, as you indicated, by doing that after first running: # ipa-ldap-updater --ldapi ./schema.update Using a schema.update tip file I found in a note from you after some hard core googling. Should that extra step have been necessary? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Tue, Aug 13, 2013 at 3:39 PM, Rob Crittenden wrote: > Bret Wortman wrote: > >> I tried this, but no joy: >> >> # /usr/sbin/ipa-upgradeconfig --debug >> : >> : >> DEBUG: caSignedLogCert.cfg >> >> **> profile >> >> validity range is 720 >> INFO: [Certificate renewal should stop the CA] >> ERROR: Unable to find certmonger request ID for auditSigning Cert >> INFO: The ipa-upgradeconfig command was successful >> # >> > > Run getcert list and sift through the output and see if you have a request > tracking for nickname auditSigningCert cert-pki-ca (or similar). > > But I still can't connect to http://ipamaster/ipa/ui/; I get a 903 error >> every time, and /var/log/httpd/error_log shows, in part: >> >> [Tue Aug 13 13:07:20.786566 2013] [:error] [pid 5890] KeyError: >> 'ipadnszone' >> [Tue Aug 13 13:07:20.786717 2013] [:error] [pid 5890] ipa: INFO: >> bretw at FOO.NET : json_metadata(None, None, >> >> object=u'all'): KeyError >> [Tue Aug 13 13:07:21.001525 2013] [:error] [pid 5890] ipa: INFO: >> bretw at FOO.NET : json_metadata(None, None, >> command=u'all'): SUCCESS >> >> DNS resolution, authentication and authorization all /appear/ to be >> working fine. >> > > The DNS schema was not updated properly. I'd run: > > # ipa-ldap-updater --upgrade > > rob > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Wed Aug 14 13:10:12 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 14 Aug 2013 09:10:12 -0400 Subject: [Freeipa-users] Upgrade failed -- how to recover? In-Reply-To: References: <520A8B67.7040301@redhat.com> Message-ID: <520B81B4.7000409@redhat.com> Bret Wortman wrote: > Rob, I got past this, as you indicated, by doing that after first running: > > # ipa-ldap-updater --ldapi ./schema.update > > Using a schema.update tip file I found in a note from you after some > hard core googling. Should that extra step have been necessary? No, it shouldn't be necessary. Can look in /var/log/ipaupgrade.log (likely humongous) for the original failure and post that section of the log? Updating schema is hard. We are actually completely revamping the way we handle schema changes between version which should be a lot more stable. rob > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Tue, Aug 13, 2013 at 3:39 PM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > I tried this, but no joy: > > # /usr/sbin/ipa-upgradeconfig --debug > : > : > DEBUG: caSignedLogCert.cfg > __> > profile > > validity range is 720 > INFO: [Certificate renewal should stop the CA] > ERROR: Unable to find certmonger request ID for auditSigning Cert > INFO: The ipa-upgradeconfig command was successful > # > > > Run getcert list and sift through the output and see if you have a > request tracking for nickname auditSigningCert cert-pki-ca (or similar). > > But I still can't connect to http://ipamaster/ipa/ui/; I get a > 903 error > every time, and /var/log/httpd/error_log shows, in part: > > [Tue Aug 13 13:07:20.786566 2013] [:error] [pid 5890] KeyError: > 'ipadnszone' > [Tue Aug 13 13:07:20.786717 2013] [:error] [pid 5890] ipa: INFO: > bretw at FOO.NET >: json_metadata(None, None, > > object=u'all'): KeyError > [Tue Aug 13 13:07:21.001525 2013] [:error] [pid 5890] ipa: INFO: > bretw at FOO.NET >: json_metadata(None, None, > command=u'all'): SUCCESS > > DNS resolution, authentication and authorization all /appear/ to be > working fine. > > > The DNS schema was not updated properly. I'd run: > > # ipa-ldap-updater --upgrade > > rob > > From rcritten at redhat.com Wed Aug 14 13:12:51 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 14 Aug 2013 09:12:51 -0400 Subject: [Freeipa-users] Freeipa-users Digest, Vol 61, Issue 21 In-Reply-To: References: Message-ID: <520B8253.3050501@redhat.com> Aissa Brahimi wrote: > Hi, > > I am having this issue: > > IPA server: CentOS6.x > Host CentOS 5.x > > 2 different host and cannot join the IPA server: > > Here the 2 different output I got: > > There was a problem importing one of the required Python modules. The > error was: > > No module named OpenSSL It is a missing dependency in the RHEL 5 client. yum install pyOpenSSL to fix it. > and .. > > > bash: ipa-client-install: command not found You either don't have the ipa-client package installed or /usr/sbin isn't in your path. rob From brian_lee1 at jabil.com Wed Aug 14 13:19:17 2013 From: brian_lee1 at jabil.com (Brian Lee) Date: Wed, 14 Aug 2013 09:19:17 -0400 Subject: [Freeipa-users] Restrict AD users from passwd Message-ID: Hi All, Our current account management policy requires that users change their AD passwords via a special portal, however I've noticed that this can be bypassed by issuing passwd on a Linux system while logged in with AD credentials, thus changing their AD password. Any thoughts on the best way to prevent this action? What I've considered so far is removing the trust in AD, effectively creating a one-way trust, but that would limit functionality for future interoperability. Additionally, we could change the permissions for passwd on each Linux system, but this would be somewhat hackish and also complicated to enforce, since we're waiting on Foreman + Puppet to properly be integrated into Katello for our configuration management solution. Any way to restrict this via the FreeIPA UI? Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Wed Aug 14 13:25:46 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 14 Aug 2013 09:25:46 -0400 Subject: [Freeipa-users] Upgrade failed -- how to recover? In-Reply-To: <520B81B4.7000409@redhat.com> References: <520A8B67.7040301@redhat.com> <520B81B4.7000409@redhat.com> Message-ID: I believe you. I'm not upset at all that things go sideways every now and again. I'm surprised it doesn't happen more. Original failure (or, at least, first occurrence of "ERROR") follows: 2013-08-13T13:56:07Z INFO [Setting up Firefox extension] 2013-08-13T13:56:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2013-08-13T13:56:07Z INFO /usr/share/ipa/html/krb.jsexists, skipping install of Firefox extension 2013-08-13T13:56:07Z INFO [Add missing CA DNS records] 2013-08-13T13:56:07Z ERROR Cannot connect to LDAP to add DNS records: cannot connect to u'ldapi://%2fvar%2frun%2fslapd-SPX-NET.socket': LDAP Server Down 2013-08-13T13:56:07Z INFO [Enabling persistent search in DNS] 2013-08-13T13:56:07Z DEBUG [Saving StateFile to '/var/lib/ipa/sysupgrade/sysupgrade.state' 2013-08-13T13:56:07Z DEBUG Persistent search enabled 2013-08-13T13:56:07Z DEBUG Connections set to 4 Then it goes on for a while, leading to: 2013-08-13T13:56:11Z DEBUG Process finished, return code=1 2013-08-13T13:56:11Z DEBUG stdout=Error connecting to DBus. Please verify that the message bus (D-Bus) service is running. 2013-08-13T13:56:11Z DEBUG stderr= 2013-08-13T13:56:11Z ERROR cretmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-retrieve-agent-submit -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca" -P XXXXXXXX -T auditSigningCert cert-pki-ca' returned non-zero exit status 1 2013-08-13T13:56:11Z DEBUG Starting external process 2013-08-13T13:56:11Z DEBUG args=/usr/bin/certutil -L -d/var/lib/pki-ca/alias -n ocspSigningCert cert-pki-ca 2013-08-13T13:56:11Z DEBUG Process finished, return code=0 Does that help at all? Do you need more? I'm upgrading a slave today and will try just doing the --upgrade (_if_ the automatic upgrade has issues again). * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Wed, Aug 14, 2013 at 9:10 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> Rob, I got past this, as you indicated, by doing that after first running: >> >> # ipa-ldap-updater --ldapi ./schema.update >> >> Using a schema.update tip file I found in a note from you after some >> hard core googling. Should that extra step have been necessary? >> > > No, it shouldn't be necessary. Can look in /var/log/ipaupgrade.log (likely > humongous) for the original failure and post that section of the log? > > Updating schema is hard. We are actually completely revamping the way we > handle schema changes between version which should be a lot more stable. > > rob > > >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Tue, Aug 13, 2013 at 3:39 PM, Rob Crittenden > > wrote: >> >> Bret Wortman wrote: >> >> I tried this, but no joy: >> >> # /usr/sbin/ipa-upgradeconfig --debug >> : >> : >> DEBUG: caSignedLogCert.cfg >> >> >> **>__> >> >> profile >> >> validity range is 720 >> INFO: [Certificate renewal should stop the CA] >> ERROR: Unable to find certmonger request ID for auditSigning Cert >> INFO: The ipa-upgradeconfig command was successful >> # >> >> >> Run getcert list and sift through the output and see if you have a >> request tracking for nickname auditSigningCert cert-pki-ca (or >> similar). >> >> But I still can't connect to http://ipamaster/ipa/ui/; I get a >> 903 error >> every time, and /var/log/httpd/error_log shows, in part: >> >> [Tue Aug 13 13:07:20.786566 2013] [:error] [pid 5890] KeyError: >> 'ipadnszone' >> [Tue Aug 13 13:07:20.786717 2013] [:error] [pid 5890] ipa: INFO: >> bretw at FOO.NET > >> >: json_metadata(None, None, >> >> object=u'all'): KeyError >> [Tue Aug 13 13:07:21.001525 2013] [:error] [pid 5890] ipa: INFO: >> bretw at FOO.NET > >> >: json_metadata(None, None, >> command=u'all'): SUCCESS >> >> DNS resolution, authentication and authorization all /appear/ to >> be >> working fine. >> >> >> The DNS schema was not updated properly. I'd run: >> >> # ipa-ldap-updater --upgrade >> >> rob >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sbose at redhat.com Wed Aug 14 13:37:49 2013 From: sbose at redhat.com (Sumit Bose) Date: Wed, 14 Aug 2013 15:37:49 +0200 Subject: [Freeipa-users] Restrict AD users from passwd In-Reply-To: References: Message-ID: <20130814133748.GA13009@localhost.localdomain> On Wed, Aug 14, 2013 at 09:19:17AM -0400, Brian Lee wrote: > Hi All, > > Our current account management policy requires that users change their AD > passwords via a special portal, however I've noticed that this can be > bypassed by issuing passwd on a Linux system while logged in with AD > credentials, thus changing their AD password. > > Any thoughts on the best way to prevent this action? > > What I've considered so far is removing the trust in AD, effectively > creating a one-way trust, but that would limit functionality for future > interoperability. > > Additionally, we could change the permissions for passwd on each Linux > system, but this would be somewhat hackish and also complicated to enforce, > since we're waiting on Foreman + Puppet to properly be integrated into > Katello for our configuration management solution. > > Any way to restrict this via the FreeIPA UI? I think the only safe way to achieve this is to block port 464 on the AD servers for the Linux hosts. Because basically what passwd is doing here via SSSD is to change the Kerberos password. The same can be done with the kpasswd command, it does not require any privileges the user only needs to know his current password. So even if we add an option to force SSSD to reject password changes for users from trusted domains there are other ways for users to change the password which cannot be controlled by IPA. Please note that changing the AD password with kpasswd would even work without trust. HTH bye, Sumit > > Thanks, > Brian > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users From brian_lee1 at jabil.com Wed Aug 14 13:48:42 2013 From: brian_lee1 at jabil.com (Brian Lee) Date: Wed, 14 Aug 2013 09:48:42 -0400 Subject: [Freeipa-users] Restrict AD users from passwd In-Reply-To: <20130814133748.GA13009@localhost.localdomain> References: <20130814133748.GA13009@localhost.localdomain> Message-ID: Hi Sumit, Thanks for the suggestion. I'll have to give this some thought, since we have 100+ AD servers, this might not be well received by the AD team. If anyone can think of a better mousetrap than this, let me know. Thanks, Brian On Wed, Aug 14, 2013 at 9:37 AM, Sumit Bose wrote: > On Wed, Aug 14, 2013 at 09:19:17AM -0400, Brian Lee wrote: > > Hi All, > > > > Our current account management policy requires that users change their AD > > passwords via a special portal, however I've noticed that this can be > > bypassed by issuing passwd on a Linux system while logged in with AD > > credentials, thus changing their AD password. > > > > Any thoughts on the best way to prevent this action? > > > > What I've considered so far is removing the trust in AD, effectively > > creating a one-way trust, but that would limit functionality for future > > interoperability. > > > > Additionally, we could change the permissions for passwd on each Linux > > system, but this would be somewhat hackish and also complicated to > enforce, > > since we're waiting on Foreman + Puppet to properly be integrated into > > Katello for our configuration management solution. > > > > Any way to restrict this via the FreeIPA UI? > > I think the only safe way to achieve this is to block port 464 on the AD > servers for the Linux hosts. Because basically what passwd is doing here > via SSSD is to change the Kerberos password. The same can be done with > the kpasswd command, it does not require any privileges the user only > needs to know his current password. So even if we add an option to force > SSSD to reject password changes for users from trusted domains there are > other ways for users to change the password which cannot be controlled > by IPA. > > Please note that changing the AD password with kpasswd would even work > without trust. > > HTH > > bye, > Sumit > > > > > Thanks, > > Brian > > > _______________________________________________ > > Freeipa-users mailing list > > Freeipa-users at redhat.com > > https://www.redhat.com/mailman/listinfo/freeipa-users > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simo at redhat.com Wed Aug 14 13:56:17 2013 From: simo at redhat.com (Simo Sorce) Date: Wed, 14 Aug 2013 09:56:17 -0400 Subject: [Freeipa-users] IPA Server UI Behind Proxy In-Reply-To: <20130814062318.GH27194@redhat.com> References: <20130814062318.GH27194@redhat.com> Message-ID: <1376488577.22218.1.camel@willson.li.ssimo.org> On Wed, 2013-08-14 at 09:23 +0300, Alexander Bokovoy wrote: > On Wed, 14 Aug 2013, Andrew Lau wrote: > >Hi, > > > >I've got my FreeIPA setup in an internal infrastructure, but I want to be > >able to have users access the web UI externally. I tweaked the > >ipa-rewrite.conf so it won't redirect me to the FQDN and then tried both a > >nginx reverse proxy and port forwarding, both works if the client manually > >sets the host name of the IPA server eg. ipa01.internaldomain.local in > >their /etc/hosts file. However if the client tries to to use eg. > >ipa.externaldomain.com with the same port forwarding or nginx proxy config, > >it'll silently error. The docs briefly touches on this - but doesn't really > >give much to go on. > > > >Any suggestions? > Couple considerations here. > > First, you may need to play with debug level to see what principal > mod_auth_kerb picks up when negotiation happens. > > Second, using Kerberos authentication requires both sides to own > Kerberos principals for authentication process to go. Browsers select > HTTP/server.fqdn as their target service principal when connection to > the server.fqdn. Your IPA server only has HTTP/ipa01.internaldomain.local > in its keytab in /etc/httpd/conf/ipa.keytab, which means it would only > be able to respond on ipa01.internaldomain.local to Kerberos auth requests. > > A way to fix this is by making HTTP/ipa.externaldomain.com service in > IPA (ipa service-add HTTP/ipa.externaldomain.com) and then use > 'ipa-getkeytab' to fetch the keytab with the key for the service. Next > step would be to merge content of this keytab with > /etc/httpd/conf/ipa.keytab: > > (echo rkt /tmp/external.keytab; echo wkt /etc/httpd/conf/ipa.keytab) |ktutil > > Then restart httpd -- I'm not sure mod_auth_kerb re-reads the keytab on > its change. FWIW please disregard these complicated instructions, just run ipa-getkeytab on the server an pass -k /etc/httpd/conf/ipa.keytab directly. ipa-getkeytab will properly append the fetched keys to the keytab and no further, error prone, manual merging will be necessary. Simo. -- Simo Sorce * Red Hat, Inc * New York From pspacek at redhat.com Wed Aug 14 13:58:10 2013 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 14 Aug 2013 15:58:10 +0200 Subject: [Freeipa-users] Restrict AD users from passwd In-Reply-To: References: <20130814133748.GA13009@localhost.localdomain> Message-ID: <520B8CF2.6090509@redhat.com> On 14.8.2013 15:48, Brian Lee wrote: > Hi Sumit, > > Thanks for the suggestion. I'll have to give this some thought, since we > have 100+ AD servers, this might not be well received by the AD team. If > anyone can think of a better mousetrap than this, let me know. > > Thanks, > Brian > > > > > On Wed, Aug 14, 2013 at 9:37 AM, Sumit Bose wrote: > >> On Wed, Aug 14, 2013 at 09:19:17AM -0400, Brian Lee wrote: >>> Hi All, >>> >>> Our current account management policy requires that users change their AD >>> passwords via a special portal, however I've noticed that this can be >>> bypassed by issuing passwd on a Linux system while logged in with AD >>> credentials, thus changing their AD password. >>> >>> Any thoughts on the best way to prevent this action? >>> >>> What I've considered so far is removing the trust in AD, effectively >>> creating a one-way trust, but that would limit functionality for future >>> interoperability. >>> >>> Additionally, we could change the permissions for passwd on each Linux >>> system, but this would be somewhat hackish and also complicated to >> enforce, >>> since we're waiting on Foreman + Puppet to properly be integrated into >>> Katello for our configuration management solution. >>> >>> Any way to restrict this via the FreeIPA UI? >> >> I think the only safe way to achieve this is to block port 464 on the AD >> servers for the Linux hosts. Because basically what passwd is doing here >> via SSSD is to change the Kerberos password. The same can be done with >> the kpasswd command, it does not require any privileges the user only >> needs to know his current password. So even if we add an option to force >> SSSD to reject password changes for users from trusted domains there are >> other ways for users to change the password which cannot be controlled >> by IPA. >> >> Please note that changing the AD password with kpasswd would even work >> without trust. IMHO the correct approach is to enforce password policy on AD side, otherwise users can use standard Kerberos protocol and do the change anyway (i.e. effectively bypass IPA and your portal completely). AFAIK AD has some checkbox which determines if the user is allowed to change own password or not. The next question is how 'the portal' does the password change and if it will continue to work if you disallow users to change own password on AD side. -- Petr^2 Spacek From simo at redhat.com Wed Aug 14 14:32:01 2013 From: simo at redhat.com (Simo Sorce) Date: Wed, 14 Aug 2013 10:32:01 -0400 Subject: [Freeipa-users] Restrict AD users from passwd In-Reply-To: References: <20130814133748.GA13009@localhost.localdomain> Message-ID: <1376490721.22218.3.camel@willson.li.ssimo.org> On Wed, 2013-08-14 at 09:48 -0400, Brian Lee wrote: > Hi Sumit, > > > Thanks for the suggestion. I'll have to give this some thought, since > we have 100+ AD servers, this might not be well received by the AD > team. If anyone can think of a better mousetrap than this, let me > know. Do you also block the 'net user' command on Windows clients ? It's the same as 'passwd' on Linux clients. I would address the problem by using proper password policies as I (now) see Petr recommended i another email. Simo. -- Simo Sorce * Red Hat, Inc * New York From brian_lee1 at jabil.com Wed Aug 14 14:38:15 2013 From: brian_lee1 at jabil.com (Brian Lee) Date: Wed, 14 Aug 2013 10:38:15 -0400 Subject: [Freeipa-users] Restrict AD users from passwd In-Reply-To: <1376490721.22218.3.camel@willson.li.ssimo.org> References: <20130814133748.GA13009@localhost.localdomain> <1376490721.22218.3.camel@willson.li.ssimo.org> Message-ID: On the AD side, they limit the potential to change the AD password by deploying a modified the msgina.dll. Otherwise, the user still has the ways to throw a wrench in the system, we're just doing our best to limit the opportunity for this action. On Wed, Aug 14, 2013 at 10:32 AM, Simo Sorce wrote: > On Wed, 2013-08-14 at 09:48 -0400, Brian Lee wrote: > > Hi Sumit, > > > > > > Thanks for the suggestion. I'll have to give this some thought, since > > we have 100+ AD servers, this might not be well received by the AD > > team. If anyone can think of a better mousetrap than this, let me > > know. > > Do you also block the 'net user' command on Windows clients ? > It's the same as 'passwd' on Linux clients. > > I would address the problem by using proper password policies as I (now) > see Petr recommended i another email. > > Simo. > > -- > Simo Sorce * Red Hat, Inc * New York > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at lightoze.net Wed Aug 14 23:26:12 2013 From: me at lightoze.net (Vladimir Kulev) Date: Thu, 15 Aug 2013 02:26:12 +0300 Subject: [Freeipa-users] ipa-server-certinstall ruined pki-tomcatd startup Message-ID: Hello, After installing FreeIPA I followed instructions from http://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP to use globally trusted certificates for HTTP/LDAP server interface to secure other systems provisioning. Then it went out that pki-tomcatd is not able to start anymore because of this: | NFO: Deploying web application directory /var/lib/pki/pki-tomcat/webapps/ca | SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback | SSLAuthenticatorWithFallback: Setting container | SSLAuthenticatorWithFallback: Initializing authenticators | SSLAuthenticatorWithFallback: Starting authenticators | 01:48:31,313 DEBUG (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to retrieve ServletContext: expandEntityReferences defaults to true | 01:48:31,320 DEBUG (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to retrieve ServletContext: expandEntityReferences defaults to true | Internal Database Error encountered: Could not connect to LDAP server host ipa.mydomain.com port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1) Meanwhile dirsrv tells me "Peer does not recognize and trust the CA that issued your certificate." I tried to fix trust by adding various certificates with certutil to /etc/dirsrv/slapd/ and /etc/pki/pki-tomcat/alias/, but nothing helped. Does anyone have a suggestion how to fix the situation? -- Best regards, Vladimir Kulev Mobile: +358400369346, +79215554422 Jabber: me at lightoze.net Skype: lightoze -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 15 12:58:57 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 15 Aug 2013 08:58:57 -0400 Subject: [Freeipa-users] ipa-server-certinstall ruined pki-tomcatd startup In-Reply-To: References: Message-ID: <520CD091.8080209@redhat.com> Vladimir Kulev wrote: > Hello, > > After installing FreeIPA I followed instructions from > http://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP to > use globally trusted certificates for HTTP/LDAP server interface to > secure other systems provisioning. What version of IPA? > Then it went out that pki-tomcatd is not able to start anymore because > of this: > | NFO: Deploying web application directory > /var/lib/pki/pki-tomcat/webapps/ca > | SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback > | SSLAuthenticatorWithFallback: Setting container > | SSLAuthenticatorWithFallback: Initializing authenticators > | SSLAuthenticatorWithFallback: Starting authenticators > | 01:48:31,313 DEBUG > (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to > retrieve ServletContext: expandEntityReferences defaults to true > | 01:48:31,320 DEBUG > (org.jboss.resteasy.plugins.providers.DocumentProvider:60) - Unable to > retrieve ServletContext: expandEntityReferences defaults to true > | Internal Database Error encountered: Could not connect to LDAP server > host ipa.mydomain.com port 636 Error > netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1) > > Meanwhile dirsrv tells me "Peer does not recognize and trust the CA that > issued your certificate." > > I tried to fix trust by adding various certificates with certutil > to /etc/dirsrv/slapd/ and /etc/pki/pki-tomcat/alias/, but nothing > helped. Does anyone have a suggestion how to fix the situation? You shouldn't need to change anything on the 389-ds side assuming it trusts its own CA properly. You should just need to add the CA that signed the 389-ds cert to dogtag and restart. What is full certutil command you are using? rob From me at lightoze.net Thu Aug 15 13:24:30 2013 From: me at lightoze.net (Vladimir Kulev) Date: Thu, 15 Aug 2013 16:24:30 +0300 Subject: [Freeipa-users] ipa-server-certinstall ruined pki-tomcatd startup In-Reply-To: <520CD091.8080209@redhat.com> References: <520CD091.8080209@redhat.com> Message-ID: On Thu, Aug 15, 2013 at 3:58 PM, Rob Crittenden wrote: > Vladimir Kulev wrote: > >> Hello, >> >> After installing FreeIPA I followed instructions from >> http://www.freeipa.org/page/**Using_3rd_part_certificates_**for_HTTP/LDAPto >> use globally trusted certificates for HTTP/LDAP server interface to >> secure other systems provisioning. >> > > What version of IPA? > FreeIPA version is 3.2.2-1.fc19, the latest for Fedora 19 > > Then it went out that pki-tomcatd is not able to start anymore because >> of this: >> | NFO: Deploying web application directory >> /var/lib/pki/pki-tomcat/**webapps/ca >> | SSLAuthenticatorWithFallback: Creating SSL authenticator with fallback >> | SSLAuthenticatorWithFallback: Setting container >> | SSLAuthenticatorWithFallback: Initializing authenticators >> | SSLAuthenticatorWithFallback: Starting authenticators >> | 01:48:31,313 DEBUG >> (org.jboss.resteasy.plugins.**providers.DocumentProvider:60) - Unable to >> retrieve ServletContext: expandEntityReferences defaults to true >> | 01:48:31,320 DEBUG >> (org.jboss.resteasy.plugins.**providers.DocumentProvider:60) - Unable to >> retrieve ServletContext: expandEntityReferences defaults to true >> | Internal Database Error encountered: Could not connect to LDAP server >> host ipa.mydomain.com port 636 Error >> >> netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1) >> >> Meanwhile dirsrv tells me "Peer does not recognize and trust the CA that >> issued your certificate." >> >> I tried to fix trust by adding various certificates with certutil >> to /etc/dirsrv/slapd/ and /etc/pki/pki-tomcat/alias/, but nothing >> helped. Does anyone have a suggestion how to fix the situation? >> > > You shouldn't need to change anything on the 389-ds side assuming it > trusts its own CA properly. > > You should just need to add the CA that signed the 389-ds cert to dogtag > and restart. What is full certutil command you are using? Here is a command: certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t "CT,C,C" -n "External CA" -i /root/ca.pem Also I tried to add intermediate CA with the following: certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t ",," -n "External Sub CA" -i /root/sub.pem External CA file is correct, I verified it with "openssl s_client -CAfile /root/ca.pem -connect ipa.mydomain.com:636" -- Best regards, Vladimir Kulev Mobile: +358400369346, +79215554422 Jabber: me at lightoze.net Skype: lightoze -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 15 15:23:24 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 15 Aug 2013 11:23:24 -0400 Subject: [Freeipa-users] ipa-server-certinstall ruined pki-tomcatd startup In-Reply-To: References: <520CD091.8080209@redhat.com> Message-ID: <520CF26C.6030606@redhat.com> Vladimir Kulev wrote: > > On Thu, Aug 15, 2013 at 3:58 PM, Rob Crittenden > wrote: > > Vladimir Kulev wrote: > > Hello, > > After installing FreeIPA I followed instructions from > http://www.freeipa.org/page/__Using_3rd_part_certificates___for_HTTP/LDAP > > to > use globally trusted certificates for HTTP/LDAP server interface to > secure other systems provisioning. > > > What version of IPA? > > > FreeIPA version is 3.2.2-1.fc19, the latest for Fedora 19 > > > Then it went out that pki-tomcatd is not able to start anymore > because > of this: > | NFO: Deploying web application directory > /var/lib/pki/pki-tomcat/__webapps/ca > | SSLAuthenticatorWithFallback: Creating SSL authenticator with > fallback > | SSLAuthenticatorWithFallback: Setting container > | SSLAuthenticatorWithFallback: Initializing authenticators > | SSLAuthenticatorWithFallback: Starting authenticators > | 01:48:31,313 DEBUG > (org.jboss.resteasy.plugins.__providers.DocumentProvider:60) - > Unable to > retrieve ServletContext: expandEntityReferences defaults to true > | 01:48:31,320 DEBUG > (org.jboss.resteasy.plugins.__providers.DocumentProvider:60) - > Unable to > retrieve ServletContext: expandEntityReferences defaults to true > | Internal Database Error encountered: Could not connect to LDAP > server > host ipa.mydomain.com > port 636 Error > > netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1) > > Meanwhile dirsrv tells me "Peer does not recognize and trust the > CA that > issued your certificate." > > I tried to fix trust by adding various certificates with certutil > to /etc/dirsrv/slapd/ and /etc/pki/pki-tomcat/alias/, but nothing > helped. Does anyone have a suggestion how to fix the situation? > > > You shouldn't need to change anything on the 389-ds side assuming it > trusts its own CA properly. > > You should just need to add the CA that signed the 389-ds cert to > dogtag and restart. What is full certutil command you are using? > > > Here is a command: > certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t "CT,C,C" -n "External > CA" -i /root/ca.pem > > Also I tried to add intermediate CA with the following: > certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t ",," -n "External Sub > CA" -i /root/sub.pem > > External CA file is correct, I verified it with "openssl s_client > -CAfile /root/ca.pem -connect ipa.mydomain.com:636 > " You should drop the sql prefix. This is creating a new cert and key database (you'll see a new cert9 and key4.db there). I don't believe that dogtag uses the sql prefix yet so it won't see the new certs you added. You should also set the trust flags on all intermediate certs as well. rob From me at lightoze.net Thu Aug 15 15:43:19 2013 From: me at lightoze.net (Vladimir Kulev) Date: Thu, 15 Aug 2013 18:43:19 +0300 Subject: [Freeipa-users] ipa-server-certinstall ruined pki-tomcatd startup In-Reply-To: <520CF26C.6030606@redhat.com> References: <520CD091.8080209@redhat.com> <520CF26C.6030606@redhat.com> Message-ID: On Thu, Aug 15, 2013 at 6:23 PM, Rob Crittenden wrote: > Here is a command: >> certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t "CT,C,C" -n "External >> CA" -i /root/ca.pem >> >> Also I tried to add intermediate CA with the following: >> certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t ",," -n "External Sub >> CA" -i /root/sub.pem >> >> External CA file is correct, I verified it with "openssl s_client >> -CAfile /root/ca.pem -connect ipa.mydomain.com:636 >> " >> > > You should drop the sql prefix. This is creating a new cert and key > database (you'll see a new cert9 and key4.db there). I don't believe that > dogtag uses the sql prefix yet so it won't see the new certs you added. > > You should also set the trust flags on all intermediate certs as well. You are right, lsof shows that java process opens only cert8.db and key3.db I did as you say, and dirsrv log output changed to "Netscape Portable Runtime error -8179 (Peer's Certificate issuer is not recognized.); unauthenticated client" Then I in addition ran this command: certutil -d /etc/dirsrv/slapd-MYDOMAIN-COM/ -A -t "CT,C,C" -n "IPA CA" -i /etc/ipa/ca.crt And eventually it worked! So there were two problems: 1) ipa-server-certinstall removed IPA CA from dirsrv nssdb (by replacing it) 2) ipa-server-certinstall did not add new dirsrv CA into pki-tomcatd nssdb Hope you can fix that either in documentation or tools :) -- Best regards, Vladimir Kulev Mobile: +358400369346, +79215554422 Jabber: me at lightoze.net Skype: lightoze -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 15 21:58:20 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 15 Aug 2013 17:58:20 -0400 Subject: [Freeipa-users] ipa-server-certinstall ruined pki-tomcatd startup In-Reply-To: References: <520CD091.8080209@redhat.com> <520CF26C.6030606@redhat.com> Message-ID: <520D4EFC.40804@redhat.com> Vladimir Kulev wrote: > > On Thu, Aug 15, 2013 at 6:23 PM, Rob Crittenden > wrote: > > Here is a command: > certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t "CT,C,C" -n > "External > CA" -i /root/ca.pem > > Also I tried to add intermediate CA with the following: > certutil -d sql:/etc/pki/pki-tomcat/alias/ -A -t ",," -n > "External Sub > CA" -i /root/sub.pem > > External CA file is correct, I verified it with "openssl s_client > -CAfile /root/ca.pem -connect ipa.mydomain.com:636 > > " > > > You should drop the sql prefix. This is creating a new cert and key > database (you'll see a new cert9 and key4.db there). I don't believe > that dogtag uses the sql prefix yet so it won't see the new certs > you added. > > You should also set the trust flags on all intermediate certs as well. > > > You are right, lsof shows that java process opens only cert8.db and key3.db > I did as you say, and dirsrv log output changed to "Netscape Portable > Runtime error -8179 (Peer's Certificate issuer is not recognized.); > unauthenticated client" > > Then I in addition ran this command: > certutil -d /etc/dirsrv/slapd-MYDOMAIN-COM/ -A -t "CT,C,C" -n "IPA CA" > -i /etc/ipa/ca.crt > > And eventually it worked! > > So there were two problems: > 1) ipa-server-certinstall removed IPA CA from dirsrv nssdb (by replacing it) > 2) ipa-server-certinstall did not add new dirsrv CA into pki-tomcatd nssdb > > Hope you can fix that either in documentation or tools :) https://fedorahosted.org/freeipa/ticket/3862 rob From aissa.brahimi at itsoninc.com Thu Aug 15 23:48:19 2013 From: aissa.brahimi at itsoninc.com (Aissa Brahimi) Date: Thu, 15 Aug 2013 16:48:19 -0700 Subject: [Freeipa-users] Freeipa-users Digest, Vol 61, Issue 26 In-Reply-To: References: Message-ID: Hi, I was able to install el5 ipa-client Schema: ipa server: centos 6.x client ipa: centos 5.x following this steps: https://www.redhat.com/archives/freeipa-users/2009-January/msg00021.html next challenge: implemente SUDO rule.. On Wed, Aug 14, 2013 at 9:00 AM, wrote: > Send Freeipa-users mailing list submissions to > freeipa-users at redhat.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.redhat.com/mailman/listinfo/freeipa-users > or, via email, send a message with subject or body 'help' to > freeipa-users-request at redhat.com > > You can reach the person managing the list at > freeipa-users-owner at redhat.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Freeipa-users digest..." > > > Today's Topics: > > 1. Re: Restrict AD users from passwd (Petr Spacek) > 2. Re: Restrict AD users from passwd (Simo Sorce) > 3. Re: Restrict AD users from passwd (Brian Lee) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 14 Aug 2013 15:58:10 +0200 > From: Petr Spacek > To: freeipa-users at redhat.com > Subject: Re: [Freeipa-users] Restrict AD users from passwd > Message-ID: <520B8CF2.6090509 at redhat.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 14.8.2013 15:48, Brian Lee wrote: > > Hi Sumit, > > > > Thanks for the suggestion. I'll have to give this some thought, since we > > have 100+ AD servers, this might not be well received by the AD team. If > > anyone can think of a better mousetrap than this, let me know. > > > > Thanks, > > Brian > > > > > > > > > > On Wed, Aug 14, 2013 at 9:37 AM, Sumit Bose wrote: > > > >> On Wed, Aug 14, 2013 at 09:19:17AM -0400, Brian Lee wrote: > >>> Hi All, > >>> > >>> Our current account management policy requires that users change their > AD > >>> passwords via a special portal, however I've noticed that this can be > >>> bypassed by issuing passwd on a Linux system while logged in with AD > >>> credentials, thus changing their AD password. > >>> > >>> Any thoughts on the best way to prevent this action? > >>> > >>> What I've considered so far is removing the trust in AD, effectively > >>> creating a one-way trust, but that would limit functionality for future > >>> interoperability. > >>> > >>> Additionally, we could change the permissions for passwd on each Linux > >>> system, but this would be somewhat hackish and also complicated to > >> enforce, > >>> since we're waiting on Foreman + Puppet to properly be integrated into > >>> Katello for our configuration management solution. > >>> > >>> Any way to restrict this via the FreeIPA UI? > >> > >> I think the only safe way to achieve this is to block port 464 on the AD > >> servers for the Linux hosts. Because basically what passwd is doing here > >> via SSSD is to change the Kerberos password. The same can be done with > >> the kpasswd command, it does not require any privileges the user only > >> needs to know his current password. So even if we add an option to force > >> SSSD to reject password changes for users from trusted domains there are > >> other ways for users to change the password which cannot be controlled > >> by IPA. > >> > >> Please note that changing the AD password with kpasswd would even work > >> without trust. > > IMHO the correct approach is to enforce password policy on AD side, > otherwise > users can use standard Kerberos protocol and do the change anyway (i.e. > effectively bypass IPA and your portal completely). > > AFAIK AD has some checkbox which determines if the user is allowed to > change > own password or not. > > The next question is how 'the portal' does the password change and if it > will > continue to work if you disallow users to change own password on AD side. > > -- > Petr^2 Spacek > > > > ------------------------------ > > Message: 2 > Date: Wed, 14 Aug 2013 10:32:01 -0400 > From: Simo Sorce > To: Brian Lee > Cc: freeipa-users at redhat.com > Subject: Re: [Freeipa-users] Restrict AD users from passwd > Message-ID: <1376490721.22218.3.camel at willson.li.ssimo.org> > Content-Type: text/plain; charset="UTF-8" > > On Wed, 2013-08-14 at 09:48 -0400, Brian Lee wrote: > > Hi Sumit, > > > > > > Thanks for the suggestion. I'll have to give this some thought, since > > we have 100+ AD servers, this might not be well received by the AD > > team. If anyone can think of a better mousetrap than this, let me > > know. > > Do you also block the 'net user' command on Windows clients ? > It's the same as 'passwd' on Linux clients. > > I would address the problem by using proper password policies as I (now) > see Petr recommended i another email. > > Simo. > > -- > Simo Sorce * Red Hat, Inc * New York > > > > ------------------------------ > > Message: 3 > Date: Wed, 14 Aug 2013 10:38:15 -0400 > From: Brian Lee > To: Simo Sorce > Cc: freeipa-users at redhat.com > Subject: Re: [Freeipa-users] Restrict AD users from passwd > Message-ID: > kUgrcvWQkyOO6Hw at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On the AD side, they limit the potential to change the AD password by > deploying a modified the msgina.dll. Otherwise, the user still has the ways > to throw a wrench in the system, we're just doing our best to limit the > opportunity for this action. > > > On Wed, Aug 14, 2013 at 10:32 AM, Simo Sorce wrote: > > > On Wed, 2013-08-14 at 09:48 -0400, Brian Lee wrote: > > > Hi Sumit, > > > > > > > > > Thanks for the suggestion. I'll have to give this some thought, since > > > we have 100+ AD servers, this might not be well received by the AD > > > team. If anyone can think of a better mousetrap than this, let me > > > know. > > > > Do you also block the 'net user' command on Windows clients ? > > It's the same as 'passwd' on Linux clients. > > > > I would address the problem by using proper password policies as I (now) > > see Petr recommended i another email. > > > > Simo. > > > > -- > > Simo Sorce * Red Hat, Inc * New York > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://www.redhat.com/archives/freeipa-users/attachments/20130814/154e7426/attachment.html > > > > ------------------------------ > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > > End of Freeipa-users Digest, Vol 61, Issue 26 > ********************************************* > -- Aissa Brahimi IT Admin - Support ItsOn, Inc. itsoninc.com mobile: 408.858.0304 -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.raehalme at codecenter.fi Mon Aug 19 13:05:40 2013 From: thomas.raehalme at codecenter.fi (Thomas Raehalme) Date: Mon, 19 Aug 2013 16:05:40 +0300 Subject: [Freeipa-users] Using subdomains (or dots) in hostnames Message-ID: Hi! We are in the process of deploying FreeIPA in our virtual environment. So far things are working smoothly and I am really impressed by the solution! One question has risen as we have added our first clients to the system. Because the total number of clients is 50 and going up, we have divided our servers to subdomains depending on the purpose of the server, ie. test servers in one subdomain, internal services on another and so on. There is, however, no need for each subdomain to have its own IPA server. Let's say we're using domain example.com. Adding clients a.example.com and b.example.com was smooth. Adding client a.sub1.example.com also had no problems until I tried to get sudoers from the IPA server (using SSSD and LDAP as suggested). The client fails to find any users matching the server name. Because the only difference compared to a fully functional server is the dot in the host name, that's probably the reason why no sudoers are found for the server in the subdomain? For IPA master I am using CentOS 6.4 and ipa-server-3.0.0-26.el6_4.4.x86_64. The clients are also CentOS 6.4 with ipa-client-3.0.0-26.el6_4.4.x86_64. Any help is appreciated! Please let me know if providing any piece of information helps. Best regards, Thomas From bret.wortman at damascusgrp.com Mon Aug 19 14:16:06 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 10:16:06 -0400 Subject: [Freeipa-users] Replication woes Message-ID: My replication situation has gotten a bit messed up. I have four replicas that are up and running and two that I'm trying to delete (one is not a replica any more, one didn't upgrade well during its fedup upgrade from F17->F18 and as such I had to do a clean OS install). # ipa-replica-manage list bad1.foo.net : master bad2.foo.net : master good1.foo.net: master good2.foo.net: master good3.foo.net: master good4.foo.net: master # ipa-replica-manage list ipamaster.foo.net good1.foo.net: replica good2.foo.net: replica good3.foo.net: replica good4.foo.net: replica # ipa-replica-manage del --force bad1.foo.net 'ipamaster.foo.net' has no replication agreement for 'bad1.foo.net' # ipa-replica-manage del --force bad2.foo.net 'ipamaster.foo.net' has no replication agreement for 'bad2.foo.net' # * * What I need to do is remove bad1 completely and then remove bad2 and re-add it as a replica. Any ideas? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Aug 19 14:26:14 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 19 Aug 2013 10:26:14 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: Message-ID: <52122B06.80203@redhat.com> Bret Wortman wrote: > My replication situation has gotten a bit messed up. > > I have four replicas that are up and running and two that I'm trying to > delete (one is not a replica any more, one didn't upgrade well during > its fedup upgrade from F17->F18 and as such I had to do a clean OS install). > > # ipa-replica-manage list > bad1.foo.net: master > bad2.foo.net: master > good1.foo.net : master > good2.foo.net : master > good3.foo.net : master > good4.foo.net : master > # ipa-replica-manage list ipamaster.foo.net > good1.foo.net : replica > good2.foo.net : replica > good3.foo.net : replica > good4.foo.net : replica > # ipa-replica-manage del --force bad1.foo.net > 'ipamaster.foo.net ' has no replication > agreement for 'bad1.foo.net ' > # ipa-replica-manage del --force bad2.foo.net > 'ipamaster.foo.net ' has no replication > agreement for 'bad2.foo.net ' > # > _ > _ > What I need to do is remove bad1 completely and then remove bad2 and > re-add it as a replica. Any ideas? I guess I'd start on bad1 and see what replication agreements it thinks it has. It is worth it to double-check on all the good hosts too, just to be sure that nobody has an agreement. Assuming it has no agreements, add the --cleanup flag to the del command. This will prompt you to erase the replica as a master. We have lots of warnings because this can be a pretty dangerous command. Once removed you can safely uninstall the replica and re-install if you'd like. rob From bret.wortman at damascusgrp.com Mon Aug 19 14:32:37 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 10:32:37 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: <52122B06.80203@redhat.com> References: <52122B06.80203@redhat.com> Message-ID: The software is actually gone from both boxes -- one is dead and the other was reinstalled when the upgrade failed. So I can't get at the database for either one. Safe to just --cleanup in that case? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 10:26 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> My replication situation has gotten a bit messed up. >> >> I have four replicas that are up and running and two that I'm trying to >> delete (one is not a replica any more, one didn't upgrade well during >> its fedup upgrade from F17->F18 and as such I had to do a clean OS >> install). >> >> # ipa-replica-manage list >> bad1.foo.net: master >> bad2.foo.net: master >> good1.foo.net : master >> good2.foo.net : master >> good3.foo.net : master >> good4.foo.net : master >> # ipa-replica-manage list ipamaster.foo.net >> good1.foo.net : replica >> good2.foo.net : replica >> good3.foo.net : replica >> good4.foo.net : replica >> # ipa-replica-manage del --force bad1.foo.net >> 'ipamaster.foo.net ' has no replication >> agreement for 'bad1.foo.net ' >> # ipa-replica-manage del --force bad2.foo.net >> 'ipamaster.foo.net ' has no replication >> agreement for 'bad2.foo.net ' >> # >> >> _ >> _ >> What I need to do is remove bad1 completely and then remove bad2 and >> re-add it as a replica. Any ideas? >> > > I guess I'd start on bad1 and see what replication agreements it thinks it > has. It is worth it to double-check on all the good hosts too, just to be > sure that nobody has an agreement. > > Assuming it has no agreements, add the --cleanup flag to the del command. > This will prompt you to erase the replica as a master. We have lots of > warnings because this can be a pretty dangerous command. > > Once removed you can safely uninstall the replica and re-install if you'd > like. > > rob > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Aug 19 14:35:24 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 19 Aug 2013 10:35:24 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52122B06.80203@redhat.com> Message-ID: <52122D2C.4040801@redhat.com> Bret Wortman wrote: > The software is actually gone from both boxes -- one is dead and the > other was reinstalled when the upgrade failed. So I can't get at the > database for either one. Safe to just --cleanup in that case? > Assuming that none of the good servers have an agreement with bad* then yes, safe to use. rob > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 10:26 AM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > My replication situation has gotten a bit messed up. > > I have four replicas that are up and running and two that I'm > trying to > delete (one is not a replica any more, one didn't upgrade well > during > its fedup upgrade from F17->F18 and as such I had to do a clean > OS install). > > # ipa-replica-manage list > bad1.foo.net : master > bad2.foo.net : master > good1.foo.net : master > good2.foo.net : master > good3.foo.net : master > good4.foo.net : master > # ipa-replica-manage list ipamaster.foo.net > > good1.foo.net : replica > good2.foo.net : replica > good3.foo.net : replica > good4.foo.net : replica > # ipa-replica-manage del --force bad1.foo.net > > 'ipamaster.foo.net > ' has no replication > agreement for 'bad1.foo.net > ' > # ipa-replica-manage del --force bad2.foo.net > > 'ipamaster.foo.net > ' has no replication > agreement for 'bad2.foo.net > ' > # > > _ > _ > What I need to do is remove bad1 completely and then remove bad2 and > re-add it as a replica. Any ideas? > > > I guess I'd start on bad1 and see what replication agreements it > thinks it has. It is worth it to double-check on all the good hosts > too, just to be sure that nobody has an agreement. > > Assuming it has no agreements, add the --cleanup flag to the del > command. This will prompt you to erase the replica as a master. We > have lots of warnings because this can be a pretty dangerous command. > > Once removed you can safely uninstall the replica and re-install if > you'd like. > > rob > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From bret.wortman at damascusgrp.com Mon Aug 19 14:37:23 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 10:37:23 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: <52122D2C.4040801@redhat.com> References: <52122B06.80203@redhat.com> <52122D2C.4040801@redhat.com> Message-ID: Not according to my poll of the good ones, so here goes. Thanks, Rob. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 10:35 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> The software is actually gone from both boxes -- one is dead and the >> other was reinstalled when the upgrade failed. So I can't get at the >> database for either one. Safe to just --cleanup in that case? >> >> > Assuming that none of the good servers have an agreement with bad* then > yes, safe to use. > > rob > > >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 10:26 AM, Rob Crittenden > > wrote: >> >> Bret Wortman wrote: >> >> My replication situation has gotten a bit messed up. >> >> I have four replicas that are up and running and two that I'm >> trying to >> delete (one is not a replica any more, one didn't upgrade well >> during >> its fedup upgrade from F17->F18 and as such I had to do a clean >> OS install). >> >> # ipa-replica-manage list >> bad1.foo.net : master >> bad2.foo.net : master >> good1.foo.net : >> master >> good2.foo.net : >> master >> good3.foo.net : >> master >> good4.foo.net : >> master >> # ipa-replica-manage list ipamaster.foo.net >> >> good1.foo.net : >> replica >> good2.foo.net : >> replica >> good3.foo.net : >> replica >> good4.foo.net : >> replica >> >> # ipa-replica-manage del --force bad1.foo.net >> >> >> 'ipamaster.foo.net >> ' has no replication >> agreement for 'bad1.foo.net >> ' >> # ipa-replica-manage del --force bad2.foo.net >> >> >> 'ipamaster.foo.net >> ' has no replication >> agreement for 'bad2.foo.net >> ' >> # >> >> _ >> _ >> What I need to do is remove bad1 completely and then remove bad2 >> and >> re-add it as a replica. Any ideas? >> >> >> I guess I'd start on bad1 and see what replication agreements it >> thinks it has. It is worth it to double-check on all the good hosts >> too, just to be sure that nobody has an agreement. >> >> Assuming it has no agreements, add the --cleanup flag to the del >> command. This will prompt you to erase the replica as a master. We >> have lots of warnings because this can be a pretty dangerous command. >> >> Once removed you can safely uninstall the replica and re-install if >> you'd like. >> >> rob >> >> >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Mon Aug 19 14:52:20 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 10:52:20 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: Message-ID: How can I tell if this is working? It's been 10 minutes and it hasn't returned; IPA response is sluggish and top doesn't show anything obviously running & sucking up CPU. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 10:16 AM, Bret Wortman wrote: > My replication situation has gotten a bit messed up. > > I have four replicas that are up and running and two that I'm trying to > delete (one is not a replica any more, one didn't upgrade well during its > fedup upgrade from F17->F18 and as such I had to do a clean OS install). > > # ipa-replica-manage list > bad1.foo.net : > master > bad2.foo.net : > master > good1.foo.net: master > good2.foo.net: master > good3.foo.net: master > good4.foo.net: master > # ipa-replica-manage list ipamaster.foo.net > good1.foo.net: replica > good2.foo.net: replica > good3.foo.net: replica > good4.foo.net: replica > # ipa-replica-manage del --force bad1.foo.net > 'ipamaster.foo.net' has no replication agreement for 'bad1.foo.net' > # ipa-replica-manage del --force bad2.foo.net > 'ipamaster.foo.net' has no replication agreement for 'bad2.foo.net' > # > * > * > What I need to do is remove bad1 completely and then remove bad2 and > re-add it as a replica. Any ideas? > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Aug 19 15:11:01 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 19 Aug 2013 11:11:01 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: Message-ID: <52123585.2070503@redhat.com> Bret Wortman wrote: > How can I tell if this is working? It's been 10 minutes and it hasn't > returned; IPA response is sluggish and top doesn't show anything > obviously running & sucking up CPU. It should be nearly instantaneous. It doesn't actually do a lot. It deletes the master from cn=masters, removes its entries from S4U2proxy delegation and in newer versions attempts to save its DNA configuration, if any. It should be safe to break out of it and re-run it. You may want to check the 389-ds logs to see what it has already done. rob > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 10:16 AM, Bret Wortman > > wrote: > > My replication situation has gotten a bit messed up. > > I have four replicas that are up and running and two that I'm trying > to delete (one is not a replica any more, one didn't upgrade well > during its fedup upgrade from F17->F18 and as such I had to do a > clean OS install). > > # ipa-replica-manage list > bad1.foo.net : > master > bad2.foo.net : > master > good1.foo.net : master > good2.foo.net : master > good3.foo.net : master > good4.foo.net : master > # ipa-replica-manage list ipamaster.foo.net > > good1.foo.net : replica > good2.foo.net : replica > good3.foo.net : replica > good4.foo.net : replica > # ipa-replica-manage del --force bad1.foo.net > 'ipamaster.foo.net ' has no replication > agreement for 'bad1.foo.net ' > # ipa-replica-manage del --force bad2.foo.net > 'ipamaster.foo.net ' has no replication > agreement for 'bad2.foo.net ' > # > _ > _ > What I need to do is remove bad1 completely and then remove bad2 and > re-add it as a replica. Any ideas? > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From bret.wortman at damascusgrp.com Mon Aug 19 15:27:10 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 11:27:10 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: <52123585.2070503@redhat.com> References: <52123585.2070503@redhat.com> Message-ID: Well, my master ground to a halt and wasn't responding. I rebooted the system and now I can't access the web UI or ssh to the master either. I have console access but that's it. The services all say they're running, but the web UI gives an "Unknown Error" dialog and ssh fails with "ssh_exchange_identification: Connection closed by remote host" whenever I try to ssh to ipamaster. I think something has gone really wrong inside my master. Any ideas? Even after the reboot, --cleanup isn't helping and just hangs. The logfiles end (as of the time I ^C'd the process) with: NSMMReplicationPlugin - agmt="cn=meTogood3.spx.net" (good3:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot determine realm for numeric host address)) NSMMReplicationPlugin - CleanAllRUV Task: Replica not online (agmt="cn= meTogood3.foo.net" (good3:389)) NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas online, retrying in 160 seconds..., So it looks like it's having trouble talking with one of my replicas and is doggedly trying to get the job done. Any idea how to get the master back working again while I troubleshoot this connectivity issue? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 11:11 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> How can I tell if this is working? It's been 10 minutes and it hasn't >> returned; IPA response is sluggish and top doesn't show anything >> obviously running & sucking up CPU. >> > > It should be nearly instantaneous. It doesn't actually do a lot. It > deletes the master from cn=masters, removes its entries from S4U2proxy > delegation and in newer versions attempts to save its DNA configuration, if > any. > > It should be safe to break out of it and re-run it. You may want to check > the 389-ds logs to see what it has already done. > > rob > > >> >> _ >> _ >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 10:16 AM, Bret Wortman >> >> >> wrote: >> >> My replication situation has gotten a bit messed up. >> >> I have four replicas that are up and running and two that I'm trying >> to delete (one is not a replica any more, one didn't upgrade well >> during its fedup upgrade from F17->F18 and as such I had to do a >> clean OS install). >> >> # ipa-replica-manage list >> bad1.foo.net >> >: >> master >> bad2.foo.net >> >: >> master >> good1.foo.net : master >> good2.foo.net : master >> good3.foo.net : master >> good4.foo.net : master >> # ipa-replica-manage list ipamaster.foo.net >> >> > >> good1.foo.net : replica >> good2.foo.net : replica >> good3.foo.net : replica >> good4.foo.net : replica >> # ipa-replica-manage del --force bad1.foo.net >> 'ipamaster.foo.net ' has no replication >> agreement for 'bad1.foo.net ' >> # ipa-replica-manage del --force bad2.foo.net >> 'ipamaster.foo.net ' has no replication >> agreement for 'bad2.foo.net ' >> # >> _ >> _ >> >> What I need to do is remove bad1 completely and then remove bad2 and >> re-add it as a replica. Any ideas? >> >> _ >> _ >> *Bret Wortman* >> >> http://damascusgrp.com/ >> >> > >> http://about.me/wortmanbret >> >> > >> >> >> >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Aug 19 15:29:36 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 19 Aug 2013 11:29:36 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> Message-ID: <521239E0.60309@redhat.com> Bret Wortman wrote: > Well, my master ground to a halt and wasn't responding. I rebooted the > system and now I can't access the web UI or ssh to the master either. I > have console access but that's it. > > The services all say they're running, but the web UI gives an "Unknown > Error" dialog and ssh fails with "ssh_exchange_identification: > Connection closed by remote host" whenever I try to ssh to ipamaster. I > think something has gone really wrong inside my master. Any ideas? Even > after the reboot, --cleanup isn't helping and just hangs. > > The logfiles end (as of the time I ^C'd the process) with: > > NSMMReplicationPlugin - agmt="cn=meTogood3.spx.net > " (good3:389): Replication bind with GSSAPI > auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: > GSSAPI Error: Unspecified GSS failure. Minor code may provide more > information (Cannot determine realm for numeric host address)) > NSMMReplicationPlugin - CleanAllRUV Task: Replica not online > (agmt="cn=meTogood3.foo.net " (good3:389)) > NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas online, > retrying in 160 seconds..., > > So it looks like it's having trouble talking with one of my replicas and > is doggedly trying to get the job done. Any idea how to get the master > back working again while I troubleshoot this connectivity issue? That suggests a DNS problem, and it might explain ssh as well depending on your configuration. rob > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 11:11 AM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > How can I tell if this is working? It's been 10 minutes and it > hasn't > returned; IPA response is sluggish and top doesn't show anything > obviously running & sucking up CPU. > > > It should be nearly instantaneous. It doesn't actually do a lot. It > deletes the master from cn=masters, removes its entries from > S4U2proxy delegation and in newer versions attempts to save its DNA > configuration, if any. > > It should be safe to break out of it and re-run it. You may want to > check the 389-ds logs to see what it has already done. > > rob > > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 10:16 AM, Bret Wortman > > >> wrote: > > My replication situation has gotten a bit messed up. > > I have four replicas that are up and running and two that > I'm trying > to delete (one is not a replica any more, one didn't > upgrade well > during its fedup upgrade from F17->F18 and as such I had to > do a > clean OS install). > > # ipa-replica-manage list > bad1.foo.net > >: > master > bad2.foo.net > >: > master > good1.foo.net : master > good2.foo.net : master > good3.foo.net : master > good4.foo.net : master > # ipa-replica-manage list ipamaster.foo.net > > > > good1.foo.net : replica > good2.foo.net : replica > good3.foo.net : replica > good4.foo.net : replica > # ipa-replica-manage del --force bad1.foo.net > > 'ipamaster.foo.net > ' has no replication > agreement for 'bad1.foo.net > ' > # ipa-replica-manage del --force bad2.foo.net > > 'ipamaster.foo.net > ' has no replication > agreement for 'bad2.foo.net > ' > # > _ > _ > > What I need to do is remove bad1 completely and then remove > bad2 and > re-add it as a replica. Any ideas? > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > > > http://about.me/wortmanbret > > > > > > > > _________________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/__mailman/listinfo/freeipa-users > > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From rcritten at redhat.com Mon Aug 19 15:58:28 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 19 Aug 2013 11:58:28 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: <521239E0.60309@redhat.com> References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> Message-ID: <521240A4.8040704@redhat.com> Rob Crittenden wrote: > Bret Wortman wrote: >> Well, my master ground to a halt and wasn't responding. I rebooted the >> system and now I can't access the web UI or ssh to the master either. I >> have console access but that's it. >> >> The services all say they're running, but the web UI gives an "Unknown >> Error" dialog and ssh fails with "ssh_exchange_identification: >> Connection closed by remote host" whenever I try to ssh to ipamaster. I >> think something has gone really wrong inside my master. Any ideas? Even >> after the reboot, --cleanup isn't helping and just hangs. >> >> The logfiles end (as of the time I ^C'd the process) with: >> >> NSMMReplicationPlugin - agmt="cn=meTogood3.spx.net >> " (good3:389): Replication bind with GSSAPI >> auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: >> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Cannot determine realm for numeric host address)) >> NSMMReplicationPlugin - CleanAllRUV Task: Replica not online >> (agmt="cn=meTogood3.foo.net " (good3:389)) >> NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas online, >> retrying in 160 seconds..., >> >> So it looks like it's having trouble talking with one of my replicas and >> is doggedly trying to get the job done. Any idea how to get the master >> back working again while I troubleshoot this connectivity issue? > > That suggests a DNS problem, and it might explain ssh as well depending > on your configuration. To be clear, you ran --cleanup against one of the bad masters, not a good one, right? rob From bret.wortman at damascusgrp.com Mon Aug 19 16:18:24 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 12:18:24 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> Message-ID: Digging further, I think this log entry might be the problem between the two servers that aren't talking: slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id[] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server ldap/localhost at SPX.NET not found in Kerberos database)) errno 2 (No such file or directory) Did I build something incorrectly when that server was set up originally? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman wrote: > I ran it on a good master, against a bad one. As in, I ran this command on > my master IPA node: > > # ipa-replica-manage del --force bad1.foo.net --cleanup > > Was that wrong? I was trying to delete the bad replica from the master, so > I figured the command needed to be run on the master. But again, my master > is now in a state where it's not resolving DNS, user logins, or sudo at the > very least. > > Oh, and I checked the node that it was complaining about earlier. The > network connection to it is the pits, but it's there. And it resolves. > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 11:58 AM, Rob Crittenden wrote: > >> Rob Crittenden wrote: >> >>> Bret Wortman wrote: >>> >>>> Well, my master ground to a halt and wasn't responding. I rebooted the >>>> system and now I can't access the web UI or ssh to the master either. I >>>> have console access but that's it. >>>> >>>> The services all say they're running, but the web UI gives an "Unknown >>>> Error" dialog and ssh fails with "ssh_exchange_identification: >>>> Connection closed by remote host" whenever I try to ssh to ipamaster. I >>>> think something has gone really wrong inside my master. Any ideas? Even >>>> after the reboot, --cleanup isn't helping and just hangs. >>>> >>>> The logfiles end (as of the time I ^C'd the process) with: >>>> >>>> NSMMReplicationPlugin - agmt="cn=meTogood3.spx.net >>>> " (good3:389): Replication bind with GSSAPI >>>> auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: >>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >>>> information (Cannot determine realm for numeric host address)) >>>> NSMMReplicationPlugin - CleanAllRUV Task: Replica not online >>>> (agmt="cn=meTogood3.foo.net " (good3:389)) >>>> NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas online, >>>> retrying in 160 seconds..., >>>> >>>> So it looks like it's having trouble talking with one of my replicas and >>>> is doggedly trying to get the job done. Any idea how to get the master >>>> back working again while I troubleshoot this connectivity issue? >>>> >>> >>> That suggests a DNS problem, and it might explain ssh as well depending >>> on your configuration. >>> >> >> To be clear, you ran --cleanup against one of the bad masters, not a good >> one, right? >> >> rob >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Mon Aug 19 16:19:39 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 12:19:39 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> Message-ID: ...and I got the web UI, authentication and sudo back via: # ipactl stop # ipactl start Not sure why that worked, but it did. I was grasping at straws, honestly. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman wrote: > Digging further, I think this log entry might be the problem between the > two servers that aren't talking: > > slapd_ldap_sasl_interactive_bind - Error: could not perform interactive > bind for id[] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic > failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more > information (Server ldap/localhost at SPX.NET not found in Kerberos > database)) errno 2 (No such file or directory) > > Did I build something incorrectly when that server was set up originally? > > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman < > bret.wortman at damascusgrp.com> wrote: > >> I ran it on a good master, against a bad one. As in, I ran this command >> on my master IPA node: >> >> # ipa-replica-manage del --force bad1.foo.net --cleanup >> >> Was that wrong? I was trying to delete the bad replica from the master, >> so I figured the command needed to be run on the master. But again, my >> master is now in a state where it's not resolving DNS, user logins, or sudo >> at the very least. >> >> Oh, and I checked the node that it was complaining about earlier. The >> network connection to it is the pits, but it's there. And it resolves. >> >> >> * >> * >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 11:58 AM, Rob Crittenden wrote: >> >>> Rob Crittenden wrote: >>> >>>> Bret Wortman wrote: >>>> >>>>> Well, my master ground to a halt and wasn't responding. I rebooted the >>>>> system and now I can't access the web UI or ssh to the master either. I >>>>> have console access but that's it. >>>>> >>>>> The services all say they're running, but the web UI gives an "Unknown >>>>> Error" dialog and ssh fails with "ssh_exchange_identification: >>>>> Connection closed by remote host" whenever I try to ssh to ipamaster. I >>>>> think something has gone really wrong inside my master. Any ideas? Even >>>>> after the reboot, --cleanup isn't helping and just hangs. >>>>> >>>>> The logfiles end (as of the time I ^C'd the process) with: >>>>> >>>>> NSMMReplicationPlugin - agmt="cn=meTogood3.spx.net >>>>> " (good3:389): Replication bind with GSSAPI >>>>> auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: >>>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >>>>> information (Cannot determine realm for numeric host address)) >>>>> NSMMReplicationPlugin - CleanAllRUV Task: Replica not online >>>>> (agmt="cn=meTogood3.foo.net " (good3:389)) >>>>> NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas online, >>>>> retrying in 160 seconds..., >>>>> >>>>> So it looks like it's having trouble talking with one of my replicas >>>>> and >>>>> is doggedly trying to get the job done. Any idea how to get the master >>>>> back working again while I troubleshoot this connectivity issue? >>>>> >>>> >>>> That suggests a DNS problem, and it might explain ssh as well depending >>>> on your configuration. >>>> >>> >>> To be clear, you ran --cleanup against one of the bad masters, not a >>> good one, right? >>> >>> rob >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Mon Aug 19 17:51:42 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 13:51:42 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> Message-ID: So, any idea how to fix the Kerberos problem? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman wrote: > ...and I got the web UI, authentication and sudo back via: > > # ipactl stop > # ipactl start > > Not sure why that worked, but it did. I was grasping at straws, honestly. > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman < > bret.wortman at damascusgrp.com> wrote: > >> Digging further, I think this log entry might be the problem between the >> two servers that aren't talking: >> >> slapd_ldap_sasl_interactive_bind - Error: could not perform interactive >> bind for id[] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic >> failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more >> information (Server ldap/localhost at SPX.NET not found in Kerberos >> database)) errno 2 (No such file or directory) >> >> Did I build something incorrectly when that server was set up originally? >> >> >> >> * >> * >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman < >> bret.wortman at damascusgrp.com> wrote: >> >>> I ran it on a good master, against a bad one. As in, I ran this command >>> on my master IPA node: >>> >>> # ipa-replica-manage del --force bad1.foo.net --cleanup >>> >>> Was that wrong? I was trying to delete the bad replica from the master, >>> so I figured the command needed to be run on the master. But again, my >>> master is now in a state where it's not resolving DNS, user logins, or sudo >>> at the very least. >>> >>> Oh, and I checked the node that it was complaining about earlier. The >>> network connection to it is the pits, but it's there. And it resolves. >>> >>> >>> * >>> * >>> *Bret Wortman* >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Mon, Aug 19, 2013 at 11:58 AM, Rob Crittenden wrote: >>> >>>> Rob Crittenden wrote: >>>> >>>>> Bret Wortman wrote: >>>>> >>>>>> Well, my master ground to a halt and wasn't responding. I rebooted the >>>>>> system and now I can't access the web UI or ssh to the master either. >>>>>> I >>>>>> have console access but that's it. >>>>>> >>>>>> The services all say they're running, but the web UI gives an "Unknown >>>>>> Error" dialog and ssh fails with "ssh_exchange_identification: >>>>>> Connection closed by remote host" whenever I try to ssh to ipamaster. >>>>>> I >>>>>> think something has gone really wrong inside my master. Any ideas? >>>>>> Even >>>>>> after the reboot, --cleanup isn't helping and just hangs. >>>>>> >>>>>> The logfiles end (as of the time I ^C'd the process) with: >>>>>> >>>>>> NSMMReplicationPlugin - agmt="cn=meTogood3.spx.net >>>>>> " (good3:389): Replication bind with GSSAPI >>>>>> auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: >>>>>> GSSAPI Error: Unspecified GSS failure. Minor code may provide more >>>>>> information (Cannot determine realm for numeric host address)) >>>>>> NSMMReplicationPlugin - CleanAllRUV Task: Replica not online >>>>>> (agmt="cn=meTogood3.foo.net " (good3:389)) >>>>>> NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas online, >>>>>> retrying in 160 seconds..., >>>>>> >>>>>> So it looks like it's having trouble talking with one of my replicas >>>>>> and >>>>>> is doggedly trying to get the job done. Any idea how to get the master >>>>>> back working again while I troubleshoot this connectivity issue? >>>>>> >>>>> >>>>> That suggests a DNS problem, and it might explain ssh as well depending >>>>> on your configuration. >>>>> >>>> >>>> To be clear, you ran --cleanup against one of the bad masters, not a >>>> good one, right? >>>> >>>> rob >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simo at redhat.com Mon Aug 19 18:02:54 2013 From: simo at redhat.com (Simo Sorce) Date: Mon, 19 Aug 2013 14:02:54 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> Message-ID: <1376935374.2814.130.camel@willson.li.ssimo.org> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: > So, any idea how to fix the Kerberos problem? > If your server is trying to get a tgt for ldap/localhost it probably means your /etc/hosts file is broken and has a line like this: 1.2.3.4 localhost my.real.name When GSSAPI tries to resolve my.realm.name it gets back that 'localhost' is the canonical name so it tries to get a TGT with that name and it fails. If /etc/host sis fine then the DNS server may be returning an IP address that later resolves to localhost again. To unbreak make sure that if you have your fully qualified name in /etc/hosts that it is on its own line pointing at the right IP address and where the FQDN name is the first in line: eg: this is ok: 1.2.3.4 server.full.name server this is not: 1.2.3.4 server server.full.name Simo. > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman > wrote: > ...and I got the web UI, authentication and sudo back via: > > > # ipactl stop > # ipactl start > > > Not sure why that worked, but it did. I was grasping at > straws, honestly. > > > > > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman > wrote: > Digging further, I think this log entry might be the > problem between the two servers that aren't talking: > > > slapd_ldap_sasl_interactive_bind - Error: could not > perform interactive bind for id[] mech [GSSAPI]: LDAP > error -2 (Local error) (SASL(-1): generic failure: > GSSAPI Error: Unspecified GSS failure. Minor code may > provide more information (Server > ldap/localhost at SPX.NET not found in Kerberos > database)) errno 2 (No such file or directory) > > > Did I build something incorrectly when that server was > set up originally? > > > > > > > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman > wrote: > I ran it on a good master, against a bad one. > As in, I ran this command on my master IPA > node: > > > # ipa-replica-manage del --force bad1.foo.net > --cleanup > > > Was that wrong? I was trying to delete the bad > replica from the master, so I figured the > command needed to be run on the master. But > again, my master is now in a state where it's > not resolving DNS, user logins, or sudo at the > very least. > > > Oh, and I checked the node that it was > complaining about earlier. The network > connection to it is the pits, but it's there. > And it resolves. > > > > > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > On Mon, Aug 19, 2013 at 11:58 AM, Rob > Crittenden wrote: > Rob Crittenden wrote: > Bret Wortman wrote: > Well, my master ground > to a halt and wasn't > responding. I rebooted > the > system and now I can't > access the web UI or > ssh to the master > either. I > have console access > but that's it. > > The services all say > they're running, but > the web UI gives an > "Unknown > Error" dialog and ssh > fails with > "ssh_exchange_identification: > Connection closed by > remote host" whenever > I try to ssh to > ipamaster. I > think something has > gone really wrong > inside my master. Any > ideas? Even > after the reboot, > --cleanup isn't > helping and just > hangs. > > The logfiles end (as > of the time I ^C'd the > process) with: > > NSMMReplicationPlugin > - > agmt="cn=meTogood3.spx.net > " (good3:389): Replication bind with GSSAPI > auth failed: LDAP > error -2 (Local error) > (SASL(-1): generic > failure: > GSSAPI Error: > Unspecified GSS > failure. Minor code > may provide more > information (Cannot > determine realm for > numeric host address)) > NSMMReplicationPlugin > - CleanAllRUV Task: > Replica not online > (agmt="cn=meTogood3.foo.net " (good3:389)) > NSMMReplicationPlugin > - CleanAllRUV Task: > Not all replicas > online, > retrying in 160 > seconds..., > > So it looks like it's > having trouble talking > with one of my > replicas and > is doggedly trying to > get the job done. Any > idea how to get the > master > back working again > while I troubleshoot > this connectivity > issue? > > That suggests a DNS problem, > and it might explain ssh as > well depending > on your configuration. > > > To be clear, you ran --cleanup against > one of the bad masters, not a good > one, right? > > rob > > > > > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Simo Sorce * Red Hat, Inc * New York From bret.wortman at damascusgrp.com Mon Aug 19 18:21:44 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Mon, 19 Aug 2013 14:21:44 -0400 Subject: [Freeipa-users] Fwd: Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> Message-ID: On my master (where this error is occurring), I've got, in /etc/hosts: 127.0.0.1 localhost localhost.localdomain ::1 localhost localhost.localdomain 1.2.3.4 ipamaster.foo.net ipamaster So that should be okay, right? # host ipamaster.foo.net ipamaster.foo.net has address 1.2.3.4 # host ipamaster ipamaster.foo.net has address 1.2.3.4 # host localhost localhost has address 127.0.0.1 localhost has IPv6 address ::1 # I checked the other system (the one I can't connect to) to be safe, and its /etc/hosts is similarly configured. It even has the master listed with its correct IP address. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce wrote: > On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: > > So, any idea how to fix the Kerberos problem? > > > > If your server is trying to get a tgt for ldap/localhost it probably > means your /etc/hosts file is broken and has a line like this: > > 1.2.3.4 localhost my.real.name > > When GSSAPI tries to resolve my.realm.name it gets back that 'localhost' > is the canonical name so it tries to get a TGT with that name and it > fails. > > If /etc/host sis fine then the DNS server may be returning an IP address > that later resolves to localhost again. > > To unbreak make sure that if you have your fully qualified name > in /etc/hosts that it is on its own line pointing at the right IP > address and where the FQDN name is the first in line: > eg: > > this is ok: > 1.2.3.4 server.full.name server > > this is not: > 1.2.3.4 server server.full.name > > Simo. > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman > > wrote: > > ...and I got the web UI, authentication and sudo back via: > > > > > > # ipactl stop > > # ipactl start > > > > > > Not sure why that worked, but it did. I was grasping at > > straws, honestly. > > > > > > > > > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > > > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman > > wrote: > > Digging further, I think this log entry might be the > > problem between the two servers that aren't talking: > > > > > > slapd_ldap_sasl_interactive_bind - Error: could not > > perform interactive bind for id[] mech [GSSAPI]: LDAP > > error -2 (Local error) (SASL(-1): generic failure: > > GSSAPI Error: Unspecified GSS failure. Minor code may > > provide more information (Server > > ldap/localhost at SPX.NET not found in Kerberos > > database)) errno 2 (No such file or directory) > > > > > > Did I build something incorrectly when that server was > > set up originally? > > > > > > > > > > > > > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > > > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman > > wrote: > > I ran it on a good master, against a bad one. > > As in, I ran this command on my master IPA > > node: > > > > > > # ipa-replica-manage del --force bad1.foo.net > > --cleanup > > > > > > Was that wrong? I was trying to delete the bad > > replica from the master, so I figured the > > command needed to be run on the master. But > > again, my master is now in a state where it's > > not resolving DNS, user logins, or sudo at the > > very least. > > > > > > Oh, and I checked the node that it was > > complaining about earlier. The network > > connection to it is the pits, but it's there. > > And it resolves. > > > > > > > > > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > On Mon, Aug 19, 2013 at 11:58 AM, Rob > > Crittenden wrote: > > Rob Crittenden wrote: > > Bret Wortman wrote: > > Well, my master ground > > to a halt and wasn't > > responding. I rebooted > > the > > system and now I can't > > access the web UI or > > ssh to the master > > either. I > > have console access > > but that's it. > > > > The services all say > > they're running, but > > the web UI gives an > > "Unknown > > Error" dialog and ssh > > fails with > > > "ssh_exchange_identification: > > Connection closed by > > remote host" whenever > > I try to ssh to > > ipamaster. I > > think something has > > gone really wrong > > inside my master. Any > > ideas? Even > > after the reboot, > > --cleanup isn't > > helping and just > > hangs. > > > > The logfiles end (as > > of the time I ^C'd the > > process) with: > > > > NSMMReplicationPlugin > > - > > agmt="cn= > meTogood3.spx.net > > < > http://meTogood3.spx.net>" (good3:389): Replication bind with GSSAPI > > auth failed: LDAP > > error -2 (Local error) > > (SASL(-1): generic > > failure: > > GSSAPI Error: > > Unspecified GSS > > failure. Minor code > > may provide more > > information (Cannot > > determine realm for > > numeric host address)) > > NSMMReplicationPlugin > > - CleanAllRUV Task: > > Replica not online > > (agmt="cn= > meTogood3.foo.net " (good3:389)) > > NSMMReplicationPlugin > > - CleanAllRUV Task: > > Not all replicas > > online, > > retrying in 160 > > seconds..., > > > > So it looks like it's > > having trouble talking > > with one of my > > replicas and > > is doggedly trying to > > get the job done. Any > > idea how to get the > > master > > back working again > > while I troubleshoot > > this connectivity > > issue? > > > > That suggests a DNS problem, > > and it might explain ssh as > > well depending > > on your configuration. > > > > > > To be clear, you ran --cleanup against > > one of the bad masters, not a good > > one, right? > > > > rob > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > Freeipa-users mailing list > > Freeipa-users at redhat.com > > https://www.redhat.com/mailman/listinfo/freeipa-users > > > -- > Simo Sorce * Red Hat, Inc * New York > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Tue Aug 20 11:55:35 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 20 Aug 2013 07:55:35 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> Message-ID: Okay, now I'm thinking I need to dump all my replicas and start them fresh. My /var/log/slapd-FOO-COM/errors is filled with messages like this: NSMMReplicationPlugin - changelog program - agmt="cn=meTogood1.foo.com" (good1:389): CSN 520a49640000001d0000 not found, we aren't as up to date, or we purged agmt="cn=meTogood1.foo.com" (good1:389) - Can't locate CSN 520a49640000001d0000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized. I assume the "consumer" is the replica, right? At present, I have two replicas known to my master that are simply gone. Another is there but they can't talk. Three more have good communication but I'm getting errors like these. Is there a good, clean way to just clobber all the replicas and start over without trashing the DNS and other identity data that is inside my master and which *is* working? Deleting them from the master hasn't been working; it tends to hang the master's DNS and other services until I Ctrl-C out and "ipactl restart" it. I'm afraid to venture out without a net here and make things worse.... * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman wrote: > On my master (where this error is occurring), I've got, in /etc/hosts: > > 127.0.0.1 localhost localhost.localdomain > ::1 localhost localhost.localdomain > 1.2.3.4 ipamaster.foo.net ipamaster > > So that should be okay, right? > > # host ipamaster.foo.net > ipamaster.foo.net has address 1.2.3.4 > # host ipamaster > ipamaster.foo.net has address 1.2.3.4 > # host localhost > localhost has address 127.0.0.1 > localhost has IPv6 address ::1 > # > > I checked the other system (the one I can't connect to) to be safe, and > its /etc/hosts is similarly configured. It even has the master listed with > its correct IP address. > > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce wrote: > >> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: >> > So, any idea how to fix the Kerberos problem? >> > >> >> If your server is trying to get a tgt for ldap/localhost it probably >> means your /etc/hosts file is broken and has a line like this: >> >> 1.2.3.4 localhost my.real.name >> >> When GSSAPI tries to resolve my.realm.name it gets back that 'localhost' >> is the canonical name so it tries to get a TGT with that name and it >> fails. >> >> If /etc/host sis fine then the DNS server may be returning an IP address >> that later resolves to localhost again. >> >> To unbreak make sure that if you have your fully qualified name >> in /etc/hosts that it is on its own line pointing at the right IP >> address and where the FQDN name is the first in line: >> eg: >> >> this is ok: >> 1.2.3.4 server.full.name server >> >> this is not: >> 1.2.3.4 server server.full.name >> >> Simo. >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman >> > wrote: >> > ...and I got the web UI, authentication and sudo back via: >> > >> > >> > # ipactl stop >> > # ipactl start >> > >> > >> > Not sure why that worked, but it did. I was grasping at >> > straws, honestly. >> > >> > >> > >> > >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > >> > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman >> > wrote: >> > Digging further, I think this log entry might be the >> > problem between the two servers that aren't talking: >> > >> > >> > slapd_ldap_sasl_interactive_bind - Error: could not >> > perform interactive bind for id[] mech [GSSAPI]: LDAP >> > error -2 (Local error) (SASL(-1): generic failure: >> > GSSAPI Error: Unspecified GSS failure. Minor code may >> > provide more information (Server >> > ldap/localhost at SPX.NET not found in Kerberos >> > database)) errno 2 (No such file or directory) >> > >> > >> > Did I build something incorrectly when that server was >> > set up originally? >> > >> > >> > >> > >> > >> > >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > >> > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman >> > wrote: >> > I ran it on a good master, against a bad one. >> > As in, I ran this command on my master IPA >> > node: >> > >> > >> > # ipa-replica-manage del --force bad1.foo.net >> > --cleanup >> > >> > >> > Was that wrong? I was trying to delete the bad >> > replica from the master, so I figured the >> > command needed to be run on the master. But >> > again, my master is now in a state where it's >> > not resolving DNS, user logins, or sudo at the >> > very least. >> > >> > >> > Oh, and I checked the node that it was >> > complaining about earlier. The network >> > connection to it is the pits, but it's there. >> > And it resolves. >> > >> > >> > >> > >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > On Mon, Aug 19, 2013 at 11:58 AM, Rob >> > Crittenden wrote: >> > Rob Crittenden wrote: >> > Bret Wortman wrote: >> > Well, my master ground >> > to a halt and wasn't >> > responding. I rebooted >> > the >> > system and now I can't >> > access the web UI or >> > ssh to the master >> > either. I >> > have console access >> > but that's it. >> > >> > The services all say >> > they're running, but >> > the web UI gives an >> > "Unknown >> > Error" dialog and ssh >> > fails with >> > >> "ssh_exchange_identification: >> > Connection closed by >> > remote host" whenever >> > I try to ssh to >> > ipamaster. I >> > think something has >> > gone really wrong >> > inside my master. Any >> > ideas? Even >> > after the reboot, >> > --cleanup isn't >> > helping and just >> > hangs. >> > >> > The logfiles end (as >> > of the time I ^C'd the >> > process) with: >> > >> > NSMMReplicationPlugin >> > - >> > agmt="cn= >> meTogood3.spx.net >> > < >> http://meTogood3.spx.net>" (good3:389): Replication bind with GSSAPI >> > auth failed: LDAP >> > error -2 (Local error) >> > (SASL(-1): generic >> > failure: >> > GSSAPI Error: >> > Unspecified GSS >> > failure. Minor code >> > may provide more >> > information (Cannot >> > determine realm for >> > numeric host address)) >> > NSMMReplicationPlugin >> > - CleanAllRUV Task: >> > Replica not online >> > (agmt="cn= >> meTogood3.foo.net " (good3:389)) >> > NSMMReplicationPlugin >> > - CleanAllRUV Task: >> > Not all replicas >> > online, >> > retrying in 160 >> > seconds..., >> > >> > So it looks like it's >> > having trouble talking >> > with one of my >> > replicas and >> > is doggedly trying to >> > get the job done. Any >> > idea how to get the >> > master >> > back working again >> > while I troubleshoot >> > this connectivity >> > issue? >> > >> > That suggests a DNS problem, >> > and it might explain ssh as >> > well depending >> > on your configuration. >> > >> > >> > To be clear, you ran --cleanup against >> > one of the bad masters, not a good >> > one, right? >> > >> > rob >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > Freeipa-users mailing list >> > Freeipa-users at redhat.com >> > https://www.redhat.com/mailman/listinfo/freeipa-users >> >> >> -- >> Simo Sorce * Red Hat, Inc * New York >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmeggins at redhat.com Tue Aug 20 13:46:54 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Tue, 20 Aug 2013 07:46:54 -0600 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> Message-ID: <5213734E.6070102@redhat.com> On 08/20/2013 05:55 AM, Bret Wortman wrote: > Okay, now I'm thinking I need to dump all my replicas and start them > fresh. My /var/log/slapd-FOO-COM/errors is filled with messages like > this: > > NSMMReplicationPlugin - changelog program - agmt="cn=meTogood1.foo.com > " (good1:389): CSN 520a49640000001d0000 not > found, we aren't as up to date, or we purged > agmt="cn=meTogood1.foo.com " (good1:389) - > Can't locate CSN 520a49640000001d0000 in the changelog (DB rc=-30988). > The consumer may need to be reinitialized. > > I assume the "consumer" is the replica, right? At present, I have two > replicas known to my master that are simply gone. Another is there but > they can't talk. Three more have good communication but I'm getting > errors like these. Is there a good, clean way to just clobber all the > replicas and start over without trashing the DNS and other identity > data that is inside my master and which /is/ working? Deleting them > from the master hasn't been working; it tends to hang the master's DNS > and other services until I Ctrl-C out and "ipactl restart" it. > > I'm afraid to venture out without a net here and make things worse.... This looks like https://fedorahosted.org/389/ticket/47386 We've never been able to reproduce this in a "controlled" environment. The original reporter has been able to get this to work in some cases by restarting ipa (ipactl restart). Before you do that, would you be able to provide some information for me? On the supplier and consumer: ldapsearch -xLLL -D "cn=directory manager" -W -b "dc=FOO,dc=COM" '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' > ruv.ldif ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" '(objectclass=nsds5replicationagreement)' > agmt.ldif dbscan -f /var/lib/dirsrv/slapd-FOO-COM/cldb/*.db4 | head -200 > cldb.txt Be sure to obscure any sensitive data in ruv.ldif, agmt.ldif, and cldb.txt - you can either attach to https://fedorahosted.org/389/ticket/47386 or email to me directly. > > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman > > > wrote: > > On my master (where this error is occurring), I've got, in /etc/hosts: > > 127.0.0.1 localhost localhost.localdomain > ::1 localhost localhost.localdomain > 1.2.3.4 ipamaster.foo.net ipamaster > > So that should be okay, right? > > # host ipamaster.foo.net > ipamaster.foo.net has address 1.2.3.4 > # host ipamaster > ipamaster.foo.net has address 1.2.3.4 > # host localhost > localhost has address 127.0.0.1 > localhost has IPv6 address ::1 > # > > I checked the other system (the one I can't connect to) to be > safe, and its /etc/hosts is similarly configured. It even has the > master listed with its correct IP address. > > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce > wrote: > > On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: > > So, any idea how to fix the Kerberos problem? > > > > If your server is trying to get a tgt for ldap/localhost it > probably > means your /etc/hosts file is broken and has a line like this: > > 1.2.3.4 localhost my.real.name > > When GSSAPI tries to resolve my.realm.name > it gets back that 'localhost' > is the canonical name so it tries to get a TGT with that name > and it > fails. > > If /etc/host sis fine then the DNS server may be returning an > IP address > that later resolves to localhost again. > > To unbreak make sure that if you have your fully qualified name > in /etc/hosts that it is on its own line pointing at the right IP > address and where the FQDN name is the first in line: > eg: > > this is ok: > 1.2.3.4 server.full.name server > > this is not: > 1.2.3.4 server server.full.name > > Simo. > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman > > > wrote: > > ...and I got the web UI, authentication and sudo > back via: > > > > > > # ipactl stop > > # ipactl start > > > > > > Not sure why that worked, but it did. I was grasping at > > straws, honestly. > > > > > > > > > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > > > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman > > > wrote: > > Digging further, I think this log entry > might be the > > problem between the two servers that aren't > talking: > > > > > > slapd_ldap_sasl_interactive_bind - Error: could not > > perform interactive bind for id[] mech > [GSSAPI]: LDAP > > error -2 (Local error) (SASL(-1): generic > failure: > > GSSAPI Error: Unspecified GSS failure. Minor > code may > > provide more information (Server > > ldap/localhost at SPX.NET > not found in Kerberos > > database)) errno 2 (No such file or directory) > > > > > > Did I build something incorrectly when that > server was > > set up originally? > > > > > > > > > > > > > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > > > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman > > > wrote: > > I ran it on a good master, against a > bad one. > > As in, I ran this command on my > master IPA > > node: > > > > > > # ipa-replica-manage del --force > bad1.foo.net > > --cleanup > > > > > > Was that wrong? I was trying to > delete the bad > > replica from the master, so I > figured the > > command needed to be run on the > master. But > > again, my master is now in a state > where it's > > not resolving DNS, user logins, or > sudo at the > > very least. > > > > > > Oh, and I checked the node that it was > > complaining about earlier. The network > > connection to it is the pits, but > it's there. > > And it resolves. > > > > > > > > > > > > Bret Wortman > > > > > > http://damascusgrp.com/ > > > > http://about.me/wortmanbret > > > > > > > > On Mon, Aug 19, 2013 at 11:58 AM, Rob > > Crittenden > wrote: > > Rob Crittenden wrote: > > Bret Wortman wrote: > > Well, my master ground > > to a halt and wasn't > > responding. I rebooted > > the > > system and now I can't > > access the web UI or > > ssh to the master > > either. I > > have console access > > but that's it. > > > > The services all say > > they're running, but > > the web UI gives an > > "Unknown > > Error" dialog and ssh > > fails with > > "ssh_exchange_identification: > > Connection closed by > > remote host" whenever > > I try to ssh to > > ipamaster. I > > think something has > > gone really wrong > > inside my master. Any > > ideas? Even > > after the reboot, > > --cleanup isn't > > helping and just > > hangs. > > > > The logfiles end (as > > of the time I ^C'd the > > process) with: > > > > NSMMReplicationPlugin > > - > > agmt="cn=meTogood3.spx.net > > > " (good3:389): > Replication bind with GSSAPI > > auth failed: LDAP > > error -2 (Local error) > > (SASL(-1): generic > > failure: > > GSSAPI Error: > > Unspecified GSS > > failure. Minor code > > may provide more > > information (Cannot > > determine realm for > > numeric host address)) > > NSMMReplicationPlugin > > - CleanAllRUV Task: > > Replica not online > > (agmt="cn=meTogood3.foo.net > " > (good3:389)) > > NSMMReplicationPlugin > > - CleanAllRUV Task: > > Not all replicas > > online, > > retrying in 160 > > seconds..., > > > > So it looks like it's > > having trouble talking > > with one of my > > replicas and > > is doggedly trying to > > get the job done. Any > > idea how to get the > > master > > back working again > > while I troubleshoot > > this connectivity > > issue? > > > > That suggests a DNS problem, > > and it might explain ssh as > > well depending > > on your configuration. > > > > > > To be clear, you ran --cleanup against > > one of the bad masters, not a good > > one, right? > > > > rob > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > Freeipa-users mailing list > > Freeipa-users at redhat.com > > https://www.redhat.com/mailman/listinfo/freeipa-users > > > -- > Simo Sorce * Red Hat, Inc * New York > > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Tue Aug 20 15:15:17 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 20 Aug 2013 11:15:17 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: <5213734E.6070102@redhat.com> References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> <5213734E.6070102@redhat.com> Message-ID: If I were going to attempt to restore to an old backup, what directories/files should I make sure to restore? I've got a backup script that tars up: /usr/share/ipa /usr/lib64/ipa /var/lib/pia /var/lib/ipa-client /var/lib/dirsrv /etc Is that enough to "roll back" to a few days ago before I started down this path? I'm now seeing messages about having the max number of CleanAllRUV tasks (4) and not being able to enqueue any more. So I'm really stuck now and don't know how soon I can get the files requested over to Rich for analysis. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Tue, Aug 20, 2013 at 9:46 AM, Rich Megginson wrote: > On 08/20/2013 05:55 AM, Bret Wortman wrote: > > Okay, now I'm thinking I need to dump all my replicas and start them > fresh. My /var/log/slapd-FOO-COM/errors is filled with messages like this: > > NSMMReplicationPlugin - changelog program - agmt="cn=meTogood1.foo.com" > (good1:389): CSN 520a49640000001d0000 not found, we aren't as up to date, > or we purged > agmt="cn=meTogood1.foo.com" (good1:389) - Can't locate CSN > 520a49640000001d0000 in the changelog (DB rc=-30988). The consumer may need > to be reinitialized. > > I assume the "consumer" is the replica, right? At present, I have two > replicas known to my master that are simply gone. Another is there but they > can't talk. Three more have good communication but I'm getting errors like > these. Is there a good, clean way to just clobber all the replicas and > start over without trashing the DNS and other identity data that is inside > my master and which *is* working? Deleting them from the master hasn't > been working; it tends to hang the master's DNS and other services until I > Ctrl-C out and "ipactl restart" it. > > I'm afraid to venture out without a net here and make things worse.... > > > This looks like https://fedorahosted.org/389/ticket/47386 > > We've never been able to reproduce this in a "controlled" environment. > > The original reporter has been able to get this to work in some cases by > restarting ipa (ipactl restart). > > Before you do that, would you be able to provide some information for me? > > On the supplier and consumer: > ldapsearch -xLLL -D "cn=directory manager" -W -b "dc=FOO,dc=COM" > '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' > > ruv.ldif > > ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" > '(objectclass=nsds5replicationagreement)' > agmt.ldif > > dbscan -f /var/lib/dirsrv/slapd-FOO-COM/cldb/*.db4 | head -200 > cldb.txt > > Be sure to obscure any sensitive data in ruv.ldif, agmt.ldif, and cldb.txt > - you can either attach to https://fedorahosted.org/389/ticket/47386 or > email to me directly. > > > > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman < > bret.wortman at damascusgrp.com> wrote: > >> On my master (where this error is occurring), I've got, in /etc/hosts: >> >> 127.0.0.1 localhost localhost.localdomain >> ::1 localhost localhost.localdomain >> 1.2.3.4 ipamaster.foo.net ipamaster >> >> So that should be okay, right? >> >> # host ipamaster.foo.net >> ipamaster.foo.net has address 1.2.3.4 >> # host ipamaster >> ipamaster.foo.net has address 1.2.3.4 >> # host localhost >> localhost has address 127.0.0.1 >> localhost has IPv6 address ::1 >> # >> >> I checked the other system (the one I can't connect to) to be safe, and >> its /etc/hosts is similarly configured. It even has the master listed with >> its correct IP address. >> >> >> >> * >> * >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce wrote: >> >>> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: >>> > So, any idea how to fix the Kerberos problem? >>> > >>> >>> If your server is trying to get a tgt for ldap/localhost it probably >>> means your /etc/hosts file is broken and has a line like this: >>> >>> 1.2.3.4 localhost my.real.name >>> >>> When GSSAPI tries to resolve my.realm.name it gets back that 'localhost' >>> is the canonical name so it tries to get a TGT with that name and it >>> fails. >>> >>> If /etc/host sis fine then the DNS server may be returning an IP address >>> that later resolves to localhost again. >>> >>> To unbreak make sure that if you have your fully qualified name >>> in /etc/hosts that it is on its own line pointing at the right IP >>> address and where the FQDN name is the first in line: >>> eg: >>> >>> this is ok: >>> 1.2.3.4 server.full.name server >>> >>> this is not: >>> 1.2.3.4 server server.full.name >>> >>> Simo. >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman >>> > wrote: >>> > ...and I got the web UI, authentication and sudo back via: >>> > >>> > >>> > # ipactl stop >>> > # ipactl start >>> > >>> > >>> > Not sure why that worked, but it did. I was grasping at >>> > straws, honestly. >>> > >>> > >>> > >>> > >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman >>> > wrote: >>> > Digging further, I think this log entry might be the >>> > problem between the two servers that aren't talking: >>> > >>> > >>> > slapd_ldap_sasl_interactive_bind - Error: could not >>> > perform interactive bind for id[] mech [GSSAPI]: LDAP >>> > error -2 (Local error) (SASL(-1): generic failure: >>> > GSSAPI Error: Unspecified GSS failure. Minor code may >>> > provide more information (Server >>> > ldap/localhost at SPX.NET not found in Kerberos >>> > database)) errno 2 (No such file or directory) >>> > >>> > >>> > Did I build something incorrectly when that server was >>> > set up originally? >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman >>> > wrote: >>> > I ran it on a good master, against a bad one. >>> > As in, I ran this command on my master IPA >>> > node: >>> > >>> > >>> > # ipa-replica-manage del --force bad1.foo.net >>> > --cleanup >>> > >>> > >>> > Was that wrong? I was trying to delete the bad >>> > replica from the master, so I figured the >>> > command needed to be run on the master. But >>> > again, my master is now in a state where it's >>> > not resolving DNS, user logins, or sudo at the >>> > very least. >>> > >>> > >>> > Oh, and I checked the node that it was >>> > complaining about earlier. The network >>> > connection to it is the pits, but it's there. >>> > And it resolves. >>> > >>> > >>> > >>> > >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 11:58 AM, Rob >>> > Crittenden wrote: >>> > Rob Crittenden wrote: >>> > Bret Wortman wrote: >>> > Well, my master ground >>> > to a halt and wasn't >>> > responding. I rebooted >>> > the >>> > system and now I can't >>> > access the web UI or >>> > ssh to the master >>> > either. I >>> > have console access >>> > but that's it. >>> > >>> > The services all say >>> > they're running, but >>> > the web UI gives an >>> > "Unknown >>> > Error" dialog and ssh >>> > fails with >>> > >>> "ssh_exchange_identification: >>> > Connection closed by >>> > remote host" whenever >>> > I try to ssh to >>> > ipamaster. I >>> > think something has >>> > gone really wrong >>> > inside my master. Any >>> > ideas? Even >>> > after the reboot, >>> > --cleanup isn't >>> > helping and just >>> > hangs. >>> > >>> > The logfiles end (as >>> > of the time I ^C'd the >>> > process) with: >>> > >>> > NSMMReplicationPlugin >>> > - >>> > agmt="cn= >>> meTogood3.spx.net >>> > < >>> http://meTogood3.spx.net>" (good3:389): Replication bind with GSSAPI >>> > auth failed: LDAP >>> > error -2 (Local error) >>> > (SASL(-1): generic >>> > failure: >>> > GSSAPI Error: >>> > Unspecified GSS >>> > failure. Minor code >>> > may provide more >>> > information (Cannot >>> > determine realm for >>> > numeric host address)) >>> > NSMMReplicationPlugin >>> > - CleanAllRUV Task: >>> > Replica not online >>> > (agmt="cn= >>> meTogood3.foo.net " (good3:389)) >>> > NSMMReplicationPlugin >>> > - CleanAllRUV Task: >>> > Not all replicas >>> > online, >>> > retrying in 160 >>> > seconds..., >>> > >>> > So it looks like it's >>> > having trouble talking >>> > with one of my >>> > replicas and >>> > is doggedly trying to >>> > get the job done. Any >>> > idea how to get the >>> > master >>> > back working again >>> > while I troubleshoot >>> > this connectivity >>> > issue? >>> > >>> > That suggests a DNS problem, >>> > and it might explain ssh as >>> > well depending >>> > on your configuration. >>> > >>> > >>> > To be clear, you ran --cleanup against >>> > one of the bad masters, not a good >>> > one, right? >>> > >>> > rob >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > _______________________________________________ >>> > Freeipa-users mailing list >>> > Freeipa-users at redhat.com >>> > https://www.redhat.com/mailman/listinfo/freeipa-users >>> >>> >>> -- >>> Simo Sorce * Red Hat, Inc * New York >>> >>> >> >> > > > _______________________________________________ > Freeipa-users mailing listFreeipa-users at redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From JR.Aquino at citrix.com Tue Aug 20 16:10:58 2013 From: JR.Aquino at citrix.com (JR Aquino) Date: Tue, 20 Aug 2013 16:10:58 +0000 Subject: [Freeipa-users] Replication woes In-Reply-To: <5213734E.6070102@redhat.com> References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> <5213734E.6070102@redhat.com> Message-ID: <8E27838E-396F-4E10-BFC8-E551CBD7EAFD@citrixonline.com> On Aug 20, 2013, at 6:46 AM, Rich Megginson > wrote: On 08/20/2013 05:55 AM, Bret Wortman wrote: Okay, now I'm thinking I need to dump all my replicas and start them fresh. My /var/log/slapd-FOO-COM/errors is filled with messages like this: NSMMReplicationPlugin - changelog program - agmt="cn=meTogood1.foo.com" (good1:389): CSN 520a49640000001d0000 not found, we aren't as up to date, or we purged agmt="cn=meTogood1.foo.com" (good1:389) - Can't locate CSN 520a49640000001d0000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized. I assume the "consumer" is the replica, right? At present, I have two replicas known to my master that are simply gone. Another is there but they can't talk. Three more have good communication but I'm getting errors like these. Is there a good, clean way to just clobber all the replicas and start over without trashing the DNS and other identity data that is inside my master and which is working? Deleting them from the master hasn't been working; it tends to hang the master's DNS and other services until I Ctrl-C out and "ipactl restart" it. I'm afraid to venture out without a net here and make things worse.... This looks like https://fedorahosted.org/389/ticket/47386 We've never been able to reproduce this in a "controlled" environment. The original reporter has been able to get this to work in some cases by restarting ipa (ipactl restart). Before you do that, would you be able to provide some information for me? On the supplier and consumer: ldapsearch -xLLL -D "cn=directory manager" -W -b "dc=FOO,dc=COM" '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' > ruv.ldif ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" '(objectclass=nsds5replicationagreement)' > agmt.ldif dbscan -f /var/lib/dirsrv/slapd-FOO-COM/cldb/*.db4 | head -200 > cldb.txt Be sure to obscure any sensitive data in ruv.ldif, agmt.ldif, and cldb.txt - you can either attach to https://fedorahosted.org/389/ticket/47386 or email to me directly. Any help you could provide in capturing the fail-state would be hugely appreciated. I've found that if you work through the issue and fix the problem, it doesn't appear to be deliberately reproducible. If you can get the debugging data that Rich needs, I can work on drafting you a basic howto on how to diagnose and fix your replication issue. Bret Wortman [http://damascusgrp.com/item/51f7de33e4b08d2bdb8b4860?format=1500w] http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman > wrote: On my master (where this error is occurring), I've got, in /etc/hosts: 127.0.0.1 localhost localhost.localdomain ::1 localhost localhost.localdomain 1.2.3.4 ipamaster.foo.net ipamaster So that should be okay, right? # host ipamaster.foo.net ipamaster.foo.net has address 1.2.3.4 # host ipamaster ipamaster.foo.net has address 1.2.3.4 # host localhost localhost has address 127.0.0.1 localhost has IPv6 address ::1 # I checked the other system (the one I can't connect to) to be safe, and its /etc/hosts is similarly configured. It even has the master listed with its correct IP address. Bret Wortman [http://damascusgrp.com/item/51f7de33e4b08d2bdb8b4860?format=1500w] http://damascusgrp.com/ http://about.me/wortmanbret On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce > wrote: On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: > So, any idea how to fix the Kerberos problem? > If your server is trying to get a tgt for ldap/localhost it probably means your /etc/hosts file is broken and has a line like this: 1.2.3.4 localhost my.real.name When GSSAPI tries to resolve my.realm.name it gets back that 'localhost' is the canonical name so it tries to get a TGT with that name and it fails. If /etc/host sis fine then the DNS server may be returning an IP address that later resolves to localhost again. To unbreak make sure that if you have your fully qualified name in /etc/hosts that it is on its own line pointing at the right IP address and where the FQDN name is the first in line: eg: this is ok: 1.2.3.4 server.full.name server this is not: 1.2.3.4 server server.full.name Simo. > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman > > wrote: > ...and I got the web UI, authentication and sudo back via: > > > # ipactl stop > # ipactl start > > > Not sure why that worked, but it did. I was grasping at > straws, honestly. > > > > > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman > > wrote: > Digging further, I think this log entry might be the > problem between the two servers that aren't talking: > > > slapd_ldap_sasl_interactive_bind - Error: could not > perform interactive bind for id[] mech [GSSAPI]: LDAP > error -2 (Local error) (SASL(-1): generic failure: > GSSAPI Error: Unspecified GSS failure. Minor code may > provide more information (Server > ldap/localhost at SPX.NET not found in Kerberos > database)) errno 2 (No such file or directory) > > > Did I build something incorrectly when that server was > set up originally? > > > > > > > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman > > wrote: > I ran it on a good master, against a bad one. > As in, I ran this command on my master IPA > node: > > > # ipa-replica-manage del --force bad1.foo.net > --cleanup > > > Was that wrong? I was trying to delete the bad > replica from the master, so I figured the > command needed to be run on the master. But > again, my master is now in a state where it's > not resolving DNS, user logins, or sudo at the > very least. > > > Oh, and I checked the node that it was > complaining about earlier. The network > connection to it is the pits, but it's there. > And it resolves. > > > > > > Bret Wortman > > > http://damascusgrp.com/ > > http://about.me/wortmanbret > > > > On Mon, Aug 19, 2013 at 11:58 AM, Rob > Crittenden > wrote: > Rob Crittenden wrote: > Bret Wortman wrote: > Well, my master ground > to a halt and wasn't > responding. I rebooted > the > system and now I can't > access the web UI or > ssh to the master > either. I > have console access > but that's it. > > The services all say > they're running, but > the web UI gives an > "Unknown > Error" dialog and ssh > fails with > "ssh_exchange_identification: > Connection closed by > remote host" whenever > I try to ssh to > ipamaster. I > think something has > gone really wrong > inside my master. Any > ideas? Even > after the reboot, > --cleanup isn't > helping and just > hangs. > > The logfiles end (as > of the time I ^C'd the > process) with: > > NSMMReplicationPlugin > - > agmt="cn=meTogood3.spx.net > >" (good3:389): Replication bind with GSSAPI > auth failed: LDAP > error -2 (Local error) > (SASL(-1): generic > failure: > GSSAPI Error: > Unspecified GSS > failure. Minor code > may provide more > information (Cannot > determine realm for > numeric host address)) > NSMMReplicationPlugin > - CleanAllRUV Task: > Replica not online > (agmt="cn=meTogood3.foo.net >" (good3:389)) > NSMMReplicationPlugin > - CleanAllRUV Task: > Not all replicas > online, > retrying in 160 > seconds..., > > So it looks like it's > having trouble talking > with one of my > replicas and > is doggedly trying to > get the job done. Any > idea how to get the > master > back working again > while I troubleshoot > this connectivity > issue? > > That suggests a DNS problem, > and it might explain ssh as > well depending > on your configuration. > > > To be clear, you ran --cleanup against > one of the bad masters, not a good > one, right? > > rob > > > > > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Simo Sorce * Red Hat, Inc * New York _______________________________________________ Freeipa-users mailing list Freeipa-users at redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users _______________________________________________ Freeipa-users mailing list Freeipa-users at redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users From mindaugas.deveikis at itreegroup.eu Thu Aug 22 07:57:19 2013 From: mindaugas.deveikis at itreegroup.eu (Mindaugas Deveikis) Date: Thu, 22 Aug 2013 10:57:19 +0300 Subject: [Freeipa-users] 389 Management Console Message-ID: <5215C45F.4050307@itreegroup.eu> Hi, Just a short question. Can I launch 389 Management Console on 9830 port on freeipa server? If yes, how can I do that? Thank's Mindaugas From sramling at redhat.com Thu Aug 22 11:25:20 2013 From: sramling at redhat.com (Sankar Ramlingam) Date: Thu, 22 Aug 2013 16:55:20 +0530 Subject: [Freeipa-users] 389 Management Console In-Reply-To: <5215C45F.4050307@itreegroup.eu> References: <5215C45F.4050307@itreegroup.eu> Message-ID: <5215F520.60606@redhat.com> On 08/22/2013 01:27 PM, Mindaugas Deveikis wrote: > Hi, > > Just a short question. Can I launch 389 Management Console on 9830 > port on freeipa server? If yes, how can I do that? I am not sure, I understood your question. Do you mean to say, accessing freeipa server from 389 Management console? To access 389 console, you need to run "389-console" from the command line. Before that, make sure 389-console, 389-admin and 389-admin-console packages are installed. Please refer to this page - http://directory.fedoraproject.org/wiki/Releases/1.3.1.6, to download/install 389-console packages. Thanks, -Sankar R. > Thank's > > Mindaugas > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users From rcritten at redhat.com Thu Aug 22 12:57:16 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 22 Aug 2013 08:57:16 -0400 Subject: [Freeipa-users] 389 Management Console In-Reply-To: <5215C45F.4050307@itreegroup.eu> References: <5215C45F.4050307@itreegroup.eu> Message-ID: <52160AAC.6060705@redhat.com> Mindaugas Deveikis wrote: > Hi, > > Just a short question. Can I launch 389 Management Console on 9830 > port on freeipa server? If yes, how can I do that? Thank's We don't support or recommend using the management console on an IPA server. What do you need the console for? rob From bwellsnc at gmail.com Mon Aug 26 17:50:11 2013 From: bwellsnc at gmail.com (bwellsnc) Date: Mon, 26 Aug 2013 13:50:11 -0400 Subject: [Freeipa-users] FreeIPA Replica ports Message-ID: I have been over the documentation and all documentations states that replication happens over port 7389. This is incorrect. It is happening over 389. I have a need for replication to operate over 7389 because I have a remote server that is located in a datacenter which I have no vpn/p2p access. Is there a way to set the replication port in IPA? Thanks! Brent -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Aug 26 18:08:44 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 26 Aug 2013 14:08:44 -0400 Subject: [Freeipa-users] FreeIPA Replica ports In-Reply-To: References: Message-ID: <521B99AC.6070509@redhat.com> bwellsnc wrote: > I have been over the documentation and all documentations states that > replication happens over port 7389. This is incorrect. It is happening > over 389. I have a need for replication to operate over 7389 because I > have a remote server that is located in a datacenter which I have no > vpn/p2p access. Is there a way to set the replication port in IPA? The documentation is a little unclear, I agree. It is trying to say that IF you want a CA on the replica then you'll need port 7389 (and a few others) opened in the firewall. Changing the port would require reconfiguring 389-ds to listen on another port (or an additional port) and configure replication over that port. We don't provide the ability to configure ports so you'd need to make code changes. If the concern is lack of security, we initially (during ipa-replica-install) to use startTLS over 389. Once the server is up we reconfigure the agreement to use GSSAPI, so the data is always encrypted. For the case of the CA, it always uses startTLS on port 7389. rob From simo at redhat.com Mon Aug 26 19:41:26 2013 From: simo at redhat.com (Simo Sorce) Date: Mon, 26 Aug 2013 15:41:26 -0400 Subject: [Freeipa-users] FreeIPA Replica ports In-Reply-To: <521B99AC.6070509@redhat.com> References: <521B99AC.6070509@redhat.com> Message-ID: <1377546086.4083.14.camel@willson.li.ssimo.org> On Mon, 2013-08-26 at 14:08 -0400, Rob Crittenden wrote: > bwellsnc wrote: > > I have been over the documentation and all documentations states that > > replication happens over port 7389. This is incorrect. It is happening > > over 389. I have a need for replication to operate over 7389 because I > > have a remote server that is located in a datacenter which I have no > > vpn/p2p access. Is there a way to set the replication port in IPA? > > The documentation is a little unclear, I agree. It is trying to say that > IF you want a CA on the replica then you'll need port 7389 (and a few > others) opened in the firewall. > > Changing the port would require reconfiguring 389-ds to listen on > another port (or an additional port) and configure replication over that > port. We don't provide the ability to configure ports so you'd need to > make code changes. > > If the concern is lack of security, we initially (during > ipa-replica-install) to use startTLS over 389. Once the server is up we > reconfigure the agreement to use GSSAPI, so the data is always > encrypted. For the case of the CA, it always uses startTLS on port 7389. We should also probably note that in newer versions of FreeIPA we have consolidated all instances in one, so only port 389 is used. Simo. -- Simo Sorce * Red Hat, Inc * New York From jessie.floyd at gmail.com Tue Aug 27 03:21:11 2013 From: jessie.floyd at gmail.com (Jessie Floyd) Date: Mon, 26 Aug 2013 21:21:11 -0600 Subject: [Freeipa-users] Intranet password replication to DMZ Message-ID: I've been working on a project where I have multiple IPA domains which can't be connected due to scope and purpose of each domain. Ideally I would like to replicte a single user's password from a core domain server to a satellite ipa domain. I've learned that the password hash is not a traditional hash and cant be replicated without some additional work. My primary site is a multi-master and the satellite site has its own multi-master configuration. As an example I have an intranet server which hosts multiple users and a DMZ domain where a limited set of admins work. How can I replicate an intranet user from the inside to the DMZ? Any pointers or ideas would be helpful. Thanks Jessie -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.moyer at digitalreasoning.com Tue Aug 27 09:13:04 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Tue, 27 Aug 2013 05:13:04 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> Message-ID: Ok, so we tried to implement this again, and as soon as we put on a server that authenticates heavily the IPA came to it's knees again. This time I was able to watch it closely and try to troubleshoot a lot more, and also know exactly what server caused it (Mercurial with help of bamboo). This runs fine on a normal old openldap servers. The user is logging in very quickly and each time it logs in I can see in the logs that the krbLastsuccessfullogin parameter (or whatever it is called) is updated over and over and over in the changelog (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY quickly and then disappear fairly quickly as well. Issue 1: This is causing severe disk latency which obviously slows everything down wait times were around 25%+ Issue 2: These changes need to be replicated to my slave server thus adding to the mess My question is, why does the IPA server fail to keep up with the load when the openLDAP server didn't have an issue. Indexes? I'm running the following: CentOS release 6.4 (Final) 389-ds-base-1.2.11.15-20.el6_4.x86_64 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 ipa-python-3.0.0-26.el6_4.4.x86_64 ipa-admintools-3.0.0-26.el6_4.4.x86_64 ipa-pki-common-theme-9.0.3-7.el6.noarch python-iniparse-0.3.1-2.1.el6.noarch ipa-server-3.0.0-26.el6_4.4.x86_64 ipa-pki-ca-theme-9.0.3-7.el6.noarch ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 libipa_hbac-1.9.2-82.7.el6_4.x86_64 ipa-client-3.0.0-26.el6_4.4.x86_64 libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 So I've implemented this server anyway (against my better judgement with these issues and just made the user that logs into mercurial a local user instead of IPA). Also note before I did that for fun I implemented a RAM disk to put the change logs on, and that dropped the wait time to 0 (except bursts where it would raise to 30 to write the access log) but the CPU drove to 100% trying to keep up with the load. I have also killed the replication as well. Any help would be appreciated. Thanks, _____________________________________________________ John Moyer Director, IT Operations On Aug 7, 2013, at 4:08 PM, John Moyer wrote: > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > Digital Reasoning Systems, Inc. > John.Moyer at digitalreasoning.com > Office: 703.678.2311 > Mobile: 240.460.0023 > Fax: 703.678.2312 > www.digitalreasoning.com > > On Aug 6, 2013, at 10:57 AM, Rich Megginson wrote: > >> On 08/05/2013 09:17 PM, John Moyer wrote: >>> Hello, >>> >>> So I've been preparing my infrastructure for a big change from an older openldap system to a nice new IPA server. I have a redundant secondary server and snapshots taken daily. I populated all my user data into IPA, and gave the users a week to set a password. They all did this and the big switch was this past weekend. We had done previous tests on each server and it all worked. We switched this past weekend and it worked great. >>> >>> This morning a light load hit it (since I've only put a small fraction of our servers on it about 15) and the primary came to it's knees. >> >> What platform? What version of ipa? What version of 389-ds-base? >> >> What was the nature of the load? Search requests? Update requests? Updates from replication? >> >> The logconv.pl tool can be used to analyze the 389-ds-base access logs. >> >> During this time of the load, are there any errors in the errors log? >> >>> Processor spiked, and logs started to fill (didn't fill at this point). >> >> I'm not sure what you mean by "logs started to fill (didn't fill at this point)." >> >>> I then decided it's probably a glitch (I'm an optimist) so I restarted IPA services. They all restarted except for named which crashed (which then caused everything to stop). I looked and now the disk was full. >> >> Which directory contained the files that caused the disk to become full? /var/log? /var/lib? Somewhere else? >> >>> So I trash the logs (had no easy place to put them at the time which I regret now) and I restart the services again. >> >> What do you mean by "trash the logs"? >> >>> IPA fully crashes now (didn't even start the DIRSRV for my domain). >> >> Which component of IPA is crashing? If it is dirsrv that is refusing to start, is it crashing? What's in /var/log/dirsrv/slapd-*/errors? >> >> If it is crashing, we will need a core file and/or stack trace - see http://port389.org/wiki/FAQ#Debugging_Crashes >> >>> >>> So here are my questions: >>> >>> 1. Any idea what caused this? Any performance issues that have been seen? >> >> It could be almost anything given the above information. >> >>> >>> 2. Are the connection settings for IPA good out of the box? I ask because in RHDS (in the first versions I used) the default connection timeouts were a MAJOR issue, >> >> How so? Details? >> >>> I used to run a network of 400 servers and I had to set the time-outs to >30sec which made my servers run really really well, >> >> Exactly which timeout settings are you talking about? >> >>> but if I used the 60 min defaults they also would come to their knees. Is there a buried setting like this? (However, I must admit there didn't seem like there were a lot of connections like when I had the issue with the 400 servers years ago). >>> >>> Also is there an easy place to set log rotation settings? (If it's log rotate just let me know, I just don't want to step on an internal app rotate). >> >> IPA does not provide a GUI nor a command line utility for managing 389 logging settings. >> >> This document gives an example of how logs are managed using the RHDS GUI (not available when using IPA). >> >> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/Monitoring_Server_and_Database_Activity.html#types-of-log-files >> >> However, all of these correspond to settings which you can set via ldapmodify: >> >> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnconfig-nsslapd_accesslog_logexpirationtime_Access_Log_Expiration_Time >> >> There are several attributes which control access log rotation parameters. >> >> >>> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> >>> >>> _______________________________________________ >>> Freeipa-users mailing list >>> Freeipa-users at redhat.com >>> https://www.redhat.com/mailman/listinfo/freeipa-users >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bret.wortman at damascusgrp.com Tue Aug 27 11:24:57 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 27 Aug 2013 07:24:57 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> <5213734E.6070102@redhat.com> Message-ID: I managed to gather some data for Rich and others to review and updated a bug for them about a week ago. Now I am getting a lot of internal pressure to resolve our problems and get our infrastructure stable again. As of yesterday, our master IPA server would accept changes to DNS but isn't actually serving DNS, nor is it pushing data to any replicas. The replicas are acting as DNS servers but aren't getting any updates, nor can updates be made locally on them. Fortunately, we aren't adding users very often, but if anyone's password expires soon, I'm worried that I'll have an account lockout situation. So I'll ask again -- can anyone see a way to preserve just the actual DNS and authentication data within IPA while dumping its other data (replication and so on), restart it cleanly, verify it's working in all respects, and set up the replicas from scratch again? I'm hearing rumblings about going back to passwd files and host tables (which is what we were doing until about 12 months ago when I brought IPA in) and I'd really rather not go back to the stone ages.... Thanks! * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Tue, Aug 20, 2013 at 11:15 AM, Bret Wortman wrote: > If I were going to attempt to restore to an old backup, what > directories/files should I make sure to restore? I've got a backup script > that tars up: > > /usr/share/ipa > /usr/lib64/ipa > /var/lib/pia > /var/lib/ipa-client > /var/lib/dirsrv > /etc > > Is that enough to "roll back" to a few days ago before I started down this > path? I'm now seeing messages about having the max number of CleanAllRUV > tasks (4) and not being able to enqueue any more. So I'm really stuck now > and don't know how soon I can get the files requested over to Rich for > analysis. > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Tue, Aug 20, 2013 at 9:46 AM, Rich Megginson wrote: > >> On 08/20/2013 05:55 AM, Bret Wortman wrote: >> >> Okay, now I'm thinking I need to dump all my replicas and start them >> fresh. My /var/log/slapd-FOO-COM/errors is filled with messages like this: >> >> NSMMReplicationPlugin - changelog program - agmt="cn=meTogood1.foo.com" >> (good1:389): CSN 520a49640000001d0000 not found, we aren't as up to date, >> or we purged >> agmt="cn=meTogood1.foo.com" (good1:389) - Can't locate CSN >> 520a49640000001d0000 in the changelog (DB rc=-30988). The consumer may need >> to be reinitialized. >> >> I assume the "consumer" is the replica, right? At present, I have two >> replicas known to my master that are simply gone. Another is there but they >> can't talk. Three more have good communication but I'm getting errors like >> these. Is there a good, clean way to just clobber all the replicas and >> start over without trashing the DNS and other identity data that is inside >> my master and which *is* working? Deleting them from the master hasn't >> been working; it tends to hang the master's DNS and other services until I >> Ctrl-C out and "ipactl restart" it. >> >> I'm afraid to venture out without a net here and make things worse.... >> >> >> This looks like https://fedorahosted.org/389/ticket/47386 >> >> We've never been able to reproduce this in a "controlled" environment. >> >> The original reporter has been able to get this to work in some cases by >> restarting ipa (ipactl restart). >> >> Before you do that, would you be able to provide some information for me? >> >> On the supplier and consumer: >> ldapsearch -xLLL -D "cn=directory manager" -W -b "dc=FOO,dc=COM" >> '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' >> > ruv.ldif >> >> ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" >> '(objectclass=nsds5replicationagreement)' > agmt.ldif >> >> dbscan -f /var/lib/dirsrv/slapd-FOO-COM/cldb/*.db4 | head -200 > cldb.txt >> >> Be sure to obscure any sensitive data in ruv.ldif, agmt.ldif, and >> cldb.txt - you can either attach to >> https://fedorahosted.org/389/ticket/47386 or email to me directly. >> >> >> >> >> >> * >> * >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman < >> bret.wortman at damascusgrp.com> wrote: >> >>> On my master (where this error is occurring), I've got, in >>> /etc/hosts: >>> >>> 127.0.0.1 localhost localhost.localdomain >>> ::1 localhost localhost.localdomain >>> 1.2.3.4 ipamaster.foo.net ipamaster >>> >>> So that should be okay, right? >>> >>> # host ipamaster.foo.net >>> ipamaster.foo.net has address 1.2.3.4 >>> # host ipamaster >>> ipamaster.foo.net has address 1.2.3.4 >>> # host localhost >>> localhost has address 127.0.0.1 >>> localhost has IPv6 address ::1 >>> # >>> >>> I checked the other system (the one I can't connect to) to be safe, >>> and its /etc/hosts is similarly configured. It even has the master listed >>> with its correct IP address. >>> >>> >>> >>> * >>> * >>> *Bret Wortman* >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce wrote: >>> >>>> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: >>>> > So, any idea how to fix the Kerberos problem? >>>> > >>>> >>>> If your server is trying to get a tgt for ldap/localhost it probably >>>> means your /etc/hosts file is broken and has a line like this: >>>> >>>> 1.2.3.4 localhost my.real.name >>>> >>>> When GSSAPI tries to resolve my.realm.name it gets back that >>>> 'localhost' >>>> is the canonical name so it tries to get a TGT with that name and it >>>> fails. >>>> >>>> If /etc/host sis fine then the DNS server may be returning an IP address >>>> that later resolves to localhost again. >>>> >>>> To unbreak make sure that if you have your fully qualified name >>>> in /etc/hosts that it is on its own line pointing at the right IP >>>> address and where the FQDN name is the first in line: >>>> eg: >>>> >>>> this is ok: >>>> 1.2.3.4 server.full.name server >>>> >>>> this is not: >>>> 1.2.3.4 server server.full.name >>>> >>>> Simo. >>>> > >>>> > Bret Wortman >>>> > >>>> > >>>> > http://damascusgrp.com/ >>>> > >>>> > http://about.me/wortmanbret >>>> > >>>> > >>>> > >>>> > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman >>>> > wrote: >>>> > ...and I got the web UI, authentication and sudo back via: >>>> > >>>> > >>>> > # ipactl stop >>>> > # ipactl start >>>> > >>>> > >>>> > Not sure why that worked, but it did. I was grasping at >>>> > straws, honestly. >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > Bret Wortman >>>> > >>>> > >>>> > http://damascusgrp.com/ >>>> > >>>> > http://about.me/wortmanbret >>>> > >>>> > >>>> > >>>> > >>>> > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman >>>> > wrote: >>>> > Digging further, I think this log entry might be the >>>> > problem between the two servers that aren't talking: >>>> > >>>> > >>>> > slapd_ldap_sasl_interactive_bind - Error: could not >>>> > perform interactive bind for id[] mech [GSSAPI]: LDAP >>>> > error -2 (Local error) (SASL(-1): generic failure: >>>> > GSSAPI Error: Unspecified GSS failure. Minor code may >>>> > provide more information (Server >>>> > ldap/localhost at SPX.NET not found in Kerberos >>>> > database)) errno 2 (No such file or directory) >>>> > >>>> > >>>> > Did I build something incorrectly when that server was >>>> > set up originally? >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > Bret Wortman >>>> > >>>> > >>>> > http://damascusgrp.com/ >>>> > >>>> > http://about.me/wortmanbret >>>> > >>>> > >>>> > >>>> > >>>> > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman >>>> > wrote: >>>> > I ran it on a good master, against a bad one. >>>> > As in, I ran this command on my master IPA >>>> > node: >>>> > >>>> > >>>> > # ipa-replica-manage del --force bad1.foo.net >>>> > --cleanup >>>> > >>>> > >>>> > Was that wrong? I was trying to delete the bad >>>> > replica from the master, so I figured the >>>> > command needed to be run on the master. But >>>> > again, my master is now in a state where it's >>>> > not resolving DNS, user logins, or sudo at the >>>> > very least. >>>> > >>>> > >>>> > Oh, and I checked the node that it was >>>> > complaining about earlier. The network >>>> > connection to it is the pits, but it's there. >>>> > And it resolves. >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > Bret Wortman >>>> > >>>> > >>>> > http://damascusgrp.com/ >>>> > >>>> > http://about.me/wortmanbret >>>> > >>>> > >>>> > >>>> > On Mon, Aug 19, 2013 at 11:58 AM, Rob >>>> > Crittenden wrote: >>>> > Rob Crittenden wrote: >>>> > Bret Wortman wrote: >>>> > Well, my master ground >>>> > to a halt and wasn't >>>> > responding. I rebooted >>>> > the >>>> > system and now I can't >>>> > access the web UI or >>>> > ssh to the master >>>> > either. I >>>> > have console access >>>> > but that's it. >>>> > >>>> > The services all say >>>> > they're running, but >>>> > the web UI gives an >>>> > "Unknown >>>> > Error" dialog and ssh >>>> > fails with >>>> > >>>> "ssh_exchange_identification: >>>> > Connection closed by >>>> > remote host" whenever >>>> > I try to ssh to >>>> > ipamaster. I >>>> > think something has >>>> > gone really wrong >>>> > inside my master. Any >>>> > ideas? Even >>>> > after the reboot, >>>> > --cleanup isn't >>>> > helping and just >>>> > hangs. >>>> > >>>> > The logfiles end (as >>>> > of the time I ^C'd the >>>> > process) with: >>>> > >>>> > NSMMReplicationPlugin >>>> > - >>>> > agmt="cn= >>>> meTogood3.spx.net >>>> > < >>>> http://meTogood3.spx.net>" (good3:389): Replication bind with GSSAPI >>>> > auth failed: LDAP >>>> > error -2 (Local error) >>>> > (SASL(-1): generic >>>> > failure: >>>> > GSSAPI Error: >>>> > Unspecified GSS >>>> > failure. Minor code >>>> > may provide more >>>> > information (Cannot >>>> > determine realm for >>>> > numeric host address)) >>>> > NSMMReplicationPlugin >>>> > - CleanAllRUV Task: >>>> > Replica not online >>>> > (agmt="cn= >>>> meTogood3.foo.net " (good3:389)) >>>> > NSMMReplicationPlugin >>>> > - CleanAllRUV Task: >>>> > Not all replicas >>>> > online, >>>> > retrying in 160 >>>> > seconds..., >>>> > >>>> > So it looks like it's >>>> > having trouble talking >>>> > with one of my >>>> > replicas and >>>> > is doggedly trying to >>>> > get the job done. Any >>>> > idea how to get the >>>> > master >>>> > back working again >>>> > while I troubleshoot >>>> > this connectivity >>>> > issue? >>>> > >>>> > That suggests a DNS problem, >>>> > and it might explain ssh as >>>> > well depending >>>> > on your configuration. >>>> > >>>> > >>>> > To be clear, you ran --cleanup against >>>> > one of the bad masters, not a good >>>> > one, right? >>>> > >>>> > rob >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > Freeipa-users mailing list >>>> > Freeipa-users at redhat.com >>>> > https://www.redhat.com/mailman/listinfo/freeipa-users >>>> >>>> >>>> -- >>>> Simo Sorce * Red Hat, Inc * New York >>>> >>>> >>> >>> >> >> >> _______________________________________________ >> Freeipa-users mailing listFreeipa-users at redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Tue Aug 27 13:32:53 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 27 Aug 2013 09:32:53 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <52123585.2070503@redhat.com> <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> <5213734E.6070102@redhat.com> Message-ID: Here's a bit more about what I'm seeing today. My master _is_ serving some DNS, but it appears that it's only serving those zones that it knew about before all this trouble started 7-10 days ago. In particular, it can only do reverse DNS on one zone (its own), but can't serve reverse DNS for any other zones, even those that are in its database and visible (and enabled) from the web UI. # nslookup 4.3.2.1 ipamaster ;; connection timed out; no servers could be reached # nslookup 6.5.2.1 ipamaster Server: ipamaster Address: 10.9.2.1 1.2.5.6.in-addr.arpa name = host1.foo.com. # Is this something that's easily rectified? The logs aren't giving me anything obviously wrong -- nothing in /var/log/dirsrv-FOO-COM/errors seems significant; just the same CLEANALLRUV errors I've been seeing for the past week. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Tue, Aug 27, 2013 at 7:24 AM, Bret Wortman wrote: > I managed to gather some data for Rich and others to review and updated a > bug for them about a week ago. Now I am getting a lot of internal pressure > to resolve our problems and get our infrastructure stable again. As of > yesterday, our master IPA server would accept changes to DNS but isn't > actually serving DNS, nor is it pushing data to any replicas. The replicas > are acting as DNS servers but aren't getting any updates, nor can updates > be made locally on them. Fortunately, we aren't adding users very often, > but if anyone's password expires soon, I'm worried that I'll have an > account lockout situation. > > So I'll ask again -- can anyone see a way to preserve just the actual DNS > and authentication data within IPA while dumping its other data > (replication and so on), restart it cleanly, verify it's working in all > respects, and set up the replicas from scratch again? I'm hearing rumblings > about going back to passwd files and host tables (which is what we were > doing until about 12 months ago when I brought IPA in) and I'd really > rather not go back to the stone ages.... > > Thanks! > > > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Tue, Aug 20, 2013 at 11:15 AM, Bret Wortman < > bret.wortman at damascusgrp.com> wrote: > >> If I were going to attempt to restore to an old backup, what >> directories/files should I make sure to restore? I've got a backup script >> that tars up: >> >> /usr/share/ipa >> /usr/lib64/ipa >> /var/lib/pia >> /var/lib/ipa-client >> /var/lib/dirsrv >> /etc >> >> Is that enough to "roll back" to a few days ago before I started down >> this path? I'm now seeing messages about having the max number of >> CleanAllRUV tasks (4) and not being able to enqueue any more. So I'm really >> stuck now and don't know how soon I can get the files requested over to >> Rich for analysis. >> >> >> * >> * >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Tue, Aug 20, 2013 at 9:46 AM, Rich Megginson wrote: >> >>> On 08/20/2013 05:55 AM, Bret Wortman wrote: >>> >>> Okay, now I'm thinking I need to dump all my replicas and start them >>> fresh. My /var/log/slapd-FOO-COM/errors is filled with messages like this: >>> >>> NSMMReplicationPlugin - changelog program - agmt="cn=meTogood1.foo.com" >>> (good1:389): CSN 520a49640000001d0000 not found, we aren't as up to date, >>> or we purged >>> agmt="cn=meTogood1.foo.com" (good1:389) - Can't locate CSN >>> 520a49640000001d0000 in the changelog (DB rc=-30988). The consumer may need >>> to be reinitialized. >>> >>> I assume the "consumer" is the replica, right? At present, I have two >>> replicas known to my master that are simply gone. Another is there but they >>> can't talk. Three more have good communication but I'm getting errors like >>> these. Is there a good, clean way to just clobber all the replicas and >>> start over without trashing the DNS and other identity data that is inside >>> my master and which *is* working? Deleting them from the master hasn't >>> been working; it tends to hang the master's DNS and other services until I >>> Ctrl-C out and "ipactl restart" it. >>> >>> I'm afraid to venture out without a net here and make things worse.... >>> >>> >>> This looks like https://fedorahosted.org/389/ticket/47386 >>> >>> We've never been able to reproduce this in a "controlled" environment. >>> >>> The original reporter has been able to get this to work in some cases by >>> restarting ipa (ipactl restart). >>> >>> Before you do that, would you be able to provide some information for me? >>> >>> On the supplier and consumer: >>> ldapsearch -xLLL -D "cn=directory manager" -W -b "dc=FOO,dc=COM" >>> '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' >>> > ruv.ldif >>> >>> ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" >>> '(objectclass=nsds5replicationagreement)' > agmt.ldif >>> >>> dbscan -f /var/lib/dirsrv/slapd-FOO-COM/cldb/*.db4 | head -200 > cldb.txt >>> >>> Be sure to obscure any sensitive data in ruv.ldif, agmt.ldif, and >>> cldb.txt - you can either attach to >>> https://fedorahosted.org/389/ticket/47386 or email to me directly. >>> >>> >>> >>> >>> >>> * >>> * >>> *Bret Wortman* >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman < >>> bret.wortman at damascusgrp.com> wrote: >>> >>>> On my master (where this error is occurring), I've got, in >>>> /etc/hosts: >>>> >>>> 127.0.0.1 localhost localhost.localdomain >>>> ::1 localhost localhost.localdomain >>>> 1.2.3.4 ipamaster.foo.net ipamaster >>>> >>>> So that should be okay, right? >>>> >>>> # host ipamaster.foo.net >>>> ipamaster.foo.net has address 1.2.3.4 >>>> # host ipamaster >>>> ipamaster.foo.net has address 1.2.3.4 >>>> # host localhost >>>> localhost has address 127.0.0.1 >>>> localhost has IPv6 address ::1 >>>> # >>>> >>>> I checked the other system (the one I can't connect to) to be safe, >>>> and its /etc/hosts is similarly configured. It even has the master listed >>>> with its correct IP address. >>>> >>>> >>>> >>>> * >>>> * >>>> *Bret Wortman* >>>> >>>> http://damascusgrp.com/ >>>> http://about.me/wortmanbret >>>> >>>> >>>> On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce wrote: >>>> >>>>> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: >>>>> > So, any idea how to fix the Kerberos problem? >>>>> > >>>>> >>>>> If your server is trying to get a tgt for ldap/localhost it probably >>>>> means your /etc/hosts file is broken and has a line like this: >>>>> >>>>> 1.2.3.4 localhost my.real.name >>>>> >>>>> When GSSAPI tries to resolve my.realm.name it gets back that >>>>> 'localhost' >>>>> is the canonical name so it tries to get a TGT with that name and it >>>>> fails. >>>>> >>>>> If /etc/host sis fine then the DNS server may be returning an IP >>>>> address >>>>> that later resolves to localhost again. >>>>> >>>>> To unbreak make sure that if you have your fully qualified name >>>>> in /etc/hosts that it is on its own line pointing at the right IP >>>>> address and where the FQDN name is the first in line: >>>>> eg: >>>>> >>>>> this is ok: >>>>> 1.2.3.4 server.full.name server >>>>> >>>>> this is not: >>>>> 1.2.3.4 server server.full.name >>>>> >>>>> Simo. >>>>> > >>>>> > Bret Wortman >>>>> > >>>>> > >>>>> > http://damascusgrp.com/ >>>>> > >>>>> > http://about.me/wortmanbret >>>>> > >>>>> > >>>>> > >>>>> > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman >>>>> > wrote: >>>>> > ...and I got the web UI, authentication and sudo back via: >>>>> > >>>>> > >>>>> > # ipactl stop >>>>> > # ipactl start >>>>> > >>>>> > >>>>> > Not sure why that worked, but it did. I was grasping at >>>>> > straws, honestly. >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Bret Wortman >>>>> > >>>>> > >>>>> > http://damascusgrp.com/ >>>>> > >>>>> > http://about.me/wortmanbret >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > On Mon, Aug 19, 2013 at 12:18 PM, Bret Wortman >>>>> > wrote: >>>>> > Digging further, I think this log entry might be the >>>>> > problem between the two servers that aren't talking: >>>>> > >>>>> > >>>>> > slapd_ldap_sasl_interactive_bind - Error: could not >>>>> > perform interactive bind for id[] mech [GSSAPI]: LDAP >>>>> > error -2 (Local error) (SASL(-1): generic failure: >>>>> > GSSAPI Error: Unspecified GSS failure. Minor code may >>>>> > provide more information (Server >>>>> > ldap/localhost at SPX.NET not found in Kerberos >>>>> > database)) errno 2 (No such file or directory) >>>>> > >>>>> > >>>>> > Did I build something incorrectly when that server >>>>> was >>>>> > set up originally? >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Bret Wortman >>>>> > >>>>> > >>>>> > http://damascusgrp.com/ >>>>> > >>>>> > http://about.me/wortmanbret >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > On Mon, Aug 19, 2013 at 12:02 PM, Bret Wortman >>>>> > wrote: >>>>> > I ran it on a good master, against a bad one. >>>>> > As in, I ran this command on my master IPA >>>>> > node: >>>>> > >>>>> > >>>>> > # ipa-replica-manage del --force >>>>> bad1.foo.net >>>>> > --cleanup >>>>> > >>>>> > >>>>> > Was that wrong? I was trying to delete the >>>>> bad >>>>> > replica from the master, so I figured the >>>>> > command needed to be run on the master. But >>>>> > again, my master is now in a state where it's >>>>> > not resolving DNS, user logins, or sudo at >>>>> the >>>>> > very least. >>>>> > >>>>> > >>>>> > Oh, and I checked the node that it was >>>>> > complaining about earlier. The network >>>>> > connection to it is the pits, but it's there. >>>>> > And it resolves. >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > Bret Wortman >>>>> > >>>>> > >>>>> > http://damascusgrp.com/ >>>>> > >>>>> > http://about.me/wortmanbret >>>>> > >>>>> > >>>>> > >>>>> > On Mon, Aug 19, 2013 at 11:58 AM, Rob >>>>> > Crittenden wrote: >>>>> > Rob Crittenden wrote: >>>>> > Bret Wortman wrote: >>>>> > Well, my master >>>>> ground >>>>> > to a halt and wasn't >>>>> > responding. I >>>>> rebooted >>>>> > the >>>>> > system and now I >>>>> can't >>>>> > access the web UI or >>>>> > ssh to the master >>>>> > either. I >>>>> > have console access >>>>> > but that's it. >>>>> > >>>>> > The services all say >>>>> > they're running, but >>>>> > the web UI gives an >>>>> > "Unknown >>>>> > Error" dialog and ssh >>>>> > fails with >>>>> > >>>>> "ssh_exchange_identification: >>>>> > Connection closed by >>>>> > remote host" whenever >>>>> > I try to ssh to >>>>> > ipamaster. I >>>>> > think something has >>>>> > gone really wrong >>>>> > inside my master. Any >>>>> > ideas? Even >>>>> > after the reboot, >>>>> > --cleanup isn't >>>>> > helping and just >>>>> > hangs. >>>>> > >>>>> > The logfiles end (as >>>>> > of the time I ^C'd >>>>> the >>>>> > process) with: >>>>> > >>>>> > NSMMReplicationPlugin >>>>> > - >>>>> > agmt="cn= >>>>> meTogood3.spx.net >>>>> > < >>>>> http://meTogood3.spx.net>" (good3:389): Replication bind with GSSAPI >>>>> > auth failed: LDAP >>>>> > error -2 (Local >>>>> error) >>>>> > (SASL(-1): generic >>>>> > failure: >>>>> > GSSAPI Error: >>>>> > Unspecified GSS >>>>> > failure. Minor code >>>>> > may provide more >>>>> > information (Cannot >>>>> > determine realm for >>>>> > numeric host >>>>> address)) >>>>> > NSMMReplicationPlugin >>>>> > - CleanAllRUV Task: >>>>> > Replica not online >>>>> > (agmt="cn= >>>>> meTogood3.foo.net " (good3:389)) >>>>> > NSMMReplicationPlugin >>>>> > - CleanAllRUV Task: >>>>> > Not all replicas >>>>> > online, >>>>> > retrying in 160 >>>>> > seconds..., >>>>> > >>>>> > So it looks like it's >>>>> > having trouble >>>>> talking >>>>> > with one of my >>>>> > replicas and >>>>> > is doggedly trying to >>>>> > get the job done. Any >>>>> > idea how to get the >>>>> > master >>>>> > back working again >>>>> > while I troubleshoot >>>>> > this connectivity >>>>> > issue? >>>>> > >>>>> > That suggests a DNS problem, >>>>> > and it might explain ssh as >>>>> > well depending >>>>> > on your configuration. >>>>> > >>>>> > >>>>> > To be clear, you ran --cleanup >>>>> against >>>>> > one of the bad masters, not a good >>>>> > one, right? >>>>> > >>>>> > rob >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > Freeipa-users mailing list >>>>> > Freeipa-users at redhat.com >>>>> > https://www.redhat.com/mailman/listinfo/freeipa-users >>>>> >>>>> >>>>> -- >>>>> Simo Sorce * Red Hat, Inc * New York >>>>> >>>>> >>>> >>>> >>> >>> >>> _______________________________________________ >>> Freeipa-users mailing listFreeipa-users at redhat.comhttps://www.redhat.com/mailman/listinfo/freeipa-users >>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Tue Aug 27 14:05:59 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 27 Aug 2013 10:05:59 -0400 Subject: [Freeipa-users] Intranet password replication to DMZ In-Reply-To: References: Message-ID: <521CB247.1020404@redhat.com> Jessie Floyd wrote: > I've been working on a project where I have multiple IPA domains which > can't be connected due to scope and purpose of each domain. Ideally I > would like to replicte a single user's password from a core domain > server to a satellite ipa domain. I've learned that the password hash > is not a traditional hash and cant be replicated without some additional > work. My primary site is a multi-master and the satellite site has its > own multi-master configuration. As an example I have an intranet server > which hosts multiple users and a DMZ domain where a limited set of > admins work. How can I replicate an intranet user from the inside to > the DMZ? Any pointers or ideas would be helpful. I'm not entirely clear what it is you want/need to do. Do you want to set up some sort of fractional replication that replicates only passwords, and the raw hashes at that? That would do you no good when it comes to Kerberos. rob From rcritten at redhat.com Tue Aug 27 14:14:18 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 27 Aug 2013 10:14:18 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> Message-ID: <521CB43A.40104@redhat.com> John Moyer wrote: > Ok, so we tried to implement this again, and as soon as we put on a > server that authenticates heavily the IPA came to it's knees again. > This time I was able to watch it closely and try to troubleshoot a lot > more, and also know exactly what server caused it (Mercurial with help > of bamboo). This runs fine on a normal old openldap servers. The > user is logging in very quickly and each time it logs in I can see in > the logs that the krbLastsuccessfullogin parameter (or whatever it is > called) is updated over and over and over in the changelog > (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY > quickly and then disappear fairly quickly as well. > > Issue 1: This is causing severe disk latency which obviously slows > everything down wait times were around 25%+ > Issue 2: These changes need to be replicated to my slave server thus > adding to the mess > > > My question is, why does the IPA server fail to keep up with the load > when the openLDAP server didn't have an issue. Indexes? > > > I'm running the following: > > CentOS release 6.4 (Final) > 389-ds-base-1.2.11.15-20.el6_4.x86_64 > 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 > ipa-python-3.0.0-26.el6_4.4.x86_64 > ipa-admintools-3.0.0-26.el6_4.4.x86_64 > ipa-pki-common-theme-9.0.3-7.el6.noarch > python-iniparse-0.3.1-2.1.el6.noarch > ipa-server-3.0.0-26.el6_4.4.x86_64 > ipa-pki-ca-theme-9.0.3-7.el6.noarch > ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 > libipa_hbac-1.9.2-82.7.el6_4.x86_64 > ipa-client-3.0.0-26.el6_4.4.x86_64 > libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 > > > So I've implemented this server anyway (against my better judgement with > these issues and just made the user that logs into mercurial a local > user instead of IPA). > > Also note before I did that for fun I implemented a RAM disk to put the > change logs on, and that dropped the wait time to 0 (except bursts where > it would raise to 30 to write the access log) but the CPU drove to 100% > trying to keep up with the load. I have also killed the replication as > well. > > Any help would be appreciated. > krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. You can confirm that they are excluded by searching the agreements: $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal They should look like: nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount rob From rcritten at redhat.com Tue Aug 27 14:23:59 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 27 Aug 2013 10:23:59 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: References: <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> <5213734E.6070102@redhat.com> Message-ID: <521CB67F.3030408@redhat.com> Bret Wortman wrote: > Here's a bit more about what I'm seeing today. > > My master _is_ serving some DNS, but it appears that it's only serving > those zones that it knew about before all this trouble started 7-10 days > ago. In particular, it can only do reverse DNS on one zone (its own), > but can't serve reverse DNS for any other zones, even those that are in > its database and visible (and enabled) from the web UI. > > # nslookup 4.3.2.1 ipamaster > ;; connection timed out; no servers could be reached > # nslookup 6.5.2.1 ipamaster > Server: ipamaster > Address: 10.9.2.1 > > 1.2.5.6.in-addr.arpa > name = > host1.foo.com . > # > > Is this something that's easily rectified? The logs aren't giving me > anything obviously wrong -- nothing in /var/log/dirsrv-FOO-COM/errors > seems significant; just the same CLEANALLRUV errors I've been seeing for > the past week. You might try restarting named. At a minimum it is going to log all the zones it manages so you can compare what it thinks it has vs what IPA has. A pattern might emerge. You can delete an existing replica and re-create it, but with the 389-ds errors I'm not sure what the repercussions would be, if any. You could end up with more dead replicas. It could be that all the RUV you have are because the deletions were done prior to 389-ds adding support for CLEANALLRUV (and it getting into IPA). rob > > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Tue, Aug 27, 2013 at 7:24 AM, Bret Wortman > > wrote: > > I managed to gather some data for Rich and others to review and > updated a bug for them about a week ago. Now I am getting a lot of > internal pressure to resolve our problems and get our infrastructure > stable again. As of yesterday, our master IPA server would accept > changes to DNS but isn't actually serving DNS, nor is it pushing > data to any replicas. The replicas are acting as DNS servers but > aren't getting any updates, nor can updates be made locally on them. > Fortunately, we aren't adding users very often, but if anyone's > password expires soon, I'm worried that I'll have an account lockout > situation. > > So I'll ask again -- can anyone see a way to preserve just the > actual DNS and authentication data within IPA while dumping its > other data (replication and so on), restart it cleanly, verify it's > working in all respects, and set up the replicas from scratch again? > I'm hearing rumblings about going back to passwd files and host > tables (which is what we were doing until about 12 months ago when I > brought IPA in) and I'd really rather not go back to the stone ages.... > > Thanks! > > > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Tue, Aug 20, 2013 at 11:15 AM, Bret Wortman > > > wrote: > > If I were going to attempt to restore to an old backup, what > directories/files should I make sure to restore? I've got a > backup script that tars up: > > /usr/share/ipa > /usr/lib64/ipa > /var/lib/pia > /var/lib/ipa-client > /var/lib/dirsrv > /etc > > Is that enough to "roll back" to a few days ago before I started > down this path? I'm now seeing messages about having the max > number of CleanAllRUV tasks (4) and not being able to enqueue > any more. So I'm really stuck now and don't know how soon I can > get the files requested over to Rich for analysis. > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Tue, Aug 20, 2013 at 9:46 AM, Rich Megginson > > wrote: > > On 08/20/2013 05:55 AM, Bret Wortman wrote: >> Okay, now I'm thinking I need to dump all my replicas and >> start them fresh. My /var/log/slapd-FOO-COM/errors is >> filled with messages like this: >> >> NSMMReplicationPlugin - changelog program - >> agmt="cn=meTogood1.foo.com " >> (good1:389): CSN 520a49640000001d0000 not found, we aren't >> as up to date, or we purged >> agmt="cn=meTogood1.foo.com " >> (good1:389) - Can't locate CSN 520a49640000001d0000 in the >> changelog (DB rc=-30988). The consumer may need to be >> reinitialized. >> >> I assume the "consumer" is the replica, right? At present, >> I have two replicas known to my master that are simply >> gone. Another is there but they can't talk. Three more >> have good communication but I'm getting errors like these. >> Is there a good, clean way to just clobber all the >> replicas and start over without trashing the DNS and other >> identity data that is inside my master and which /is/ >> working? Deleting them from the master hasn't been >> working; it tends to hang the master's DNS and other >> services until I Ctrl-C out and "ipactl restart" it. >> >> I'm afraid to venture out without a net here and make >> things worse.... > > This looks like https://fedorahosted.org/389/ticket/47386 > > We've never been able to reproduce this in a "controlled" > environment. > > The original reporter has been able to get this to work in > some cases by restarting ipa (ipactl restart). > > Before you do that, would you be able to provide some > information for me? > > On the supplier and consumer: > ldapsearch -xLLL -D "cn=directory manager" -W -b > "dc=FOO,dc=COM" > '(&(objectclass=nstombstone)(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff))' > > ruv.ldif > > ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" > '(objectclass=nsds5replicationagreement)' > agmt.ldif > > dbscan -f /var/lib/dirsrv/slapd-FOO-COM/cldb/*.db4 | head > -200 > cldb.txt > > Be sure to obscure any sensitive data in ruv.ldif, > agmt.ldif, and cldb.txt - you can either attach to > https://fedorahosted.org/389/ticket/47386 or email to me > directly. > > >> >> >> >> _ >> _ >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman >> > > wrote: >> >> On my master (where this error is occurring), I've >> got, in /etc/hosts: >> >> 127.0.0.1 localhost localhost.localdomain >> ::1 localhost localhost.localdomain >> 1.2.3.4 ipamaster.foo.net >> ipamaster >> >> So that should be okay, right? >> >> # host ipamaster.foo.net >> ipamaster.foo.net has >> address 1.2.3.4 >> # host ipamaster >> ipamaster.foo.net has >> address 1.2.3.4 >> # host localhost >> localhost has address 127.0.0.1 >> localhost has IPv6 address ::1 >> # >> >> I checked the other system (the one I can't connect >> to) to be safe, and its /etc/hosts is similarly >> configured. It even has the master listed with its >> correct IP address. >> >> >> >> _ >> _ >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce >> > wrote: >> >> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman wrote: >> > So, any idea how to fix the Kerberos problem? >> > >> >> If your server is trying to get a tgt for >> ldap/localhost it probably >> means your /etc/hosts file is broken and has a >> line like this: >> >> 1.2.3.4 localhost my.real.name >> >> When GSSAPI tries to resolve my.realm.name >> it gets back that 'localhost' >> is the canonical name so it tries to get a TGT >> with that name and it >> fails. >> >> If /etc/host sis fine then the DNS server may be >> returning an IP address >> that later resolves to localhost again. >> >> To unbreak make sure that if you have your fully >> qualified name >> in /etc/hosts that it is on its own line pointing >> at the right IP >> address and where the FQDN name is the first in line: >> eg: >> >> this is ok: >> 1.2.3.4 server.full.name >> server >> >> this is not: >> 1.2.3.4 server server.full.name >> >> >> Simo. >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman >> > > > wrote: >> > ...and I got the web UI, authentication >> and sudo back via: >> > >> > >> > # ipactl stop >> > # ipactl start >> > >> > >> > Not sure why that worked, but it did. I >> was grasping at >> > straws, honestly. >> > >> > >> > >> > >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > >> > On Mon, Aug 19, 2013 at 12:18 PM, Bret >> Wortman >> > > > wrote: >> > Digging further, I think this >> log entry might be the >> > problem between the two servers >> that aren't talking: >> > >> > >> > slapd_ldap_sasl_interactive_bind - Error: could not >> > perform interactive bind for >> id[] mech [GSSAPI]: LDAP >> > error -2 (Local error) >> (SASL(-1): generic failure: >> > GSSAPI Error: Unspecified GSS >> failure. Minor code may >> > provide more information (Server >> > ldap/localhost at SPX.NET >> not found in Kerberos >> > database)) errno 2 (No such file >> or directory) >> > >> > >> > Did I build something >> incorrectly when that server was >> > set up originally? >> > >> > >> > >> > >> > >> > >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > >> > On Mon, Aug 19, 2013 at 12:02 >> PM, Bret Wortman >> > > > wrote: >> > I ran it on a good >> master, against a bad one. >> > As in, I ran this >> command on my master IPA >> > node: >> > >> > >> > # ipa-replica-manage del >> --force bad1.foo.net >> > --cleanup >> > >> > >> > Was that wrong? I was >> trying to delete the bad >> > replica from the master, >> so I figured the >> > command needed to be run >> on the master. But >> > again, my master is now >> in a state where it's >> > not resolving DNS, user >> logins, or sudo at the >> > very least. >> > >> > >> > Oh, and I checked the >> node that it was >> > complaining about >> earlier. The network >> > connection to it is the >> pits, but it's there. >> > And it resolves. >> > >> > >> > >> > >> > >> > Bret Wortman >> > >> > >> > http://damascusgrp.com/ >> > >> > http://about.me/wortmanbret >> > >> > >> > >> > On Mon, Aug 19, 2013 at >> 11:58 AM, Rob >> > Crittenden > > wrote: >> > Rob Crittenden wrote: >> > Bret Wortman wrote: >> > Well, my master ground >> > to a halt and wasn't >> > responding. I rebooted >> > the >> > system and now I can't >> > access the web UI or >> > ssh to the master >> > either. I >> > have console access >> > but that's it. >> > >> > The services all say >> > they're running, but >> > the web UI gives an >> > "Unknown >> > Error" dialog and ssh >> > fails with >> > "ssh_exchange_identification: >> > Connection closed by >> > remote host" whenever >> > I try to ssh to >> > ipamaster. I >> > think something has >> > gone really wrong >> > inside my master. Any >> > ideas? Even >> > after the reboot, >> > --cleanup isn't >> > helping and just >> > hangs. >> > >> > The logfiles end (as >> > of the time I ^C'd the >> > process) with: >> > >> > NSMMReplicationPlugin >> > - >> > agmt="cn=meTogood3.spx.net >> >> > " >> (good3:389): Replication bind with GSSAPI >> > auth failed: LDAP >> > error -2 (Local error) >> > (SASL(-1): generic >> > failure: >> > GSSAPI Error: >> > Unspecified GSS >> > failure. Minor code >> > may provide more >> > information (Cannot >> > determine realm for >> > numeric host address)) >> > NSMMReplicationPlugin >> > - CleanAllRUV Task: >> > Replica not online >> > (agmt="cn=meTogood3.foo.net >> >> " (good3:389)) >> > NSMMReplicationPlugin >> > - CleanAllRUV Task: >> > Not all replicas >> > online, >> > retrying in 160 >> > seconds..., >> > >> > So it looks like it's >> > having trouble talking >> > with one of my >> > replicas and >> > is doggedly trying to >> > get the job done. Any >> > idea how to get the >> > master >> > back working again >> > while I troubleshoot >> > this connectivity >> > issue? >> > >> > That suggests a DNS problem, >> > and it might explain ssh as >> > well depending >> > on your configuration. >> > >> > >> > To be clear, you ran --cleanup against >> > one of the bad masters, not a good >> > one, right? >> > >> > rob >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > Freeipa-users mailing list >> > Freeipa-users at redhat.com >> >> > >> https://www.redhat.com/mailman/listinfo/freeipa-users >> >> >> -- >> Simo Sorce * Red Hat, Inc * New York >> >> >> >> >> >> >> _______________________________________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-users > > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From john.moyer at digitalreasoning.com Tue Aug 27 14:36:39 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Tue, 27 Aug 2013 10:36:39 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <521CB43A.40104@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> Message-ID: That looks like the output I just got shown below: dn: cn=mapping tree,cn=config dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ 2Cdc\3Dcom,cn=mapping tree,cn=config nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts uccessfulauth krblastfailedauth krbloginfailedcount Thanks, _____________________________________________________ John Moyer Director, IT Operations On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: > John Moyer wrote: >> Ok, so we tried to implement this again, and as soon as we put on a >> server that authenticates heavily the IPA came to it's knees again. >> This time I was able to watch it closely and try to troubleshoot a lot >> more, and also know exactly what server caused it (Mercurial with help >> of bamboo). This runs fine on a normal old openldap servers. The >> user is logging in very quickly and each time it logs in I can see in >> the logs that the krbLastsuccessfullogin parameter (or whatever it is >> called) is updated over and over and over in the changelog >> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >> quickly and then disappear fairly quickly as well. >> >> Issue 1: This is causing severe disk latency which obviously slows >> everything down wait times were around 25%+ >> Issue 2: These changes need to be replicated to my slave server thus >> adding to the mess >> >> >> My question is, why does the IPA server fail to keep up with the load >> when the openLDAP server didn't have an issue. Indexes? >> >> >> I'm running the following: >> >> CentOS release 6.4 (Final) >> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >> ipa-python-3.0.0-26.el6_4.4.x86_64 >> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >> ipa-pki-common-theme-9.0.3-7.el6.noarch >> python-iniparse-0.3.1-2.1.el6.noarch >> ipa-server-3.0.0-26.el6_4.4.x86_64 >> ipa-pki-ca-theme-9.0.3-7.el6.noarch >> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >> ipa-client-3.0.0-26.el6_4.4.x86_64 >> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >> >> >> So I've implemented this server anyway (against my better judgement with >> these issues and just made the user that logs into mercurial a local >> user instead of IPA). >> >> Also note before I did that for fun I implemented a RAM disk to put the >> change logs on, and that dropped the wait time to 0 (except bursts where >> it would raise to 30 to write the access log) but the CPU drove to 100% >> trying to keep up with the load. I have also killed the replication as >> well. >> >> Any help would be appreciated. >> > > krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. > > You can confirm that they are excluded by searching the agreements: > > $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal > > They should look like: > > nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount > > nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount > > rob -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bret.wortman at damascusgrp.com Tue Aug 27 14:45:03 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Tue, 27 Aug 2013 10:45:03 -0400 Subject: [Freeipa-users] Replication woes In-Reply-To: <521CB67F.3030408@redhat.com> References: <521239E0.60309@redhat.com> <521240A4.8040704@redhat.com> <1376935374.2814.130.camel@willson.li.ssimo.org> <5213734E.6070102@redhat.com> <521CB67F.3030408@redhat.com> Message-ID: Thanks, Rob. I restarted named and I can see that it's only loading these (timestamps omitted for clarity): zone 0.in-addr.arpa/IN: loaded serial 0 zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.wholebunchofzeros.0.ip6.arpa/IN:loaded serial 0 zone localhost/IN: loaded serial 0 zone localhost.localdomain/IN: loaded serial 0 all zones loaded running zone foo.net/IN: sending notifies (serial 2012102269) zone 3.2.1.in-addr.arpa/IN: sending notifies (serial 2012101940) Looking at some earlier /var/log/messages files, it used to emit "sending notifies" messages for all the other zones as well -- what might have changed to restrict it to just these two? To your other question, we didn't try deleting replicas until after upgrading the master to IPA-3.1.5-1.fc18. The replicas _are_ still at a lower level; they began having problems when one crashed and needed to be rebuilt on the new baseline, which wasn't possible because the original couldn't be deleted because of a communication problem with a third replica. And the dominoes began to fall.... Hey, it's still a better system than what we had before! I think we just hit a real oddball situation which is making recovery difficult. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Tue, Aug 27, 2013 at 10:23 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> Here's a bit more about what I'm seeing today. >> >> My master _is_ serving some DNS, but it appears that it's only serving >> those zones that it knew about before all this trouble started 7-10 days >> ago. In particular, it can only do reverse DNS on one zone (its own), >> but can't serve reverse DNS for any other zones, even those that are in >> its database and visible (and enabled) from the web UI. >> >> # nslookup 4.3.2.1 ipamaster >> ;; connection timed out; no servers could be reached >> # nslookup 6.5.2.1 ipamaster >> Server: ipamaster >> Address: 10.9.2.1 >> >> 1.2.5.6.in-addr.arpa >> >> **> name = >> host1.foo.com . >> >> # >> >> Is this something that's easily rectified? The logs aren't giving me >> anything obviously wrong -- nothing in /var/log/dirsrv-FOO-COM/errors >> seems significant; just the same CLEANALLRUV errors I've been seeing for >> the past week. >> > > You might try restarting named. At a minimum it is going to log all the > zones it manages so you can compare what it thinks it has vs what IPA has. > A pattern might emerge. > > You can delete an existing replica and re-create it, but with the 389-ds > errors I'm not sure what the repercussions would be, if any. You could end > up with more dead replicas. It could be that all the RUV you have are > because the deletions were done prior to 389-ds adding support for > CLEANALLRUV (and it getting into IPA). > > rob > >> >> >> >> _ >> _ >> *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Tue, Aug 27, 2013 at 7:24 AM, Bret Wortman >> >> >> wrote: >> >> I managed to gather some data for Rich and others to review and >> updated a bug for them about a week ago. Now I am getting a lot of >> internal pressure to resolve our problems and get our infrastructure >> stable again. As of yesterday, our master IPA server would accept >> changes to DNS but isn't actually serving DNS, nor is it pushing >> data to any replicas. The replicas are acting as DNS servers but >> aren't getting any updates, nor can updates be made locally on them. >> Fortunately, we aren't adding users very often, but if anyone's >> password expires soon, I'm worried that I'll have an account lockout >> situation. >> >> So I'll ask again -- can anyone see a way to preserve just the >> actual DNS and authentication data within IPA while dumping its >> other data (replication and so on), restart it cleanly, verify it's >> working in all respects, and set up the replicas from scratch again? >> I'm hearing rumblings about going back to passwd files and host >> tables (which is what we were doing until about 12 months ago when I >> brought IPA in) and I'd really rather not go back to the stone >> ages.... >> >> Thanks! >> >> >> >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Tue, Aug 20, 2013 at 11:15 AM, Bret Wortman >> >> >> >> >> wrote: >> >> If I were going to attempt to restore to an old backup, what >> directories/files should I make sure to restore? I've got a >> backup script that tars up: >> >> /usr/share/ipa >> /usr/lib64/ipa >> /var/lib/pia >> /var/lib/ipa-client >> /var/lib/dirsrv >> /etc >> >> Is that enough to "roll back" to a few days ago before I started >> down this path? I'm now seeing messages about having the max >> number of CleanAllRUV tasks (4) and not being able to enqueue >> any more. So I'm really stuck now and don't know how soon I can >> get the files requested over to Rich for analysis. >> >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Tue, Aug 20, 2013 at 9:46 AM, Rich Megginson >> > wrote: >> >> On 08/20/2013 05:55 AM, Bret Wortman wrote: >> >>> Okay, now I'm thinking I need to dump all my replicas and >>> start them fresh. My /var/log/slapd-FOO-COM/errors is >>> filled with messages like this: >>> >>> NSMMReplicationPlugin - changelog program - >>> agmt="cn=meTogood1.foo.com " >>> >>> (good1:389): CSN 520a49640000001d0000 not found, we aren't >>> as up to date, or we purged >>> agmt="cn=meTogood1.foo.com " >>> >>> (good1:389) - Can't locate CSN 520a49640000001d0000 in the >>> changelog (DB rc=-30988). The consumer may need to be >>> reinitialized. >>> >>> I assume the "consumer" is the replica, right? At present, >>> I have two replicas known to my master that are simply >>> gone. Another is there but they can't talk. Three more >>> have good communication but I'm getting errors like these. >>> Is there a good, clean way to just clobber all the >>> replicas and start over without trashing the DNS and other >>> identity data that is inside my master and which /is/ >>> >>> working? Deleting them from the master hasn't been >>> working; it tends to hang the master's DNS and other >>> services until I Ctrl-C out and "ipactl restart" it. >>> >>> I'm afraid to venture out without a net here and make >>> things worse.... >>> >> >> This looks like https://fedorahosted.org/389/**ticket/47386 >> >> We've never been able to reproduce this in a "controlled" >> environment. >> >> The original reporter has been able to get this to work in >> some cases by restarting ipa (ipactl restart). >> >> Before you do that, would you be able to provide some >> information for me? >> >> On the supplier and consumer: >> ldapsearch -xLLL -D "cn=directory manager" -W -b >> "dc=FOO,dc=COM" >> '(&(objectclass=nstombstone)(**nsuniqueid=ffffffff-ffffffff-* >> *ffffffff-ffffffff))' >> > ruv.ldif >> >> ldapsearch -xLLL -D "cn=directory manager" -W -b "cn=config" >> '(objectclass=**nsds5replicationagreement)' > agmt.ldif >> >> dbscan -f /var/lib/dirsrv/slapd-FOO-COM/**cldb/*.db4 | head >> -200 > cldb.txt >> >> Be sure to obscure any sensitive data in ruv.ldif, >> agmt.ldif, and cldb.txt - you can either attach to >> https://fedorahosted.org/389/**ticket/47386or email to me >> directly. >> >> >> >>> >>> >>> _ >>> _ >>> *Bret Wortman* >>> >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Mon, Aug 19, 2013 at 2:21 PM, Bret Wortman >>> >> >> >>> wrote: >>> >>> On my master (where this error is occurring), I've >>> got, in /etc/hosts: >>> >>> 127.0.0.1 localhost localhost.localdomain >>> ::1 localhost localhost.localdomain >>> 1.2.3.4 ipamaster.foo.net >>> >>> ipamaster >>> >>> So that should be okay, right? >>> >>> # host ipamaster.foo.net >>> ipamaster.foo.net has >>> >>> address 1.2.3.4 >>> # host ipamaster >>> ipamaster.foo.net has >>> >>> address 1.2.3.4 >>> # host localhost >>> localhost has address 127.0.0.1 >>> localhost has IPv6 address ::1 >>> # >>> >>> I checked the other system (the one I can't connect >>> to) to be safe, and its /etc/hosts is similarly >>> configured. It even has the master listed with its >>> correct IP address. >>> >>> >>> >>> _ >>> _ >>> *Bret Wortman* >>> >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Mon, Aug 19, 2013 at 2:02 PM, Simo Sorce >>> > wrote: >>> >>> On Mon, 2013-08-19 at 13:51 -0400, Bret Wortman >>> wrote: >>> > So, any idea how to fix the Kerberos problem? >>> > >>> >>> If your server is trying to get a tgt for >>> ldap/localhost it probably >>> means your /etc/hosts file is broken and has a >>> line like this: >>> >>> 1.2.3.4 localhost my.real.name >>> >>> >>> When GSSAPI tries to resolve my.realm.name >>> it gets back that 'localhost' >>> >>> is the canonical name so it tries to get a TGT >>> with that name and it >>> fails. >>> >>> If /etc/host sis fine then the DNS server may be >>> returning an IP address >>> that later resolves to localhost again. >>> >>> To unbreak make sure that if you have your fully >>> qualified name >>> in /etc/hosts that it is on its own line pointing >>> at the right IP >>> address and where the FQDN name is the first in line: >>> eg: >>> >>> this is ok: >>> 1.2.3.4 server.full.name >>> >>> server >>> >>> this is not: >>> 1.2.3.4 server server.full.name >>> >>> >>> >>> Simo. >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 12:19 PM, Bret Wortman >>> > >> >> >>> wrote: >>> > ...and I got the web UI, authentication >>> and sudo back via: >>> > >>> > >>> > # ipactl stop >>> > # ipactl start >>> > >>> > >>> > Not sure why that worked, but it did. I >>> was grasping at >>> > straws, honestly. >>> > >>> > >>> > >>> > >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 12:18 PM, Bret >>> Wortman >>> > >> >> >>> wrote: >>> > Digging further, I think this >>> log entry might be the >>> > problem between the two servers >>> that aren't talking: >>> > >>> > >>> > slapd_ldap_sasl_interactive_**bind - Error: could >>> not >>> > perform interactive bind for >>> id[] mech [GSSAPI]: LDAP >>> > error -2 (Local error) >>> (SASL(-1): generic failure: >>> > GSSAPI Error: Unspecified GSS >>> failure. Minor code may >>> > provide more information (Server >>> > ldap/localhost at SPX.NET >>> not found in Kerberos >>> >>> > database)) errno 2 (No such file >>> or directory) >>> > >>> > >>> > Did I build something >>> incorrectly when that server was >>> > set up originally? >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at 12:02 >>> PM, Bret Wortman >>> > >> >> >>> wrote: >>> > I ran it on a good >>> master, against a bad one. >>> > As in, I ran this >>> command on my master IPA >>> > node: >>> > >>> > >>> > # ipa-replica-manage del >>> --force bad1.foo.net >>> >>> > --cleanup >>> > >>> > >>> > Was that wrong? I was >>> trying to delete the bad >>> > replica from the master, >>> so I figured the >>> > command needed to be run >>> on the master. But >>> > again, my master is now >>> in a state where it's >>> > not resolving DNS, user >>> logins, or sudo at the >>> > very least. >>> > >>> > >>> > Oh, and I checked the >>> node that it was >>> > complaining about >>> earlier. The network >>> > connection to it is the >>> pits, but it's there. >>> > And it resolves. >>> > >>> > >>> > >>> > >>> > >>> > Bret Wortman >>> > >>> > >>> > http://damascusgrp.com/ >>> > >>> > http://about.me/wortmanbret >>> > >>> > >>> > >>> > On Mon, Aug 19, 2013 at >>> 11:58 AM, Rob >>> > Crittenden >> > wrote: >>> > Rob Crittenden wrote: >>> > Bret Wortman wrote: >>> > Well, my master ground >>> > to a halt and wasn't >>> > responding. I rebooted >>> > the >>> > system and now I can't >>> > access the web UI or >>> > ssh to the master >>> > either. I >>> > have console access >>> > but that's it. >>> > >>> > The services all say >>> > they're running, but >>> > the web UI gives an >>> > "Unknown >>> > Error" dialog and ssh >>> > fails with >>> > "ssh_exchange_identification: >>> > Connection closed by >>> > remote host" whenever >>> > I try to ssh to >>> > ipamaster. I >>> > think something has >>> > gone really wrong >>> > inside my master. Any >>> > ideas? Even >>> > after the reboot, >>> > --cleanup isn't >>> > helping and just >>> > hangs. >>> > >>> > The logfiles end (as >>> > of the time I ^C'd the >>> > process) with: >>> > >>> > NSMMReplicationPlugin >>> > - >>> > agmt="cn=meTogood3.spx.net >>> >>> > " >>> (good3:389): Replication bind with GSSAPI >>> > auth failed: LDAP >>> > error -2 (Local error) >>> > (SASL(-1): generic >>> > failure: >>> > GSSAPI Error: >>> > Unspecified GSS >>> > failure. Minor code >>> > may provide more >>> > information (Cannot >>> > determine realm for >>> > numeric host address)) >>> > NSMMReplicationPlugin >>> > - CleanAllRUV Task: >>> > Replica not online >>> > (agmt="cn=meTogood3.foo.net >>> >>> " (good3:389)) >>> > NSMMReplicationPlugin >>> > - CleanAllRUV Task: >>> > Not all replicas >>> > online, >>> > retrying in 160 >>> > seconds..., >>> > >>> > So it looks like it's >>> > having trouble talking >>> > with one of my >>> > replicas and >>> > is doggedly trying to >>> > get the job done. Any >>> > idea how to get the >>> > master >>> > back working again >>> > while I troubleshoot >>> > this connectivity >>> > issue? >>> > >>> > That suggests a DNS problem, >>> > and it might explain ssh as >>> > well depending >>> > on your configuration. >>> > >>> > >>> > To be clear, you ran --cleanup against >>> > one of the bad masters, not a good >>> > one, right? >>> > >>> > rob >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > ______________________________**_________________ >>> > Freeipa-users mailing list >>> > Freeipa-users at redhat.com >>> >>> > >>> >>> > >>> https://www.redhat.com/** >>> mailman/listinfo/freeipa-users >>> >>> >>> -- >>> Simo Sorce * Red Hat, Inc * New York >>> >>> >>> >>> >>> >>> >>> ______________________________**_________________ >>> Freeipa-users mailing list >>> Freeipa-users at redhat.com >>> > >>> https://www.redhat.com/**mailman/listinfo/freeipa-users >>> >> >> >> >> >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.moyer at digitalreasoning.com Tue Aug 27 16:07:49 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Tue, 27 Aug 2013 12:07:49 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> Message-ID: Is there any way to see what fields are index'ed? Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 27, 2013, at 10:36 AM, John Moyer wrote: > That looks like the output I just got shown below: > > > dn: cn=mapping tree,cn=config > > dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config > > dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config > > dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ > 2Cdc\3Dcom,cn=mapping tree,cn=config > nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial > entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount > nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts > uccessfulauth krblastfailedauth krbloginfailedcount > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > > > On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: > >> John Moyer wrote: >>> Ok, so we tried to implement this again, and as soon as we put on a >>> server that authenticates heavily the IPA came to it's knees again. >>> This time I was able to watch it closely and try to troubleshoot a lot >>> more, and also know exactly what server caused it (Mercurial with help >>> of bamboo). This runs fine on a normal old openldap servers. The >>> user is logging in very quickly and each time it logs in I can see in >>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>> called) is updated over and over and over in the changelog >>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>> quickly and then disappear fairly quickly as well. >>> >>> Issue 1: This is causing severe disk latency which obviously slows >>> everything down wait times were around 25%+ >>> Issue 2: These changes need to be replicated to my slave server thus >>> adding to the mess >>> >>> >>> My question is, why does the IPA server fail to keep up with the load >>> when the openLDAP server didn't have an issue. Indexes? >>> >>> >>> I'm running the following: >>> >>> CentOS release 6.4 (Final) >>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>> python-iniparse-0.3.1-2.1.el6.noarch >>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>> >>> >>> So I've implemented this server anyway (against my better judgement with >>> these issues and just made the user that logs into mercurial a local >>> user instead of IPA). >>> >>> Also note before I did that for fun I implemented a RAM disk to put the >>> change logs on, and that dropped the wait time to 0 (except bursts where >>> it would raise to 30 to write the access log) but the CPU drove to 100% >>> trying to keep up with the load. I have also killed the replication as >>> well. >>> >>> Any help would be appreciated. >>> >> >> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >> >> You can confirm that they are excluded by searching the agreements: >> >> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >> >> They should look like: >> >> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >> >> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >> >> rob > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rcritten at redhat.com Tue Aug 27 17:13:51 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 27 Aug 2013 13:13:51 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> Message-ID: <521CDE4F.50307@redhat.com> John Moyer wrote: > Is there any way to see what fields are index'ed? $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' Your best bet is to use the logconv.pl tool to examine your logs. rob > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > Digital Reasoning Systems, Inc. > John.Moyer at digitalreasoning.com > Office: 703.678.2311 > Mobile: 240.460.0023 > Fax: 703.678.2312 > www.digitalreasoning.com > > On Aug 27, 2013, at 10:36 AM, John Moyer wrote: > >> That looks like the output I just got shown below: >> >> >> dn: cn=mapping tree,cn=config >> >> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >> >> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >> >> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >> 2Cdc\3Dcom,cn=mapping tree,cn=config >> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >> uccessfulauth krblastfailedauth krbloginfailedcount >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> >> >> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >> >>> John Moyer wrote: >>>> Ok, so we tried to implement this again, and as soon as we put on a >>>> server that authenticates heavily the IPA came to it's knees again. >>>> This time I was able to watch it closely and try to troubleshoot a lot >>>> more, and also know exactly what server caused it (Mercurial with help >>>> of bamboo). This runs fine on a normal old openldap servers. The >>>> user is logging in very quickly and each time it logs in I can see in >>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>> called) is updated over and over and over in the changelog >>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>> quickly and then disappear fairly quickly as well. >>>> >>>> Issue 1: This is causing severe disk latency which obviously slows >>>> everything down wait times were around 25%+ >>>> Issue 2: These changes need to be replicated to my slave server thus >>>> adding to the mess >>>> >>>> >>>> My question is, why does the IPA server fail to keep up with the load >>>> when the openLDAP server didn't have an issue. Indexes? >>>> >>>> >>>> I'm running the following: >>>> >>>> CentOS release 6.4 (Final) >>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>> python-iniparse-0.3.1-2.1.el6.noarch >>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>> >>>> >>>> So I've implemented this server anyway (against my better judgement with >>>> these issues and just made the user that logs into mercurial a local >>>> user instead of IPA). >>>> >>>> Also note before I did that for fun I implemented a RAM disk to put the >>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>> trying to keep up with the load. I have also killed the replication as >>>> well. >>>> >>>> Any help would be appreciated. >>>> >>> >>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>> >>> You can confirm that they are excluded by searching the agreements: >>> >>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>> >>> They should look like: >>> >>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>> >>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>> >>> rob >> > From john.moyer at digitalreasoning.com Tue Aug 27 17:23:25 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Tue, 27 Aug 2013 13:23:25 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <521CDE4F.50307@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> Message-ID: <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? Start of Log: 27/Aug/2013:02:36:08 End of Log: 27/Aug/2013:12:17:15 Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds Restarts: 2 Total Connections: 45224 SSL Connections: 44735 Peak Concurrent Connections: 76 Total Operations: 132568 Total Results: 132737 Overall Performance: 100.0% Searches: 61318 (1.76/sec) (105.52/min) Modifications: 277 (0.01/sec) (0.48/min) Adds: 10 (0.00/sec) (0.02/min) Deletes: 12 (0.00/sec) (0.02/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds: 62143 (1.78/sec) (106.94/min) Proxied Auth Operations: 0 Persistent Searches: 3 Internal Operations: 0 Entry Operations: 0 Extended Operations: 8808 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 353 Entire Search Base Queries: 106 Unindexed Searches: 319 FDs Taken: 45262 FDs Returned: 45210 Highest FD Taken: 139 Broken Pipes: 0 Connections Reset By Peer: 0 Resource Unavailable: 0 Binds: 62143 Unbinds: 44539 LDAP v2 Binds: 2 LDAP v3 Binds: 62141 SSL Client Binds: 0 Failed SSL Client Binds: 0 SASL Binds: 1466 1458 GSSAPI 8 EXTERNAL Directory Manager Binds: 10 Anonymous Binds: 1476 Other Binds: 60657 Thanks, _____________________________________________________ John Moyer Director, IT Operations On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: > John Moyer wrote: >> Is there any way to see what fields are index'ed? > > $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' > > Your best bet is to use the logconv.pl tool to examine your logs. > > rob > >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> Digital Reasoning Systems, Inc. >> John.Moyer at digitalreasoning.com >> Office: 703.678.2311 >> Mobile: 240.460.0023 >> Fax: 703.678.2312 >> www.digitalreasoning.com >> >> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >> >>> That looks like the output I just got shown below: >>> >>> >>> dn: cn=mapping tree,cn=config >>> >>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>> >>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>> >>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>> uccessfulauth krblastfailedauth krbloginfailedcount >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> >>> >>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>> >>>> John Moyer wrote: >>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>> server that authenticates heavily the IPA came to it's knees again. >>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>> more, and also know exactly what server caused it (Mercurial with help >>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>> user is logging in very quickly and each time it logs in I can see in >>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>> called) is updated over and over and over in the changelog >>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>> quickly and then disappear fairly quickly as well. >>>>> >>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>> everything down wait times were around 25%+ >>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>> adding to the mess >>>>> >>>>> >>>>> My question is, why does the IPA server fail to keep up with the load >>>>> when the openLDAP server didn't have an issue. Indexes? >>>>> >>>>> >>>>> I'm running the following: >>>>> >>>>> CentOS release 6.4 (Final) >>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>> >>>>> >>>>> So I've implemented this server anyway (against my better judgement with >>>>> these issues and just made the user that logs into mercurial a local >>>>> user instead of IPA). >>>>> >>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>> trying to keep up with the load. I have also killed the replication as >>>>> well. >>>>> >>>>> Any help would be appreciated. >>>>> >>>> >>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>> >>>> You can confirm that they are excluded by searching the agreements: >>>> >>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>> >>>> They should look like: >>>> >>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>> >>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>> >>>> rob >>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rcritten at redhat.com Tue Aug 27 20:45:02 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 27 Aug 2013 16:45:02 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> Message-ID: <521D0FCE.3080800@redhat.com> John Moyer wrote: > Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. rob > > > > Start of Log: 27/Aug/2013:02:36:08 > End of Log: 27/Aug/2013:12:17:15 > > Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds > > Restarts: 2 > Total Connections: 45224 > SSL Connections: 44735 > Peak Concurrent Connections: 76 > Total Operations: 132568 > Total Results: 132737 > Overall Performance: 100.0% > > Searches: 61318 (1.76/sec) (105.52/min) > Modifications: 277 (0.01/sec) (0.48/min) > Adds: 10 (0.00/sec) (0.02/min) > Deletes: 12 (0.00/sec) (0.02/min) > Mod RDNs: 0 (0.00/sec) (0.00/min) > Compares: 0 (0.00/sec) (0.00/min) > Binds: 62143 (1.78/sec) (106.94/min) > > Proxied Auth Operations: 0 > Persistent Searches: 3 > Internal Operations: 0 > Entry Operations: 0 > Extended Operations: 8808 > Abandoned Requests: 0 > Smart Referrals Received: 0 > > VLV Operations: 0 > VLV Unindexed Searches: 0 > SORT Operations: 353 > > Entire Search Base Queries: 106 > Unindexed Searches: 319 > > FDs Taken: 45262 > FDs Returned: 45210 > Highest FD Taken: 139 > > Broken Pipes: 0 > Connections Reset By Peer: 0 > Resource Unavailable: 0 > > Binds: 62143 > Unbinds: 44539 > > LDAP v2 Binds: 2 > LDAP v3 Binds: 62141 > SSL Client Binds: 0 > Failed SSL Client Binds: 0 > SASL Binds: 1466 > 1458 GSSAPI > 8 EXTERNAL > > Directory Manager Binds: 10 > Anonymous Binds: 1476 > Other Binds: 60657 > > > > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: > >> John Moyer wrote: >>> Is there any way to see what fields are index'ed? >> >> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >> >> Your best bet is to use the logconv.pl tool to examine your logs. >> >> rob >> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> Digital Reasoning Systems, Inc. >>> John.Moyer at digitalreasoning.com >>> Office: 703.678.2311 >>> Mobile: 240.460.0023 >>> Fax: 703.678.2312 >>> www.digitalreasoning.com >>> >>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>> >>>> That looks like the output I just got shown below: >>>> >>>> >>>> dn: cn=mapping tree,cn=config >>>> >>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>> >>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>> >>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> >>>> >>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>> >>>>> John Moyer wrote: >>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>> called) is updated over and over and over in the changelog >>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>> quickly and then disappear fairly quickly as well. >>>>>> >>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>> everything down wait times were around 25%+ >>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>> adding to the mess >>>>>> >>>>>> >>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>> >>>>>> >>>>>> I'm running the following: >>>>>> >>>>>> CentOS release 6.4 (Final) >>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>> >>>>>> >>>>>> So I've implemented this server anyway (against my better judgement with >>>>>> these issues and just made the user that logs into mercurial a local >>>>>> user instead of IPA). >>>>>> >>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>> trying to keep up with the load. I have also killed the replication as >>>>>> well. >>>>>> >>>>>> Any help would be appreciated. >>>>>> >>>>> >>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>> >>>>> You can confirm that they are excluded by searching the agreements: >>>>> >>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>> >>>>> They should look like: >>>>> >>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>> >>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>> >>>>> rob >>>> >>> >> > From natxo.asenjo at gmail.com Wed Aug 28 09:44:20 2013 From: natxo.asenjo at gmail.com (natxo asenjo) Date: Wed, 28 Aug 2013 11:44:20 +0200 Subject: [Freeipa-users] kerberized nfsv4 client Message-ID: <521DC674.6090405@gmail.com> hi, probably a stupid question but why do we need to have a host spn in the kerberos domain for the nfsv4 client to work? I do not need a host spn principal to access a cifs share on a Windows AD environment, I can just kinit user at AD.domain from my laptop that is not joined to the AD domain and once I got the ticket I can use smbclient -k or with the nautilus file manager I can browse to the shares get the cifs tickets accessing the shares. With kerberized nfsv4 the host needs to be joined to the ipa domain or it will not work, and that is a shame, but there surely is a perfectly valid reason for this that I have not found yet. Thanks for your insights on this matter. -- groet, natxo From ovalousek at vendavo.com Wed Aug 28 10:00:30 2013 From: ovalousek at vendavo.com (Ondrej Valousek) Date: Wed, 28 Aug 2013 10:00:30 +0000 Subject: [Freeipa-users] kerberized nfsv4 client In-Reply-To: <521DC674.6090405@gmail.com> References: <521DC674.6090405@gmail.com> Message-ID: <1B2E2C093FF3B7459F3C605C42E4B5040B775073@exmb1> Because with NFS (v3 or v4) it is a bit more complicated. With smbclient, you are actually not "mounting" the filesystem so that the smbclient is happy with just your TGT. With NFS, you typically need two tickets: 1. one host (or nfs) so that root can mount the filesystem using Kerberos security 2. second user TGT so that you can actually read the (already) mounted filesystem But you can run gssd with the -n argument which tells it not to look for SPNs (actually this is not SPN, we are talking about UPN in this case), but take a TGT from already pre-created kerberos database in /tmp So yes, with a bit of effort you can use kerberized NFS even from a client not joined to IPA domain. Ondrej -----Original Message----- From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of natxo asenjo Sent: Wednesday, August 28, 2013 11:44 AM To: freeipa-users at redhat.com Subject: [Freeipa-users] kerberized nfsv4 client hi, probably a stupid question but why do we need to have a host spn in the kerberos domain for the nfsv4 client to work? I do not need a host spn principal to access a cifs share on a Windows AD environment, I can just kinit user at AD.domain from my laptop that is not joined to the AD domain and once I got the ticket I can use smbclient -k or with the nautilus file manager I can browse to the shares get the cifs tickets accessing the shares. With kerberized nfsv4 the host needs to be joined to the ipa domain or it will not work, and that is a shame, but there surely is a perfectly valid reason for this that I have not found yet. Thanks for your insights on this matter. -- groet, natxo _______________________________________________ Freeipa-users mailing list Freeipa-users at redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users From natxo.asenjo at gmail.com Wed Aug 28 11:00:24 2013 From: natxo.asenjo at gmail.com (natxo asenjo) Date: Wed, 28 Aug 2013 13:00:24 +0200 Subject: [Freeipa-users] kerberized nfsv4 client In-Reply-To: <1B2E2C093FF3B7459F3C605C42E4B5040B775073@exmb1> References: <521DC674.6090405@gmail.com> <1B2E2C093FF3B7459F3C605C42E4B5040B775073@exmb1> Message-ID: <521DD848.2060706@gmail.com> On 08/28/2013 12:00 PM, Ondrej Valousek wrote: > Because with NFS (v3 or v4) it is a bit more complicated. > With smbclient, you are actually not "mounting" the filesystem so that the smbclient is happy with just your TGT. > > With NFS, you typically need two tickets: > 1. one host (or nfs) so that root can mount the filesystem using Kerberos security even though one mounts it from autofs? When using autofs from /net/host/share I can do that as non-root. > 2. second user TGT so that you can actually read the (already) mounted filesystem > > But you can run gssd with the -n argument which tells it not to look for SPNs (actually this is not SPN, we are talking about UPN in this case), but take a TGT from already pre-created kerberos database in /tmp > > So yes, with a bit of effort you can use kerberized NFS even from a client not joined to IPA domain. ok, nice to know. -- groet, natxo From bret.wortman at damascusgrp.com Wed Aug 28 12:18:28 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 28 Aug 2013 08:18:28 -0400 Subject: [Freeipa-users] Scorched earth Message-ID: Today, I'm going to wipe my master, install f18 from scratch, then install the freeipa-server RPMs again and manually load all our hosts, dns entries, and users from scratch (I'm building scripts to do this for me using the command line tools). We'll then do the same for each replica so that our system will basically be starting clean again. Are there any files that I really ought to back up and restore as part of this effort, like certificates, that might make it easier for clients to deal with us after the master comes back on line? Or am I safe to just nuke the box and start clean? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Wed Aug 28 13:56:24 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 28 Aug 2013 09:56:24 -0400 Subject: [Freeipa-users] Scorched earth In-Reply-To: References: Message-ID: <521E0188.7010508@redhat.com> Bret Wortman wrote: > Today, I'm going to wipe my master, install f18 from scratch, then > install the freeipa-server RPMs again and manually load all our hosts, > dns entries, and users from scratch (I'm building scripts to do this for > me using the command line tools). We'll then do the same for each > replica so that our system will basically be starting clean again. > > Are there any files that I really ought to back up and restore as part > of this effort, like certificates, that might make it easier for clients > to deal with us after the master comes back on line? Or am I safe to > just nuke the box and start clean? You'll end up with a new CA so you'll need to re-enroll any client machines. Browsers will see the most grief as there will be a different CA with the same subject. Depending on how you are migrating your users they will all likely need to reset their passwords, or go through the migration step. rob From bret.wortman at damascusgrp.com Wed Aug 28 14:16:20 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 28 Aug 2013 10:16:20 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E0188.7010508@redhat.com> Message-ID: Ugh. Well that certainly hurts, but I just don't see an alternative. I hope Puppet can at least make the re-enrollment a bit easier. I'm still hand-copying some of the configuration and user group details and crafting the load scripts so if anyone has a bright idea in the next few hours, I'd love to hear it! * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Wed, Aug 28, 2013 at 9:56 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> Today, I'm going to wipe my master, install f18 from scratch, then >> install the freeipa-server RPMs again and manually load all our hosts, >> dns entries, and users from scratch (I'm building scripts to do this for >> me using the command line tools). We'll then do the same for each >> replica so that our system will basically be starting clean again. >> >> Are there any files that I really ought to back up and restore as part >> of this effort, like certificates, that might make it easier for clients >> to deal with us after the master comes back on line? Or am I safe to >> just nuke the box and start clean? >> > > You'll end up with a new CA so you'll need to re-enroll any client > machines. Browsers will see the most grief as there will be a different CA > with the same subject. > > Depending on how you are migrating your users they will all likely need to > reset their passwords, or go through the migration step. > > rob > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.moyer at digitalreasoning.com Wed Aug 28 14:53:21 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Wed, 28 Aug 2013 10:53:21 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <521D0FCE.3080800@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> Message-ID: <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: > John Moyer wrote: >> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? > > There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. > > rob > > > >> >> >> Start of Log: 27/Aug/2013:02:36:08 >> End of Log: 27/Aug/2013:12:17:15 >> >> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >> >> Restarts: 2 >> Total Connections: 45224 >> SSL Connections: 44735 >> Peak Concurrent Connections: 76 >> Total Operations: 132568 >> Total Results: 132737 >> Overall Performance: 100.0% >> >> Searches: 61318 (1.76/sec) (105.52/min) >> Modifications: 277 (0.01/sec) (0.48/min) >> Adds: 10 (0.00/sec) (0.02/min) >> Deletes: 12 (0.00/sec) (0.02/min) >> Mod RDNs: 0 (0.00/sec) (0.00/min) >> Compares: 0 (0.00/sec) (0.00/min) >> Binds: 62143 (1.78/sec) (106.94/min) >> >> Proxied Auth Operations: 0 >> Persistent Searches: 3 >> Internal Operations: 0 >> Entry Operations: 0 >> Extended Operations: 8808 >> Abandoned Requests: 0 >> Smart Referrals Received: 0 >> >> VLV Operations: 0 >> VLV Unindexed Searches: 0 >> SORT Operations: 353 >> >> Entire Search Base Queries: 106 >> Unindexed Searches: 319 >> >> FDs Taken: 45262 >> FDs Returned: 45210 >> Highest FD Taken: 139 >> >> Broken Pipes: 0 >> Connections Reset By Peer: 0 >> Resource Unavailable: 0 >> >> Binds: 62143 >> Unbinds: 44539 >> >> LDAP v2 Binds: 2 >> LDAP v3 Binds: 62141 >> SSL Client Binds: 0 >> Failed SSL Client Binds: 0 >> SASL Binds: 1466 >> 1458 GSSAPI >> 8 EXTERNAL >> >> Directory Manager Binds: 10 >> Anonymous Binds: 1476 >> Other Binds: 60657 >> >> >> >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >> >>> John Moyer wrote: >>>> Is there any way to see what fields are index'ed? >>> >>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>> >>> Your best bet is to use the logconv.pl tool to examine your logs. >>> >>> rob >>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> Digital Reasoning Systems, Inc. >>>> John.Moyer at digitalreasoning.com >>>> Office: 703.678.2311 >>>> Mobile: 240.460.0023 >>>> Fax: 703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>> >>>>> That looks like the output I just got shown below: >>>>> >>>>> >>>>> dn: cn=mapping tree,cn=config >>>>> >>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>> >>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>> >>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> >>>>> >>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>> >>>>>> John Moyer wrote: >>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>> called) is updated over and over and over in the changelog >>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>> quickly and then disappear fairly quickly as well. >>>>>>> >>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>> everything down wait times were around 25%+ >>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>> adding to the mess >>>>>>> >>>>>>> >>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>> >>>>>>> >>>>>>> I'm running the following: >>>>>>> >>>>>>> CentOS release 6.4 (Final) >>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>> >>>>>>> >>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>> user instead of IPA). >>>>>>> >>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>> well. >>>>>>> >>>>>>> Any help would be appreciated. >>>>>>> >>>>>> >>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>> >>>>>> You can confirm that they are excluded by searching the agreements: >>>>>> >>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>> >>>>>> They should look like: >>>>>> >>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>> >>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>> >>>>>> rob >>>>> >>>> >>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rcritten at redhat.com Wed Aug 28 15:40:51 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 28 Aug 2013 11:40:51 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> Message-ID: <521E1A03.1060904@redhat.com> John Moyer wrote: > So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes Note that you'll want to create the same index on all hosts. This configuration is not replicated. You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update rob > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > Digital Reasoning Systems, Inc. > John.Moyer at digitalreasoning.com > Office: 703.678.2311 > Mobile: 240.460.0023 > Fax: 703.678.2312 > www.digitalreasoning.com > > On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: > >> John Moyer wrote: >>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >> >> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >> >> rob >> >>> >>> >>> >>> Start of Log: 27/Aug/2013:02:36:08 >>> End of Log: 27/Aug/2013:12:17:15 >>> >>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>> >>> Restarts: 2 >>> Total Connections: 45224 >>> SSL Connections: 44735 >>> Peak Concurrent Connections: 76 >>> Total Operations: 132568 >>> Total Results: 132737 >>> Overall Performance: 100.0% >>> >>> Searches: 61318 (1.76/sec) (105.52/min) >>> Modifications: 277 (0.01/sec) (0.48/min) >>> Adds: 10 (0.00/sec) (0.02/min) >>> Deletes: 12 (0.00/sec) (0.02/min) >>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>> Compares: 0 (0.00/sec) (0.00/min) >>> Binds: 62143 (1.78/sec) (106.94/min) >>> >>> Proxied Auth Operations: 0 >>> Persistent Searches: 3 >>> Internal Operations: 0 >>> Entry Operations: 0 >>> Extended Operations: 8808 >>> Abandoned Requests: 0 >>> Smart Referrals Received: 0 >>> >>> VLV Operations: 0 >>> VLV Unindexed Searches: 0 >>> SORT Operations: 353 >>> >>> Entire Search Base Queries: 106 >>> Unindexed Searches: 319 >>> >>> FDs Taken: 45262 >>> FDs Returned: 45210 >>> Highest FD Taken: 139 >>> >>> Broken Pipes: 0 >>> Connections Reset By Peer: 0 >>> Resource Unavailable: 0 >>> >>> Binds: 62143 >>> Unbinds: 44539 >>> >>> LDAP v2 Binds: 2 >>> LDAP v3 Binds: 62141 >>> SSL Client Binds: 0 >>> Failed SSL Client Binds: 0 >>> SASL Binds: 1466 >>> 1458 GSSAPI >>> 8 EXTERNAL >>> >>> Directory Manager Binds: 10 >>> Anonymous Binds: 1476 >>> Other Binds: 60657 >>> >>> >>> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>> >>>> John Moyer wrote: >>>>> Is there any way to see what fields are index'ed? >>>> >>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>> >>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>> >>>> rob >>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> Digital Reasoning Systems, Inc. >>>>> John.Moyer at digitalreasoning.com >>>>> Office: 703.678.2311 >>>>> Mobile: 240.460.0023 >>>>> Fax: 703.678.2312 >>>>> www.digitalreasoning.com >>>>> >>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>> >>>>>> That looks like the output I just got shown below: >>>>>> >>>>>> >>>>>> dn: cn=mapping tree,cn=config >>>>>> >>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>> >>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>> >>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> >>>>>> >>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>> >>>>>>> John Moyer wrote: >>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>> called) is updated over and over and over in the changelog >>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>> >>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>> everything down wait times were around 25%+ >>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>> adding to the mess >>>>>>>> >>>>>>>> >>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>> >>>>>>>> >>>>>>>> I'm running the following: >>>>>>>> >>>>>>>> CentOS release 6.4 (Final) >>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>> >>>>>>>> >>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>> user instead of IPA). >>>>>>>> >>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>> well. >>>>>>>> >>>>>>>> Any help would be appreciated. >>>>>>>> >>>>>>> >>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>> >>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>> >>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>> >>>>>>> They should look like: >>>>>>> >>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>> >>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>> >>>>>>> rob >>>>>> >>>>> >>>> >>> >> > From dpal at redhat.com Thu Aug 29 00:30:34 2013 From: dpal at redhat.com (Dmitri Pal) Date: Wed, 28 Aug 2013 20:30:34 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E0188.7010508@redhat.com> Message-ID: <521E962A.7090700@redhat.com> On 08/28/2013 10:16 AM, Bret Wortman wrote: > Ugh. Well that certainly hurts, but I just don't see an alternative. I > hope Puppet can at least make the re-enrollment a bit easier. > > I'm still hand-copying some of the configuration and user group > details and crafting the load scripts so if anyone has a bright idea > in the next few hours, I'd love to hear it! > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Wed, Aug 28, 2013 at 9:56 AM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > Today, I'm going to wipe my master, install f18 from scratch, then > install the freeipa-server RPMs again and manually load all > our hosts, > dns entries, and users from scratch (I'm building scripts to > do this for > me using the command line tools). We'll then do the same for each > replica so that our system will basically be starting clean again. > > Are there any files that I really ought to back up and restore > as part > of this effort, like certificates, that might make it easier > for clients > to deal with us after the master comes back on line? Or am I > safe to > just nuke the box and start clean? > > > You'll end up with a new CA so you'll need to re-enroll any client > machines. Browsers will see the most grief as there will be a > different CA with the same subject. > > Depending on how you are migrating your users they will all likely > need to reset their passwords, or go through the migration step. > And migration step means you carry forward user data as if you migrated from an LDAP server. In this case you can complete migration using either a migration web page or just using SSSD. If the migration is enabled and you migrated LDAP password attributes from the older IPA then SSSD and/or migration page would be able to capture user password and create kerberos hashes completing the migration. This at least would not require people to change the passwords. > > rob > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From lroot at redhat.com Thu Aug 29 00:33:35 2013 From: lroot at redhat.com (Lynn Root) Date: Wed, 28 Aug 2013 17:33:35 -0700 Subject: [Freeipa-users] Using IPA's CA for Puppet In-Reply-To: <1079065438.3204101.1377729532432.JavaMail.root@redhat.com> Message-ID: Hey folks - Inspired by this blog post [1], I decided to recreate & update the process of setting up Puppet to use IPA's CA/PKI system [2]. It's just a PoC, so use at your own risk :) Any issues/typos, let me know. Cheers, LR [1]: http://jcape.name/2012/01/16/using-the-freeipa-pki-with-puppet/ [2]: http://www.freeipa.org/page/Using_IPA%27s_CA_for_Puppet -- Lynn Root Associate Software Engineer Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnansi at redhat.com Thu Aug 29 00:55:20 2013 From: jnansi at redhat.com (Jatin Nansi) Date: Thu, 29 Aug 2013 10:55:20 +1000 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E0188.7010508@redhat.com> Message-ID: <521E9BF8.6050302@redhat.com> On 08/29/2013 12:16 AM, Bret Wortman wrote: > Ugh. Well that certainly hurts, but I just don't see an alternative. I > hope Puppet can at least make the re-enrollment a bit easier. > > I'm still hand-copying some of the configuration and user group > details and crafting the load scripts so if anyone has a bright idea > in the next few hours, I'd love to hear it! Is there a reason why you must scorch earth? You could try a rolling update approach first - install a fresh IPA system, make it a replica of the existing IPA setup. Then reinstall existing IPA systems and use the updated system to set them up. Jatin From bret.wortman at damascusgrp.com Thu Aug 29 02:01:26 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Wed, 28 Aug 2013 19:01:26 -0700 (PDT) Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: <521E9BF8.6050302@redhat.com> References: <521E9BF8.6050302@redhat.com> Message-ID: <1377741685734.9f79c198@Nodemailer> I was actually considering something like a few hours ago. It's a VM, so making another isn't that hard. Replication is the source of all my problems, though, so I'm concerned about whether it will work. Certainly worth the attempt! I'll report back later tomorrow. On Wed, Aug 28, 2013 at 8:56 PM, Jatin Nansi wrote: > On 08/29/2013 12:16 AM, Bret Wortman wrote: >> Ugh. Well that certainly hurts, but I just don't see an alternative. I >> hope Puppet can at least make the re-enrollment a bit easier. >> >> I'm still hand-copying some of the configuration and user group >> details and crafting the load scripts so if anyone has a bright idea >> in the next few hours, I'd love to hear it! > Is there a reason why you must scorch earth? You could try a rolling > update approach first - install a fresh IPA system, make it a replica of > the existing IPA setup. Then reinstall existing IPA systems and use the > updated system to set them up. > Jatin > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpal at redhat.com Thu Aug 29 02:42:23 2013 From: dpal at redhat.com (Dmitri Pal) Date: Wed, 28 Aug 2013 22:42:23 -0400 Subject: [Freeipa-users] Intranet password replication to DMZ In-Reply-To: <521CB247.1020404@redhat.com> References: <521CB247.1020404@redhat.com> Message-ID: <521EB50F.3070501@redhat.com> On 08/27/2013 10:05 AM, Rob Crittenden wrote: > Jessie Floyd wrote: >> I've been working on a project where I have multiple IPA domains which >> can't be connected due to scope and purpose of each domain. Ideally I >> would like to replicte a single user's password from a core domain >> server to a satellite ipa domain. I've learned that the password hash >> is not a traditional hash and cant be replicated without some additional >> work. My primary site is a multi-master and the satellite site has its >> own multi-master configuration. As an example I have an intranet server >> which hosts multiple users and a DMZ domain where a limited set of >> admins work. How can I replicate an intranet user from the inside to >> the DMZ? Any pointers or ideas would be helpful. > > I'm not entirely clear what it is you want/need to do. > > Do you want to set up some sort of fractional replication that > replicates only passwords, and the raw hashes at that? That would do > you no good when it comes to Kerberos. > > rob > You would need to intercept password change operation in KDC and DS of one domain then connect to other domain and do password update operation there. Sort of passync by not from AD to IPA but rather from IPA to IPA. But may be it would be easier to not replicate password hashes from the central domain to the DMZ domains but rather use Kerberos to Kerberos trusts and set them manually? If the initial authentication to acquire TGT always happens in the internal domain that might fly. > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From bret.wortman at damascusgrp.com Thu Aug 29 12:07:00 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 08:07:00 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: <1377741685734.9f79c198@Nodemailer> References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> Message-ID: Okay, I have a replica built and running. My original, "sick" server is ipamaster and the new one is ipamaster2. All I've done thus far on ipamaster2 is run ipa-replica-install --setup-dns --no-forwarders replica-info-ipamaster2.foo.net.gpg. What additional steps do I need to take to ensure that the process of shutting down ipamaster, wiping it out, building it up fresh and then replicating ipamaster2 back to ipamaster and making ipamaster again the center of the universe and my certificate authority work correctly, cleanly, and with minimal fuss? Given the mess I got our servers already, I figured I should ask *before* I start messing about today. I *think* the process should look something like this (I don't want you all thinking I'm looking for someone to do *all* my thinking for me): 1. Take snapshot of ipamaster (just in case) 2. [ipamaster2]# ipa-ca-install /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg(I should've done this during the ipa-ca-install, but since the ca step is so rare, I didn't have it in my wiki notes). 3. [ipamaster]# reboot This reboot will trigger a Cobbler & Puppet-based wipe of the system and reinstallation of F18 and freeipa-server. While that's going on: 4. [ipamaster2]# ipa-replica-prepare ipamaster.foo.net1.2.3.4 When ipamaster is back up: 5. [ipamaster]# cd /var/lib/ipa && scp ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg. 6. [ipamaster]# ipa-replica-install --setup-dns --no-forwarders --setup-ca replica-info-ipamaster.foo.net.gpg Usually, there's some reason I need to go back to ipamaster2 and either delete a dns entry or ipa host-del the system. After the replica install is done: 7. Shut down and delete the ipamaster2 VM. 8. Upgrade existing "replicas" to F18 and latest IPA version. 9. Establish replication agreements with now-functioning ipamaster. Does that sound right? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Wed, Aug 28, 2013 at 10:01 PM, Bret Wortman wrote: > I was actually considering something like a few hours ago. It's a VM, so > making another isn't that hard. Replication is the source of all my > problems, though, so I'm concerned about whether it will work. Certainly > worth the attempt! > > I'll report back later tomorrow. > > > On Wed, Aug 28, 2013 at 8:56 PM, Jatin Nansi wrote: > >> On 08/29/2013 12:16 AM, Bret Wortman wrote: >> > Ugh. Well that certainly hurts, but I just don't see an alternative. I >> > hope Puppet can at least make the re-enrollment a bit easier. >> > >> > I'm still hand-copying some of the configuration and user group >> > details and crafting the load scripts so if anyone has a bright idea >> > in the next few hours, I'd love to hear it! >> Is there a reason why you must scorch earth? You could try a rolling >> update approach first - install a fresh IPA system, make it a replica of >> the existing IPA setup. Then reinstall existing IPA systems and use the >> updated system to set them up. >> >> Jatin >> >> _______________________________________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-users >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simo at redhat.com Thu Aug 29 13:09:18 2013 From: simo at redhat.com (Simo Sorce) Date: Thu, 29 Aug 2013 09:09:18 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> Message-ID: <1377781758.2917.43.camel@willson.li.ssimo.org> On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: > Okay, I have a replica built and running. My original, "sick" server > is ipamaster and the new one is ipamaster2. All I've done thus far on > ipamaster2 is run ipa-replica-install --setup-dns --no-forwarders > replica-info-ipamaster2.foo.net.gpg. > > > What additional steps do I need to take to ensure that the process of > shutting down ipamaster, wiping it out, building it up fresh and then > replicating ipamaster2 back to ipamaster and making ipamaster again > the center of the universe and my certificate authority work > correctly, cleanly, and with minimal fuss? Given the mess I got our > servers already, I figured I should ask before I start messing about > today. > > > I think the process should look something like this (I don't want you > all thinking I'm looking for someone to do all my thinking for me): > > > 1. Take snapshot of ipamaster (just in case) > 2. [ipamaster2]# > ipa-ca-install /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg (I > should've done this during the ipa-ca-install, but since the ca step > is so rare, I didn't have it in my wiki notes). > 3. [ipamaster]# reboot > > > This reboot will trigger a Cobbler & Puppet-based wipe of the system > and reinstallation of F18 and freeipa-server. While that's going on: > > > 4. [ipamaster2]# ipa-replica-prepare ipamaster.foo.net 1.2.3.4 You need to use ipa-replica-manage to remove the original ipamaster before you can prepare to add a new one. After it is fully removed and replica file generated you need to restart at yleast 389ds on ipamaster2 this is due to the fact that DS does nto purge valid tickets, and it holds a ticket valid for the old ipamaster, however when you reinstall the new the name will match so replication between ipamaster2 -> ipamaster may fail because ipamsater2 has a wrong ticket (using old key you just nuked before the reinstall). > > When ipamaster is back up: > > > 5. [ipamaster]# cd /var/lib/ipa && scp You can copy in /root > ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg . > 6. [ipamaster]# ipa-replica-install --setup-dns --no-forwarders > --setup-ca replica-info-ipamaster.foo.net.gpg > > > Usually, there's some reason I need to go back to ipamaster2 and > either delete a dns entry or ipa host-del the system. Uh ? Sound like this is going to screw up things, why should you delete DNS entries ? ipa host-del of a master is *certainly* going to break replication and basically everything. Is this what you did in your old setup ? > After the replica install is done: > > > 7. Shut down and delete the ipamaster2 VM. Do not forget to ipa-replica-manage remove it first. > 8. Upgrade existing "replicas" to F18 and latest IPA version. > 9. Establish replication agreements with now-functioning ipamaster. > > > Does that sound right? > > See above. Simo. -- Simo Sorce * Red Hat, Inc * New York From bret.wortman at damascusgrp.com Thu Aug 29 13:14:47 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 09:14:47 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: <1377781758.2917.43.camel@willson.li.ssimo.org> References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> Message-ID: On Thu, Aug 29, 2013 at 9:09 AM, Simo Sorce wrote: > On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: > > Okay, I have a replica built and running. My original, "sick" server > > is ipamaster and the new one is ipamaster2. All I've done thus far on > > ipamaster2 is run ipa-replica-install --setup-dns --no-forwarders > > replica-info-ipamaster2.foo.net.gpg. > > > > > > What additional steps do I need to take to ensure that the process of > > shutting down ipamaster, wiping it out, building it up fresh and then > > replicating ipamaster2 back to ipamaster and making ipamaster again > > the center of the universe and my certificate authority work > > correctly, cleanly, and with minimal fuss? Given the mess I got our > > servers already, I figured I should ask before I start messing about > > today. > > > > > > I think the process should look something like this (I don't want you > > all thinking I'm looking for someone to do all my thinking for me): > > > > > > 1. Take snapshot of ipamaster (just in case) > > 2. [ipamaster2]# > > ipa-ca-install /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg (I > > should've done this during the ipa-ca-install, but since the ca step > > is so rare, I didn't have it in my wiki notes). > > 3. [ipamaster]# reboot > > > > > > This reboot will trigger a Cobbler & Puppet-based wipe of the system > > and reinstallation of F18 and freeipa-server. While that's going on: > > > > > > 4. [ipamaster2]# ipa-replica-prepare ipamaster.foo.net 1.2.3.4 > > You need to use ipa-replica-manage to remove the original ipamaster > before you can prepare to add a new one. > > After it is fully removed and replica file generated you need to restart > at yleast 389ds on ipamaster2 this is due to the fact that DS does nto > purge valid tickets, and it holds a ticket valid for the old ipamaster, > however when you reinstall the new the name will match so replication > between ipamaster2 -> ipamaster may fail because ipamsater2 has a wrong > ticket (using old key you just nuked before the reinstall). > > > Got it. Glad I asked! I'll add these steps to my procedure. > > When ipamaster is back up: > > > > > > 5. [ipamaster]# cd /var/lib/ipa && scp > > You can copy in /root > > I usually do it in /var/lib/ipa I guess because that's where the server puts the file, so it makes it easy for me to remember that's where it is. But point taken. > > ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg . > > 6. [ipamaster]# ipa-replica-install --setup-dns --no-forwarders > > --setup-ca replica-info-ipamaster.foo.net.gpg > > > > > > Usually, there's some reason I need to go back to ipamaster2 and > > either delete a dns entry or ipa host-del the system. > > Uh ? Sound like this is going to screw up things, why should you delete > DNS entries ? > ipa host-del of a master is *certainly* going to break replication and > basically everything. Is this what you did in your old setup ? > Only if ipa-replica-install said I needed to. > > > After the replica install is done: > > > > > > 7. Shut down and delete the ipamaster2 VM. > > Do not forget to ipa-replica-manage remove it first. > Awesome. This is why I asked. > > > 8. Upgrade existing "replicas" to F18 and latest IPA version. > > 9. Establish replication agreements with now-functioning ipamaster. > > > > > > Does that sound right? > > > > > See above. > > Simo. > > > -- > Simo Sorce * Red Hat, Inc * New York > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simo at redhat.com Thu Aug 29 13:21:50 2013 From: simo at redhat.com (Simo Sorce) Date: Thu, 29 Aug 2013 09:21:50 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> Message-ID: <1377782510.2917.48.camel@willson.li.ssimo.org> On Thu, 2013-08-29 at 09:14 -0400, Bret Wortman wrote: > On Thu, Aug 29, 2013 at 9:09 AM, Simo Sorce wrote: > On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: > > Okay, I have a replica built and running. My original, > "sick" server > > is ipamaster and the new one is ipamaster2. All I've done > thus far on > > ipamaster2 is run ipa-replica-install --setup-dns > --no-forwarders > > replica-info-ipamaster2.foo.net.gpg. > > > > > > What additional steps do I need to take to ensure that the > process of > > shutting down ipamaster, wiping it out, building it up fresh > and then > > replicating ipamaster2 back to ipamaster and making > ipamaster again > > the center of the universe and my certificate authority work > > correctly, cleanly, and with minimal fuss? Given the mess I > got our > > servers already, I figured I should ask before I start > messing about > > today. > > > > > > I think the process should look something like this (I don't > want you > > all thinking I'm looking for someone to do all my thinking > for me): > > > > > > 1. Take snapshot of ipamaster (just in case) > > 2. [ipamaster2]# > > > ipa-ca-install /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg (I > > should've done this during the ipa-ca-install, but since the > ca step > > is so rare, I didn't have it in my wiki notes). > > 3. [ipamaster]# reboot > > > > > > This reboot will trigger a Cobbler & Puppet-based wipe of > the system > > and reinstallation of F18 and freeipa-server. While that's > going on: > > > > > > 4. [ipamaster2]# ipa-replica-prepare ipamaster.foo.net > 1.2.3.4 > > > You need to use ipa-replica-manage to remove the original > ipamaster > before you can prepare to add a new one. > > After it is fully removed and replica file generated you need > to restart > at yleast 389ds on ipamaster2 this is due to the fact that DS > does nto > purge valid tickets, and it holds a ticket valid for the old > ipamaster, > however when you reinstall the new the name will match so > replication > between ipamaster2 -> ipamaster may fail because ipamsater2 > has a wrong > ticket (using old key you just nuked before the reinstall). > > > > > > Got it. Glad I asked! I'll add these steps to my procedure. > > > When ipamaster is back up: > > > > > > 5. [ipamaster]# cd /var/lib/ipa && scp > > > You can copy in /root > > > I usually do it in /var/lib/ipa I guess because that's where the > server puts the file, so it makes it easy for me to remember that's > where it is. But point taken. > > > > ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg . > > 6. [ipamaster]# ipa-replica-install --setup-dns > --no-forwarders > > --setup-ca replica-info-ipamaster.foo.net.gpg > > > > > > Usually, there's some reason I need to go back to ipamaster2 > and > > either delete a dns entry or ipa host-del the system. > > > Uh ? Sound like this is going to screw up things, why should > you delete > DNS entries ? > ipa host-del of a master is *certainly* going to break > replication and > basically everything. Is this what you did in your old setup ? > > > Only if ipa-replica-install said I needed to. ok this means you previously uninstalled a replica directly on the machine but tdid not remove it from the domain, this is bad practice. you should use ipa-replica-manage before you retire a machine if possible, otherwise you leave dangling replication agreements, DNS names, ID ranges (this means you loose ID space), and keys. > > After the replica install is done: > > > > > > 7. Shut down and delete the ipamaster2 VM. > > > Do not forget to ipa-replica-manage remove it first. > > > Awesome. This is why I asked. > > > 8. Upgrade existing "replicas" to F18 and latest IPA > version. > > 9. Establish replication agreements with now-functioning > ipamaster. > > > > > > Does that sound right? > > > > > > See above. > > Simo. > > > -- > Simo Sorce * Red Hat, Inc * New York > > > -- Simo Sorce * Red Hat, Inc * New York From bret.wortman at damascusgrp.com Thu Aug 29 13:24:48 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 09:24:48 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: <1377782510.2917.48.camel@willson.li.ssimo.org> References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> Message-ID: Agreed, but not always possible. I had a replica crash hard and it wasn't possible to remove it. In other news: [ipamaster2]# ipa-ca-install replica-info-ipamaster2.spx.net.gpg A selfsign CA can not be added Is there a way around this? How can I ensure that I can transfer the CA back to ipamaster after it's been erased & reinstalled? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Thu, Aug 29, 2013 at 9:21 AM, Simo Sorce wrote: > On Thu, 2013-08-29 at 09:14 -0400, Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 9:09 AM, Simo Sorce wrote: > > On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: > > > Okay, I have a replica built and running. My original, > > "sick" server > > > is ipamaster and the new one is ipamaster2. All I've done > > thus far on > > > ipamaster2 is run ipa-replica-install --setup-dns > > --no-forwarders > > > replica-info-ipamaster2.foo.net.gpg. > > > > > > > > > What additional steps do I need to take to ensure that the > > process of > > > shutting down ipamaster, wiping it out, building it up fresh > > and then > > > replicating ipamaster2 back to ipamaster and making > > ipamaster again > > > the center of the universe and my certificate authority work > > > correctly, cleanly, and with minimal fuss? Given the mess I > > got our > > > servers already, I figured I should ask before I start > > messing about > > > today. > > > > > > > > > I think the process should look something like this (I don't > > want you > > > all thinking I'm looking for someone to do all my thinking > > for me): > > > > > > > > > 1. Take snapshot of ipamaster (just in case) > > > 2. [ipamaster2]# > > > > > ipa-ca-install /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg > (I > > > should've done this during the ipa-ca-install, but since the > > ca step > > > is so rare, I didn't have it in my wiki notes). > > > 3. [ipamaster]# reboot > > > > > > > > > This reboot will trigger a Cobbler & Puppet-based wipe of > > the system > > > and reinstallation of F18 and freeipa-server. While that's > > going on: > > > > > > > > > 4. [ipamaster2]# ipa-replica-prepare ipamaster.foo.net > > 1.2.3.4 > > > > > > You need to use ipa-replica-manage to remove the original > > ipamaster > > before you can prepare to add a new one. > > > > After it is fully removed and replica file generated you need > > to restart > > at yleast 389ds on ipamaster2 this is due to the fact that DS > > does nto > > purge valid tickets, and it holds a ticket valid for the old > > ipamaster, > > however when you reinstall the new the name will match so > > replication > > between ipamaster2 -> ipamaster may fail because ipamsater2 > > has a wrong > > ticket (using old key you just nuked before the reinstall). > > > > > > > > > > > Got it. Glad I asked! I'll add these steps to my procedure. > > > > > When ipamaster is back up: > > > > > > > > > 5. [ipamaster]# cd /var/lib/ipa && scp > > > > > > You can copy in /root > > > > > > I usually do it in /var/lib/ipa I guess because that's where the > > server puts the file, so it makes it easy for me to remember that's > > where it is. But point taken. > > > > > > > ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg . > > > 6. [ipamaster]# ipa-replica-install --setup-dns > > --no-forwarders > > > --setup-ca replica-info-ipamaster.foo.net.gpg > > > > > > > > > Usually, there's some reason I need to go back to ipamaster2 > > and > > > either delete a dns entry or ipa host-del the system. > > > > > > Uh ? Sound like this is going to screw up things, why should > > you delete > > DNS entries ? > > ipa host-del of a master is *certainly* going to break > > replication and > > basically everything. Is this what you did in your old setup ? > > > > > > Only if ipa-replica-install said I needed to. > > ok this means you previously uninstalled a replica directly on the > machine but tdid not remove it from the domain, this is bad practice. > you should use ipa-replica-manage before you retire a machine if > possible, otherwise you leave dangling replication agreements, DNS > names, ID ranges (this means you loose ID space), and keys. > > > > After the replica install is done: > > > > > > > > > 7. Shut down and delete the ipamaster2 VM. > > > > > > Do not forget to ipa-replica-manage remove it first. > > > > > > Awesome. This is why I asked. > > > > > 8. Upgrade existing "replicas" to F18 and latest IPA > > version. > > > 9. Establish replication agreements with now-functioning > > ipamaster. > > > > > > > > > Does that sound right? > > > > > > > > > > See above. > > > > Simo. > > > > > > -- > > Simo Sorce * Red Hat, Inc * New York > > > > > > > > > -- > Simo Sorce * Red Hat, Inc * New York > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Thu Aug 29 14:51:58 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 10:51:58 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> Message-ID: A bit of googling has led me to understand that we must have created the original server with --selfsign, and that locked us into something bad which is now causing us problems. I'm not sure how this happened, since we actually created our original instance on a different server, created ipamaster as a replica of that one, then ran ipa-ca-install on ipamaster to make it the new CA. How did it end up in this state? Anyway, is there ANY way around this? Can I simply ignore this, break the replication agreement as Simo suggested, rebuild ipamaster, replicate ipamaster2 to the new ipamaster, and then somehow make ipamaster be a CA using Dogtag? Will that screw up all the clients? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Thu, Aug 29, 2013 at 9:24 AM, Bret Wortman wrote: > Agreed, but not always possible. I had a replica crash hard and it wasn't > possible to remove it. > > In other news: > > [ipamaster2]# ipa-ca-install replica-info-ipamaster2.spx.net.gpg > A selfsign CA can not be added > > Is there a way around this? How can I ensure that I can transfer the CA > back to ipamaster after it's been erased & reinstalled? > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Thu, Aug 29, 2013 at 9:21 AM, Simo Sorce wrote: > >> On Thu, 2013-08-29 at 09:14 -0400, Bret Wortman wrote: >> > On Thu, Aug 29, 2013 at 9:09 AM, Simo Sorce wrote: >> > On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: >> > > Okay, I have a replica built and running. My original, >> > "sick" server >> > > is ipamaster and the new one is ipamaster2. All I've done >> > thus far on >> > > ipamaster2 is run ipa-replica-install --setup-dns >> > --no-forwarders >> > > replica-info-ipamaster2.foo.net.gpg. >> > > >> > > >> > > What additional steps do I need to take to ensure that the >> > process of >> > > shutting down ipamaster, wiping it out, building it up fresh >> > and then >> > > replicating ipamaster2 back to ipamaster and making >> > ipamaster again >> > > the center of the universe and my certificate authority work >> > > correctly, cleanly, and with minimal fuss? Given the mess I >> > got our >> > > servers already, I figured I should ask before I start >> > messing about >> > > today. >> > > >> > > >> > > I think the process should look something like this (I don't >> > want you >> > > all thinking I'm looking for someone to do all my thinking >> > for me): >> > > >> > > >> > > 1. Take snapshot of ipamaster (just in case) >> > > 2. [ipamaster2]# >> > > >> > ipa-ca-install /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg >> (I >> > > should've done this during the ipa-ca-install, but since the >> > ca step >> > > is so rare, I didn't have it in my wiki notes). >> > > 3. [ipamaster]# reboot >> > > >> > > >> > > This reboot will trigger a Cobbler & Puppet-based wipe of >> > the system >> > > and reinstallation of F18 and freeipa-server. While that's >> > going on: >> > > >> > > >> > > 4. [ipamaster2]# ipa-replica-prepare ipamaster.foo.net >> > 1.2.3.4 >> > >> > >> > You need to use ipa-replica-manage to remove the original >> > ipamaster >> > before you can prepare to add a new one. >> > >> > After it is fully removed and replica file generated you need >> > to restart >> > at yleast 389ds on ipamaster2 this is due to the fact that DS >> > does nto >> > purge valid tickets, and it holds a ticket valid for the old >> > ipamaster, >> > however when you reinstall the new the name will match so >> > replication >> > between ipamaster2 -> ipamaster may fail because ipamsater2 >> > has a wrong >> > ticket (using old key you just nuked before the reinstall). >> > > >> > >> > >> > >> > Got it. Glad I asked! I'll add these steps to my procedure. >> > >> > > When ipamaster is back up: >> > > >> > > >> > > 5. [ipamaster]# cd /var/lib/ipa && scp >> > >> > >> > You can copy in /root >> > >> > >> > I usually do it in /var/lib/ipa I guess because that's where the >> > server puts the file, so it makes it easy for me to remember that's >> > where it is. But point taken. >> > >> > > >> > ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg . >> > > 6. [ipamaster]# ipa-replica-install --setup-dns >> > --no-forwarders >> > > --setup-ca replica-info-ipamaster.foo.net.gpg >> > > >> > > >> > > Usually, there's some reason I need to go back to ipamaster2 >> > and >> > > either delete a dns entry or ipa host-del the system. >> > >> > >> > Uh ? Sound like this is going to screw up things, why should >> > you delete >> > DNS entries ? >> > ipa host-del of a master is *certainly* going to break >> > replication and >> > basically everything. Is this what you did in your old setup ? >> > >> > >> > Only if ipa-replica-install said I needed to. >> >> ok this means you previously uninstalled a replica directly on the >> machine but tdid not remove it from the domain, this is bad practice. >> you should use ipa-replica-manage before you retire a machine if >> possible, otherwise you leave dangling replication agreements, DNS >> names, ID ranges (this means you loose ID space), and keys. >> >> > > After the replica install is done: >> > > >> > > >> > > 7. Shut down and delete the ipamaster2 VM. >> > >> > >> > Do not forget to ipa-replica-manage remove it first. >> > >> > >> > Awesome. This is why I asked. >> > >> > > 8. Upgrade existing "replicas" to F18 and latest IPA >> > version. >> > > 9. Establish replication agreements with now-functioning >> > ipamaster. >> > > >> > > >> > > Does that sound right? >> > > >> > > >> > >> > See above. >> > >> > Simo. >> > >> > >> > -- >> > Simo Sorce * Red Hat, Inc * New York >> > >> > >> > >> >> >> -- >> Simo Sorce * Red Hat, Inc * New York >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 29 15:10:43 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Aug 2013 11:10:43 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> Message-ID: <521F6473.9040605@redhat.com> Bret Wortman wrote: > A bit of googling has led me to understand that we must have created the > original server with --selfsign, and that locked us into something bad > which is now causing us problems. I'm not sure how this happened, since > we actually created our original instance on a different server, created > ipamaster as a replica of that one, then ran ipa-ca-install on ipamaster > to make it the new CA. How did it end up in this state? > > Anyway, is there ANY way around this? Can I simply ignore this, break > the replication agreement as Simo suggested, rebuild ipamaster, > replicate ipamaster2 to the new ipamaster, and then somehow make > ipamaster be a CA using Dogtag? Will that screw up all the clients? I think we should pause and take a look at your installation. I'd check all your current masters, whether they are currently working or not. Look at the value of ra_plugin in /etc/ipa/default.conf. That controls what IPA thinks the CA is. Then check to see if you have dogtag running on any of these systems. This will include a 2nd 389-ds instance, /etc/dirsrv/slapd-PKI-IPA and, depending on your distro, a PKI service like pki-tomcatd at pki-tomcat.service. You can optionally see if /etc/pki/pki-tomcat exists. There is currently no way post-install to add a dogtag instance. rob > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Thu, Aug 29, 2013 at 9:24 AM, Bret Wortman > > wrote: > > Agreed, but not always possible. I had a replica crash hard and it > wasn't possible to remove it. > > In other news: > > [ipamaster2]# ipa-ca-install replica-info-ipamaster2.spx.net.gpg > A selfsign CA can not be added > > Is there a way around this? How can I ensure that I can transfer the > CA back to ipamaster after it's been erased & reinstalled? > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Thu, Aug 29, 2013 at 9:21 AM, Simo Sorce > wrote: > > On Thu, 2013-08-29 at 09:14 -0400, Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 9:09 AM, Simo Sorce > wrote: > > On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: > > > Okay, I have a replica built and running. My original, > > "sick" server > > > is ipamaster and the new one is ipamaster2. All > I've done > > thus far on > > > ipamaster2 is run ipa-replica-install --setup-dns > > --no-forwarders > > > replica-info-ipamaster2.foo.net.gpg. > > > > > > > > > What additional steps do I need to take to ensure > that the > > process of > > > shutting down ipamaster, wiping it out, building it > up fresh > > and then > > > replicating ipamaster2 back to ipamaster and making > > ipamaster again > > > the center of the universe and my certificate > authority work > > > correctly, cleanly, and with minimal fuss? Given > the mess I > > got our > > > servers already, I figured I should ask before I start > > messing about > > > today. > > > > > > > > > I think the process should look something like this > (I don't > > want you > > > all thinking I'm looking for someone to do all my > thinking > > for me): > > > > > > > > > 1. Take snapshot of ipamaster (just in case) > > > 2. [ipamaster2]# > > > > > ipa-ca-install > /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg (I > > > should've done this during the ipa-ca-install, but > since the > > ca step > > > is so rare, I didn't have it in my wiki notes). > > > 3. [ipamaster]# reboot > > > > > > > > > This reboot will trigger a Cobbler & Puppet-based > wipe of > > the system > > > and reinstallation of F18 and freeipa-server. While > that's > > going on: > > > > > > > > > 4. [ipamaster2]# ipa-replica-prepare > ipamaster.foo.net > > 1.2.3.4 > > > > > > You need to use ipa-replica-manage to remove the original > > ipamaster > > before you can prepare to add a new one. > > > > After it is fully removed and replica file generated > you need > > to restart > > at yleast 389ds on ipamaster2 this is due to the fact > that DS > > does nto > > purge valid tickets, and it holds a ticket valid for > the old > > ipamaster, > > however when you reinstall the new the name will match so > > replication > > between ipamaster2 -> ipamaster may fail because > ipamsater2 > > has a wrong > > ticket (using old key you just nuked before the > reinstall). > > > > > > > > > > > Got it. Glad I asked! I'll add these steps to my procedure. > > > > > When ipamaster is back up: > > > > > > > > > 5. [ipamaster]# cd /var/lib/ipa && scp > > > > > > You can copy in /root > > > > > > I usually do it in /var/lib/ipa I guess because that's where the > > server puts the file, so it makes it easy for me to remember > that's > > where it is. But point taken. > > > > > > > > ipamaster2:/var/lib/ipa/replica-info-ipamaster.foo.net.gpg . > > > 6. [ipamaster]# ipa-replica-install --setup-dns > > --no-forwarders > > > --setup-ca replica-info-ipamaster.foo.net.gpg > > > > > > > > > Usually, there's some reason I need to go back to > ipamaster2 > > and > > > either delete a dns entry or ipa host-del the system. > > > > > > Uh ? Sound like this is going to screw up things, why > should > > you delete > > DNS entries ? > > ipa host-del of a master is *certainly* going to break > > replication and > > basically everything. Is this what you did in your > old setup ? > > > > > > Only if ipa-replica-install said I needed to. > > ok this means you previously uninstalled a replica directly on the > machine but tdid not remove it from the domain, this is bad > practice. > you should use ipa-replica-manage before you retire a machine if > possible, otherwise you leave dangling replication agreements, DNS > names, ID ranges (this means you loose ID space), and keys. > > > > After the replica install is done: > > > > > > > > > 7. Shut down and delete the ipamaster2 VM. > > > > > > Do not forget to ipa-replica-manage remove it first. > > > > > > Awesome. This is why I asked. > > > > > 8. Upgrade existing "replicas" to F18 and latest IPA > > version. > > > 9. Establish replication agreements with > now-functioning > > ipamaster. > > > > > > > > > Does that sound right? > > > > > > > > > > See above. > > > > Simo. > > > > > > -- > > Simo Sorce * Red Hat, Inc * New York > > > > > > > > > -- > Simo Sorce * Red Hat, Inc * New York > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From bret.wortman at damascusgrp.com Thu Aug 29 15:26:25 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 11:26:25 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: <521F6473.9040605@redhat.com> References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> Message-ID: On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> A bit of googling has led me to understand that we must have created the >> original server with --selfsign, and that locked us into something bad >> which is now causing us problems. I'm not sure how this happened, since >> we actually created our original instance on a different server, created >> ipamaster as a replica of that one, then ran ipa-ca-install on ipamaster >> to make it the new CA. How did it end up in this state? >> >> Anyway, is there ANY way around this? Can I simply ignore this, break >> the replication agreement as Simo suggested, rebuild ipamaster, >> replicate ipamaster2 to the new ipamaster, and then somehow make >> ipamaster be a CA using Dogtag? Will that screw up all the clients? >> > > I think we should pause and take a look at your installation. > > I'd check all your current masters, whether they are currently working or > not. Look at the value of ra_plugin in /etc/ipa/default.conf. That controls > what IPA thinks the CA is. > > on ipamaster: ra_plugin=dogtag and either that same value or the ra_plugin doesn't exist on the replicas. On ipamaster2, the one I just installed, there is no ra_plugin in the file. > Then check to see if you have dogtag running on any of these systems. This > will include a 2nd 389-ds instance, /etc/dirsrv/slapd-PKI-IPA and, > depending on your distro, a PKI service like pki-tomcatd at pki-tomcat.**service. > You can optionally see if /etc/pki/pki-tomcat exists. > > ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA directory, with files updated fairly recently (within the past 30 minutes - lse.ldif and lse.ldif.bak, others updated yesterday). I also have a pki-tomcatd at .service file and a pki-tomcatd.target. no /etc/pki/pki-tomcat. ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does have pki-tomcatd.target and pki-tomcatd at .service. No /etc/pki/pki-tomcat. > There is currently no way post-install to add a dogtag instance. > > rob > > >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Thu, Aug 29, 2013 at 9:24 AM, Bret Wortman >> >> >> wrote: >> >> Agreed, but not always possible. I had a replica crash hard and it >> wasn't possible to remove it. >> >> In other news: >> >> [ipamaster2]# ipa-ca-install replica-info-ipamaster2.spx.**net.gpg >> A selfsign CA can not be added >> >> Is there a way around this? How can I ensure that I can transfer the >> CA back to ipamaster after it's been erased & reinstalled? >> >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Thu, Aug 29, 2013 at 9:21 AM, Simo Sorce > > wrote: >> >> On Thu, 2013-08-29 at 09:14 -0400, Bret Wortman wrote: >> > On Thu, Aug 29, 2013 at 9:09 AM, Simo Sorce > > wrote: >> > On Thu, 2013-08-29 at 08:07 -0400, Bret Wortman wrote: >> > > Okay, I have a replica built and running. My >> original, >> > "sick" server >> > > is ipamaster and the new one is ipamaster2. All >> I've done >> > thus far on >> > > ipamaster2 is run ipa-replica-install --setup-dns >> > --no-forwarders >> > > replica-info-ipamaster2.foo.**net.gpg. >> > > >> > > >> > > What additional steps do I need to take to ensure >> that the >> > process of >> > > shutting down ipamaster, wiping it out, building it >> up fresh >> > and then >> > > replicating ipamaster2 back to ipamaster and making >> > ipamaster again >> > > the center of the universe and my certificate >> authority work >> > > correctly, cleanly, and with minimal fuss? Given >> the mess I >> > got our >> > > servers already, I figured I should ask before I >> start >> > messing about >> > > today. >> > > >> > > >> > > I think the process should look something like this >> (I don't >> > want you >> > > all thinking I'm looking for someone to do all my >> thinking >> > for me): >> > > >> > > >> > > 1. Take snapshot of ipamaster (just in case) >> > > 2. [ipamaster2]# >> > > >> > ipa-ca-install >> /var/lib/ipa/replica-info-**ipamaster2.foo.net.gpg (I >> > > should've done this during the ipa-ca-install, but >> since the >> > ca step >> > > is so rare, I didn't have it in my wiki notes). >> > > 3. [ipamaster]# reboot >> > > >> > > >> > > This reboot will trigger a Cobbler & Puppet-based >> wipe of >> > the system >> > > and reinstallation of F18 and freeipa-server. While >> that's >> > going on: >> > > >> > > >> > > 4. [ipamaster2]# ipa-replica-prepare >> ipamaster.foo.net >> >> > 1.2.3.4 >> > >> > >> > You need to use ipa-replica-manage to remove the >> original >> > ipamaster >> > before you can prepare to add a new one. >> > >> > After it is fully removed and replica file generated >> you need >> > to restart >> > at yleast 389ds on ipamaster2 this is due to the fact >> that DS >> > does nto >> > purge valid tickets, and it holds a ticket valid for >> the old >> > ipamaster, >> > however when you reinstall the new the name will match >> so >> > replication >> > between ipamaster2 -> ipamaster may fail because >> ipamsater2 >> > has a wrong >> > ticket (using old key you just nuked before the >> reinstall). >> > > >> > >> > >> > >> > Got it. Glad I asked! I'll add these steps to my procedure. >> > >> > > When ipamaster is back up: >> > > >> > > >> > > 5. [ipamaster]# cd /var/lib/ipa && scp >> > >> > >> > You can copy in /root >> > >> > >> > I usually do it in /var/lib/ipa I guess because that's where >> the >> > server puts the file, so it makes it easy for me to remember >> that's >> > where it is. But point taken. >> > >> > > >> > >> ipamaster2:/var/lib/ipa/**replica-info-ipamaster.foo.**net.gpg >> . >> > > 6. [ipamaster]# ipa-replica-install --setup-dns >> > --no-forwarders >> > > --setup-ca replica-info-ipamaster.foo.**net.gpg >> > > >> > > >> > > Usually, there's some reason I need to go back to >> ipamaster2 >> > and >> > > either delete a dns entry or ipa host-del the system. >> > >> > >> > Uh ? Sound like this is going to screw up things, why >> should >> > you delete >> > DNS entries ? >> > ipa host-del of a master is *certainly* going to break >> > replication and >> > basically everything. Is this what you did in your >> old setup ? >> > >> > >> > Only if ipa-replica-install said I needed to. >> >> ok this means you previously uninstalled a replica directly on the >> machine but tdid not remove it from the domain, this is bad >> practice. >> you should use ipa-replica-manage before you retire a machine if >> possible, otherwise you leave dangling replication agreements, DNS >> names, ID ranges (this means you loose ID space), and keys. >> >> > > After the replica install is done: >> > > >> > > >> > > 7. Shut down and delete the ipamaster2 VM. >> > >> > >> > Do not forget to ipa-replica-manage remove it first. >> > >> > >> > Awesome. This is why I asked. >> > >> > > 8. Upgrade existing "replicas" to F18 and latest IPA >> > version. >> > > 9. Establish replication agreements with >> now-functioning >> > ipamaster. >> > > >> > > >> > > Does that sound right? >> > > >> > > >> > >> > See above. >> > >> > Simo. >> > >> > >> > -- >> > Simo Sorce * Red Hat, Inc * New York >> > >> > >> > >> >> >> -- >> Simo Sorce * Red Hat, Inc * New York >> >> >> >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 29 15:40:33 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Aug 2013 11:40:33 -0400 Subject: [Freeipa-users] Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> Message-ID: <521F6B71.1080906@redhat.com> Bret Wortman wrote: > On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > A bit of googling has led me to understand that we must have > created the > original server with --selfsign, and that locked us into > something bad > which is now causing us problems. I'm not sure how this > happened, since > we actually created our original instance on a different server, > created > ipamaster as a replica of that one, then ran ipa-ca-install on > ipamaster > to make it the new CA. How did it end up in this state? > > Anyway, is there ANY way around this? Can I simply ignore this, > break > the replication agreement as Simo suggested, rebuild ipamaster, > replicate ipamaster2 to the new ipamaster, and then somehow make > ipamaster be a CA using Dogtag? Will that screw up all the clients? > > > I think we should pause and take a look at your installation. > > I'd check all your current masters, whether they are currently > working or not. Look at the value of ra_plugin in > /etc/ipa/default.conf. That controls what IPA thinks the CA is. > > on ipamaster: ra_plugin=dogtag > > and either that same value or the ra_plugin doesn't exist on the > replicas. On ipamaster2, the one I just installed, there is no ra_plugin > in the file. > > Then check to see if you have dogtag running on any of these > systems. This will include a 2nd 389-ds instance, > /etc/dirsrv/slapd-PKI-IPA and, depending on your distro, a PKI > service like pki-tomcatd at pki-tomcat.__service. You can optionally > see if /etc/pki/pki-tomcat exists. > > ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA directory, with > files updated fairly recently (within the past 30 minutes - lse.ldif and > lse.ldif.bak, others updated yesterday). I also have a > pki-tomcatd at .service file and a pki-tomcatd.target. no /etc/pki/pki-tomcat. > > ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does have > pki-tomcatd.target and pki-tomcatd at .service. No /etc/pki/pki-tomcat. Ok. When you created the replica file for ipamaster2, did you create it on ipamaster? Only a replica that is a CA can create a replica with a CA. If you generated the replica file on another master, I *think* what you can safely do is this: - prepare a replica on ipamaster for ipamaster2 and copy the file there - on ipamaster2 run ipa-ca-install against the updated replica file rob From bret.wortman at damascusgrp.com Thu Aug 29 15:42:46 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 11:42:46 -0400 Subject: [Freeipa-users] Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> Message-ID: On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden > > wrote: >> >> Bret Wortman wrote: >> >> A bit of googling has led me to understand that we must have >> created the >> original server with --selfsign, and that locked us into >> something bad >> which is now causing us problems. I'm not sure how this >> happened, since >> we actually created our original instance on a different server, >> created >> ipamaster as a replica of that one, then ran ipa-ca-install on >> ipamaster >> to make it the new CA. How did it end up in this state? >> >> Anyway, is there ANY way around this? Can I simply ignore this, >> break >> the replication agreement as Simo suggested, rebuild ipamaster, >> replicate ipamaster2 to the new ipamaster, and then somehow make >> ipamaster be a CA using Dogtag? Will that screw up all the >> clients? >> >> >> I think we should pause and take a look at your installation. >> >> I'd check all your current masters, whether they are currently >> working or not. Look at the value of ra_plugin in >> /etc/ipa/default.conf. That controls what IPA thinks the CA is. >> >> on ipamaster: ra_plugin=dogtag >> >> and either that same value or the ra_plugin doesn't exist on the >> replicas. On ipamaster2, the one I just installed, there is no ra_plugin >> in the file. >> >> Then check to see if you have dogtag running on any of these >> systems. This will include a 2nd 389-ds instance, >> /etc/dirsrv/slapd-PKI-IPA and, depending on your distro, a PKI >> service like pki-tomcatd at pki-tomcat.__**service. You can optionally >> >> see if /etc/pki/pki-tomcat exists. >> >> ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA directory, with >> files updated fairly recently (within the past 30 minutes - lse.ldif and >> lse.ldif.bak, others updated yesterday). I also have a >> pki-tomcatd at .service file and a pki-tomcatd.target. no >> /etc/pki/pki-tomcat. >> >> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does have >> pki-tomcatd.target and pki-tomcatd at .service. No /etc/pki/pki-tomcat. >> > > Ok. When you created the replica file for ipamaster2, did you create it on > ipamaster? Only a replica that is a CA can create a replica with a CA. > > Yes. So I'm not sure what went askew. > If you generated the replica file on another master, I *think* what you > can safely do is this: > > - prepare a replica on ipamaster for ipamaster2 and copy the file there > - on ipamaster2 run ipa-ca-install against the updated replica file > > rob > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpal at redhat.com Thu Aug 29 15:55:01 2013 From: dpal at redhat.com (Dmitri Pal) Date: Thu, 29 Aug 2013 11:55:01 -0400 Subject: [Freeipa-users] Using subdomains (or dots) in hostnames In-Reply-To: References: Message-ID: <521F6ED5.4060201@redhat.com> On 08/19/2013 09:05 AM, Thomas Raehalme wrote: > Hi! > > We are in the process of deploying FreeIPA in our virtual environment. > So far things are working smoothly and I am really impressed by the > solution! > > One question has risen as we have added our first clients to the > system. Because the total number of clients is 50 and going up, we > have divided our servers to subdomains depending on the purpose of the > server, ie. test servers in one subdomain, internal services on > another and so on. There is, however, no need for each subdomain to > have its own IPA server. > > Let's say we're using domain example.com. Adding clients a.example.com > and b.example.com was smooth. Adding client a.sub1.example.com also > had no problems until I tried to get sudoers from the IPA server > (using SSSD and LDAP as suggested). The client fails to find any users > matching the server name. Because the only difference compared to a > fully functional server is the dot in the host name, that's probably > the reason why no sudoers are found for the server in the subdomain? > > For IPA master I am using CentOS 6.4 and > ipa-server-3.0.0-26.el6_4.4.x86_64. The clients are also CentOS 6.4 > with ipa-client-3.0.0-26.el6_4.4.x86_64. > > Any help is appreciated! Please let me know if providing any piece of > information helps. > > Best regards, > Thomas > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users Was there any help provided for this request? -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From rcritten at redhat.com Thu Aug 29 15:58:16 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Aug 2013 11:58:16 -0400 Subject: [Freeipa-users] Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> Message-ID: <521F6F98.3070909@redhat.com> Bret Wortman wrote: > On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden >wrote: > > Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden > > >> wrote: > > Bret Wortman wrote: > > A bit of googling has led me to understand that we must > have > created the > original server with --selfsign, and that locked us into > something bad > which is now causing us problems. I'm not sure how this > happened, since > we actually created our original instance on a > different server, > created > ipamaster as a replica of that one, then ran > ipa-ca-install on > ipamaster > to make it the new CA. How did it end up in this state? > > Anyway, is there ANY way around this? Can I simply > ignore this, > break > the replication agreement as Simo suggested, rebuild > ipamaster, > replicate ipamaster2 to the new ipamaster, and then > somehow make > ipamaster be a CA using Dogtag? Will that screw up all > the clients? > > > I think we should pause and take a look at your installation. > > I'd check all your current masters, whether they are currently > working or not. Look at the value of ra_plugin in > /etc/ipa/default.conf. That controls what IPA thinks the CA is. > > on ipamaster: ra_plugin=dogtag > > and either that same value or the ra_plugin doesn't exist on the > replicas. On ipamaster2, the one I just installed, there is no > ra_plugin > in the file. > > Then check to see if you have dogtag running on any of these > systems. This will include a 2nd 389-ds instance, > /etc/dirsrv/slapd-PKI-IPA and, depending on your distro, a PKI > service like pki-tomcatd at pki-tomcat.____service. You can > optionally > > see if /etc/pki/pki-tomcat exists. > > ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA directory, with > files updated fairly recently (within the past 30 minutes - > lse.ldif and > lse.ldif.bak, others updated yesterday). I also have a > pki-tomcatd at .service file and a pki-tomcatd.target. no > /etc/pki/pki-tomcat. > > ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does have > pki-tomcatd.target and pki-tomcatd at .service. No /etc/pki/pki-tomcat. > > > Ok. When you created the replica file for ipamaster2, did you create > it on ipamaster? Only a replica that is a CA can create a replica > with a CA. > > Yes. So I'm not sure what went askew. Ok. I think we need to see what's in the prepared file. It is just a gpg-encrypted tarball. Can you do something like: gpg -d replica-info-pacer.greyoak.com.gpg |tar xf - This will create a realm_info subdirectory. The file cacert.p12 should be in there. rob From pviktori at redhat.com Thu Aug 29 16:01:39 2013 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 29 Aug 2013 18:01:39 +0200 Subject: [Freeipa-users] Announcing FreeIPA 3.3.1 Message-ID: <521F7063.6060500@redhat.com> The FreeIPA team is proud to announce FreeIPA v3.3.1! This is a bugfix release. It can be downloaded from http://www.freeipa.org/page/Downloads. Fedora 19 builds will be ready soon. == Highlights in 3.3.1 == === Bug fixes === * ipa-server-certinstall now works correctly both with a CA subsystem and in CA-less installations * The --subject option in ipa-server-install is now handled correctly * During installation, directory server tuning is performed correctly on sysV and systemd systems * During installation, the CA service is stopped during configuration file changes to prevent race conditions === Test improvements === * Integration tests for CA-less installation, Kerberos flags, and related Web UI parts were added to the test suite * Test suite now passes after ipa-adtrust-install == Upgrading == === FreeIPA servers with CA installed prior to version 3.1 === Manual upgrade procedure is required for FreeIPA servers installed with version prior to 3.1. Please see http://www.freeipa.org/page/Howto/Dogtag9ToDogtag10Migration for details. === Other FreeIPA servers and clients === An IPA server can be upgraded simply by installing updated rpms. The server does not need to be shut down in advance. Please note that if you are doing the upgrade in special environment (e.g. FedUp) which does not allow running the LDAP server during upgrade process, upgrade scripts need to be run manually after the first boot: # ipa-upgradeconfig # ipa-ldap-updater --upgrade Also note that the performance improvements require an extended set of indexes to be configured. RPM update for an IPA server with a excessive number of users may require several minutes to finish. If you have multiple servers you may upgrade them one at a time. It is expected that all servers will be upgraded in a relatively short period (days or weeks, not months). They should be able to co-exist peacefully but new features will not be available on old servers and enrolling a new client against an old server will result in the SSH keys not being uploaded. Downgrading a server once upgraded is not supported. Upgrading from 2.2.0 and later versions is supported. Upgrading from previous versions is not supported and has not been tested. An enrolled client does not need the new packages installed unless you want to re-enroll it. SSH keys for already installed clients are not uploaded, you will have to re-enroll the client or manually upload the keys. == Feedback == Please provide comments, bugs and other feedback via the freeipa-users mailing list (http://www.redhat.com/mailman/listinfo/freeipa-users) or #freeipa channel on Freenode. == Detailed Changelog since 3.3.0 == === Alexander Bokovoy (1): === * Remove systemd upgrader as it is not used anymore === Ana Krivokapic (4): === * Handle --subject option in ipa-server-install * Fix broken replica installation * Add integration tests for Kerberos Flags * Fix tests which fail after ipa-adtrust-install === Jakub Hrozek (1): === * EXTDOM: Do not overwrite domain_name for INP_SID === Jan Cholasta (12): === * Make PKCS#12 handling in ipa-server-certinstall closer to what other tools do. * Port ipa-server-certinstall to the admintool framework. * Remove unused NSSDatabase and CertDB method find_root_cert_from_pkcs12. * Ignore empty mod error when updating DS SSL config in ipa-server-certinstall. * Replace only the cert instead of the whole NSS DB in ipa-server-certinstall. * Untrack old and track new cert with certmonger in ipa-server-certinstall. * Add --pin option to ipa-server-certinstall. * Ask for PKCS#12 password interactively in ipa-server-certinstall. * Fix nsSaslMapping object class before configuring SASL mappings. * Add --dirman-password option to ipa-server-certinstall. * Fix ipa-server-certinstall usage string. * Fix service-disable in CA-less install. === Martin Kosek (3): === * Prevent *.pyo and *.pyc multilib problems * Remove rpmlint warnings in spec file * Fix selected minor issues in the spec file and license === Nathaniel McCallum (1): === * Bypass ipa-replica-conncheck ssh tests when ssh is not installed === Petr Viktorin (4): === * Allow freeipa-tests to work with older paramiko versions * Add missing license header to ipa-test-config * Add CA-less install tests * Add man pages for testing tools === Petr Vobornik (7): === * Removal of deprecated selenium tests * Add base-id, range-size and range-type options to trust-add dialog * Hide 'New Certificate' action on CA-less install * Web UI integration tests: CA-less * Web UI Integration tests: Kerberos Flags * Web UI integration tests: ID range types * Update idrange search facet after trust creation === Rob Crittenden (1): === * Re-order NULL check in ipa_lockout. === Simo Sorce (3): === * pwd-plugin: Fix ignored return error * kdb-mspac: Fix out of bounds memset * kdb-princ: Fix memory leak === Sumit Bose (1): === * CLDAP: make sure an empty reply is returned on any error === Tomas Babej (6): === * Remove support for IPA deployments with no persistent search * Remove redundant shebangs * Perform dirsrv tuning at platform level * Make CS.cfg edits with CA instance stopped * Fix incorrect error message occurence when re-adding the trust * Log proper error message when defaultNamingContext not found From bret.wortman at damascusgrp.com Thu Aug 29 17:07:17 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Thu, 29 Aug 2013 13:07:17 -0400 Subject: [Freeipa-users] Fwd: Fwd: Scorched earth In-Reply-To: <521F6F98.3070909@redhat.com> References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> Message-ID: What passpharase would this be encrypted with? If it's something I set a year ago and never needed to know again, then we may be screwed. If it's saved somewhere, where should I look? * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden wrote: > Bret Wortman wrote: > >> On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden > >**wrote: >> >> Bret Wortman wrote: >> >> On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden >> >> >> wrote: >> >> Bret Wortman wrote: >> >> A bit of googling has led me to understand that we must >> have >> created the >> original server with --selfsign, and that locked us into >> something bad >> which is now causing us problems. I'm not sure how this >> happened, since >> we actually created our original instance on a >> different server, >> created >> ipamaster as a replica of that one, then ran >> ipa-ca-install on >> ipamaster >> to make it the new CA. How did it end up in this state? >> >> Anyway, is there ANY way around this? Can I simply >> ignore this, >> break >> the replication agreement as Simo suggested, rebuild >> ipamaster, >> replicate ipamaster2 to the new ipamaster, and then >> somehow make >> ipamaster be a CA using Dogtag? Will that screw up all >> the clients? >> >> >> I think we should pause and take a look at your installation. >> >> I'd check all your current masters, whether they are >> currently >> working or not. Look at the value of ra_plugin in >> /etc/ipa/default.conf. That controls what IPA thinks the CA >> is. >> >> on ipamaster: ra_plugin=dogtag >> >> and either that same value or the ra_plugin doesn't exist on the >> replicas. On ipamaster2, the one I just installed, there is no >> ra_plugin >> in the file. >> >> Then check to see if you have dogtag running on any of these >> systems. This will include a 2nd 389-ds instance, >> /etc/dirsrv/slapd-PKI-IPA and, depending on your distro, a >> PKI >> service like pki-tomcatd at pki-tomcat.____**service. You can >> >> optionally >> >> see if /etc/pki/pki-tomcat exists. >> >> ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA directory, >> with >> files updated fairly recently (within the past 30 minutes - >> lse.ldif and >> lse.ldif.bak, others updated yesterday). I also have a >> pki-tomcatd at .service file and a pki-tomcatd.target. no >> /etc/pki/pki-tomcat. >> >> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does have >> pki-tomcatd.target and pki-tomcatd at .service. No >> /etc/pki/pki-tomcat. >> >> >> Ok. When you created the replica file for ipamaster2, did you create >> it on ipamaster? Only a replica that is a CA can create a replica >> with a CA. >> >> Yes. So I'm not sure what went askew. >> > > Ok. I think we need to see what's in the prepared file. It is just a > gpg-encrypted tarball. Can you do something like: > > gpg -d replica-info-pacer.greyoak.**com.gpg |tar xf - > > This will create a realm_info subdirectory. The file cacert.p12 should be > in there. > > rob > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.bezdicka at gooddata.com Thu Aug 29 17:08:59 2013 From: lukas.bezdicka at gooddata.com (=?UTF-8?B?THVrw6HFoSBCZXpkacSNa2E=?=) Date: Thu, 29 Aug 2013 19:08:59 +0200 Subject: [Freeipa-users] Using subdomains (or dots) in hostnames In-Reply-To: <521F6ED5.4060201@redhat.com> References: <521F6ED5.4060201@redhat.com> Message-ID: In our deployment we use subdomains but set NIS domain to main domain: example.com has subdomains na.example.com wa.example.com ... all machines work fine with that but in /etc/sysconfig/network we have NISDOMAIN='example.com' This way sudo rules get evaluated see getent netgroup On Thu, Aug 29, 2013 at 5:55 PM, Dmitri Pal wrote: > On 08/19/2013 09:05 AM, Thomas Raehalme wrote: > > Hi! > > > > We are in the process of deploying FreeIPA in our virtual environment. > > So far things are working smoothly and I am really impressed by the > > solution! > > > > One question has risen as we have added our first clients to the > > system. Because the total number of clients is 50 and going up, we > > have divided our servers to subdomains depending on the purpose of the > > server, ie. test servers in one subdomain, internal services on > > another and so on. There is, however, no need for each subdomain to > > have its own IPA server. > > > > Let's say we're using domain example.com. Adding clients a.example.com > > and b.example.com was smooth. Adding client a.sub1.example.com also > > had no problems until I tried to get sudoers from the IPA server > > (using SSSD and LDAP as suggested). The client fails to find any users > > matching the server name. Because the only difference compared to a > > fully functional server is the dot in the host name, that's probably > > the reason why no sudoers are found for the server in the subdomain? > > > > For IPA master I am using CentOS 6.4 and > > ipa-server-3.0.0-26.el6_4.4.x86_64. The clients are also CentOS 6.4 > > with ipa-client-3.0.0-26.el6_4.4.x86_64. > > > > Any help is appreciated! Please let me know if providing any piece of > > information helps. > > > > Best regards, > > Thomas > > > > _______________________________________________ > > Freeipa-users mailing list > > Freeipa-users at redhat.com > > https://www.redhat.com/mailman/listinfo/freeipa-users > > Was there any help provided for this request? > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager for IdM portfolio > Red Hat Inc. > > > ------------------------------- > Looking to carve out IT costs? > www.redhat.com/carveoutcosts/ > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Thu Aug 29 17:36:32 2013 From: jhrozek at redhat.com (Jakub Hrozek) Date: Thu, 29 Aug 2013 19:36:32 +0200 Subject: [Freeipa-users] Using subdomains (or dots) in hostnames In-Reply-To: References: Message-ID: <20130829173632.GF5929@hendrix.brq.redhat.com> On Mon, Aug 19, 2013 at 04:05:40PM +0300, Thomas Raehalme wrote: > Hi! > > We are in the process of deploying FreeIPA in our virtual environment. > So far things are working smoothly and I am really impressed by the > solution! > > One question has risen as we have added our first clients to the > system. Because the total number of clients is 50 and going up, we > have divided our servers to subdomains depending on the purpose of the > server, ie. test servers in one subdomain, internal services on > another and so on. There is, however, no need for each subdomain to > have its own IPA server. > > Let's say we're using domain example.com. Adding clients a.example.com > and b.example.com was smooth. Adding client a.sub1.example.com also > had no problems until I tried to get sudoers from the IPA server > (using SSSD and LDAP as suggested). The client fails to find any users > matching the server name. Because the only difference compared to a > fully functional server is the dot in the host name, that's probably > the reason why no sudoers are found for the server in the subdomain? > > For IPA master I am using CentOS 6.4 and > ipa-server-3.0.0-26.el6_4.4.x86_64. The clients are also CentOS 6.4 > with ipa-client-3.0.0-26.el6_4.4.x86_64. > > Any help is appreciated! Please let me know if providing any piece of > information helps. > > Best regards, > Thomas Sorry Thomas, the subject line fooled me and I didn't see this might be a SSSD issue. What do you use in nsswitch.conf for sudoers? ldap or sss? If sss, can you also paste your sssd.conf? Can you paste the output of sudo along with the -D parameter to get some debugging? From michal.dwuznik at gmail.com Thu Aug 29 17:41:32 2013 From: michal.dwuznik at gmail.com (=?UTF-8?B?TWljaGHFgiBEd3XFvG5paw==?=) Date: Thu, 29 Aug 2013 19:41:32 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze Message-ID: Hi folks, did anyone succeed in connecting such an old thing recently to freeipa server? Is there a document (or an archive post) about connecting a 'non ipa aware' client step by step? I got as far as woing Kerberos with no issues, hit a wall with ldap part.. Regards Michal -- Michal Dwuznik -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Thu Aug 29 17:51:58 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Aug 2013 13:51:58 -0400 Subject: [Freeipa-users] Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> Message-ID: <521F8A3E.2060003@redhat.com> Bret Wortman wrote: > What passpharase would this be encrypted with? If it's something I set a > year ago and never needed to know again, then we may be screwed. If it's > saved somewhere, where should I look? It's the same passphrase you'd use to install a replica, the DM password. rob > > > _ > _ > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden > > >>__wrote: > > Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden > > > > >>> wrote: > > Bret Wortman wrote: > > A bit of googling has led me to understand > that we must > have > created the > original server with --selfsign, and that > locked us into > something bad > which is now causing us problems. I'm not sure > how this > happened, since > we actually created our original instance on a > different server, > created > ipamaster as a replica of that one, then ran > ipa-ca-install on > ipamaster > to make it the new CA. How did it end up in > this state? > > Anyway, is there ANY way around this? Can I simply > ignore this, > break > the replication agreement as Simo suggested, > rebuild > ipamaster, > replicate ipamaster2 to the new ipamaster, and > then > somehow make > ipamaster be a CA using Dogtag? Will that > screw up all > the clients? > > > I think we should pause and take a look at your > installation. > > I'd check all your current masters, whether they > are currently > working or not. Look at the value of ra_plugin in > /etc/ipa/default.conf. That controls what IPA > thinks the CA is. > > on ipamaster: ra_plugin=dogtag > > and either that same value or the ra_plugin doesn't > exist on the > replicas. On ipamaster2, the one I just installed, > there is no > ra_plugin > in the file. > > Then check to see if you have dogtag running on > any of these > systems. This will include a 2nd 389-ds instance, > /etc/dirsrv/slapd-PKI-IPA and, depending on your > distro, a PKI > service like pki-tomcatd at pki-tomcat.______service. > You can > > optionally > > see if /etc/pki/pki-tomcat exists. > > ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA > directory, with > files updated fairly recently (within the past 30 minutes - > lse.ldif and > lse.ldif.bak, others updated yesterday). I also have a > pki-tomcatd at .service file and a pki-tomcatd.target. no > /etc/pki/pki-tomcat. > > ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does have > pki-tomcatd.target and pki-tomcatd at .service. No > /etc/pki/pki-tomcat. > > > Ok. When you created the replica file for ipamaster2, did > you create > it on ipamaster? Only a replica that is a CA can create a > replica > with a CA. > > Yes. So I'm not sure what went askew. > > > Ok. I think we need to see what's in the prepared file. It is just a > gpg-encrypted tarball. Can you do something like: > > gpg -d replica-info-pacer.greyoak.__com.gpg |tar xf - > > This will create a realm_info subdirectory. The file cacert.p12 > should be in there. > > rob > > From rcritten at redhat.com Thu Aug 29 17:56:31 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Aug 2013 13:56:31 -0400 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: References: Message-ID: <521F8B4F.7000204@redhat.com> Micha? Dwu?nik wrote: > Hi folks, > > did anyone succeed in connecting such an old thing recently to freeipa > server? > > Is there a document (or an archive post) about connecting a 'non ipa > aware' client step by step? > I got as far as woing Kerberos with no issues, hit a wall with ldap part.. You might try this: http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/linux-manual.html rob From michal.dwuznik at gmail.com Thu Aug 29 19:00:20 2013 From: michal.dwuznik at gmail.com (=?UTF-8?B?TWljaGHFgiBEd3XFvG5paw==?=) Date: Thu, 29 Aug 2013 21:00:20 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: <521F8B4F.7000204@redhat.com> References: <521F8B4F.7000204@redhat.com> Message-ID: As for now I have set up a 'known good' client on RH based distro, to get the feeling how the config files look like when configured correctly. Thanks for the nice reference M. On Thu, Aug 29, 2013 at 7:56 PM, Rob Crittenden wrote: > Micha? Dwu?nik wrote: > >> Hi folks, >> >> did anyone succeed in connecting such an old thing recently to freeipa >> server? >> >> Is there a document (or an archive post) about connecting a 'non ipa >> aware' client step by step? >> I got as far as woing Kerberos with no issues, hit a wall with ldap part.. >> > > You might try this: http://docs.fedoraproject.org/** > en-US/Fedora/17/html/FreeIPA_**Guide/linux-manual.html > > rob > > -- Michal Dwuznik -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.dwuznik at gmail.com Thu Aug 29 23:49:37 2013 From: michal.dwuznik at gmail.com (=?UTF-8?B?TWljaGHFgiBEd3XFvG5paw==?=) Date: Fri, 30 Aug 2013 01:49:37 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: References: <521F8B4F.7000204@redhat.com> Message-ID: Ok, going step by step I did the following on squeeze: set up ntp, time synced with ipa server test setup is done on ipa.localdomain (server) client.localdomain (client on Scientific Linux 6.4, looks ok after ipa-client-install, ssh works for test users tester and tester2) client2.localdomain is the Debian Squeeze client added host client2.localdomain on IPA server, added 'managedby', got the keytab and put the 'client2.keytab' in /etc/krb5.keytab on client2 most important part of /etc/krb5.conf: [realms] LOCALDOMAIN = { kdc = ipa.localdomain admin_server = ipa.localdomain } [domain_realm] .localdomain = LOCALDOMAIN localdomain = LOCALDOMAIN default_domain = localdomain [libdefaults] default_realm = LOCALDOMAIN The following lets me think the KRB5 part of the setup is done correctly: root at client2:/etc# kinit admin Password for admin at LOCALDOMAIN: root at client2:/etc# kdestroy root at client2:/etc# kinit tester Password for tester at LOCALDOMAIN: root at client2:/etc# klis -su: klis: command not found root at client2:/etc# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: tester at LOCALDOMAIN Valid starting Expires Service principal 08/30/13 00:35:50 08/31/13 00:35:47 krbtgt/LOCALDOMAIN at LOCALDOMAIN root at client2:/etc# kpasswd tester Password for tester at LOCALDOMAIN: Enter new password: Enter it again: Password changed. I guess that's the point of snapshotting 'KRB done' state (can I be wrong?) DNS for all the hosts involved is similar to: root at client2:/etc# nslookup ipa Server: 192.168.137.29 Address: 192.168.137.29#53 Name: ipa.localdomain Address: 192.168.137.13 root at client2:/etc# nslookup 192.168.137.13 Server: 192.168.137.29 Address: 192.168.137.29#53 13.137.168.192.in-addr.arpa name = ipa.localdomain. Now I guess it's time for certificates, where I do have some doubts... I've added the SSH host keys via web interface, now the cert part: having generated the CSR afte creating the new database: certutil -R -d . -a -g 2048 -s 'CN=client2.localdomain,O=LOCALDOMAIN' (in the /etc/pki dir) I paste the CSR and Issue the certificate for host (/etc/pi contains newly created cert8.db key3.db secmod.db ) Which of those should be used to add the cert to? (like certutil -A -d /etc/pki/nssdb -n "IPA CA" -t CT,C,C -a -i */path/to/* ca.crt) All of the tries result in: root at client2:/etc/pki# certutil -A -d /etc/pki/cert8.db -n "IPA CA" -t CT,C,C -a -i ./ca.crt certutil: function failed: security library: bad database. root at client2:/etc/pki# certutil -A -d /etc/pki/secmod.db -n "IPA CA" -t CT,C,C -a -i ./ca.crt certutil: function failed: security library: bad database. root at client2:/etc/pki# certutil -A -d /etc/pki/key3.db -n "IPA CA" -t CT,C,C -a -i ./ca.crt certutil: function failed: security library: bad database. Could someone show me my mistake? Regards Michal On Thu, Aug 29, 2013 at 9:00 PM, Micha? Dwu?nik wrote: > As for now I have set up a 'known good' client on RH based distro, to get > the feeling how the config files > look like when configured correctly. > > Thanks for the nice reference > > M. > > > On Thu, Aug 29, 2013 at 7:56 PM, Rob Crittenden wrote: > >> Micha? Dwu?nik wrote: >> >>> Hi folks, >>> >>> did anyone succeed in connecting such an old thing recently to freeipa >>> server? >>> >>> Is there a document (or an archive post) about connecting a 'non ipa >>> aware' client step by step? >>> I got as far as woing Kerberos with no issues, hit a wall with ldap >>> part.. >>> >> >> You might try this: http://docs.fedoraproject.org/** >> en-US/Fedora/17/html/FreeIPA_**Guide/linux-manual.html >> >> rob >> >> > > > -- > Michal Dwuznik > -- Michal Dwuznik -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.dwuznik at gmail.com Fri Aug 30 00:12:01 2013 From: michal.dwuznik at gmail.com (=?UTF-8?B?TWljaGHFgiBEd3XFvG5paw==?=) Date: Fri, 30 Aug 2013 02:12:01 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: References: <521F8B4F.7000204@redhat.com> Message-ID: Sorry for quick continuation... Certificate added to nss DB in /etc/pki certutil -A -d /etc/pki/ -n "IPA CA" -t CT,C,C -a -i pki/ca.crt sssd configured according to http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/linux-manual.html How do I test now, before changing PAM options that the pieces fit together? (Sorry for being a bit too tired...) M. On Fri, Aug 30, 2013 at 1:49 AM, Micha? Dwu?nik wrote: > Ok, going step by step I did the following on squeeze: > > set up ntp, time synced with ipa server > > test setup is done on > ipa.localdomain (server) > client.localdomain > (client on Scientific Linux 6.4, looks ok after ipa-client-install, ssh > works for test users tester and tester2) > > client2.localdomain is the Debian Squeeze client > > added host client2.localdomain on IPA server, added 'managedby', got the > keytab and put the 'client2.keytab' in /etc/krb5.keytab on client2 > > most important part of /etc/krb5.conf: > > [realms] > LOCALDOMAIN = { > kdc = ipa.localdomain > admin_server = ipa.localdomain > } > > [domain_realm] > .localdomain = LOCALDOMAIN > localdomain = LOCALDOMAIN > default_domain = localdomain > > [libdefaults] > default_realm = LOCALDOMAIN > > > The following lets me think the KRB5 part of the setup is done correctly: > > root at client2:/etc# kinit admin > Password for admin at LOCALDOMAIN: > root at client2:/etc# kdestroy > root at client2:/etc# kinit tester > Password for tester at LOCALDOMAIN: > root at client2:/etc# klis > -su: klis: command not found > root at client2:/etc# klist > Ticket cache: FILE:/tmp/krb5cc_0 > Default principal: tester at LOCALDOMAIN > > Valid starting Expires Service principal > 08/30/13 00:35:50 08/31/13 00:35:47 krbtgt/LOCALDOMAIN at LOCALDOMAIN > > > root at client2:/etc# kpasswd tester > Password for tester at LOCALDOMAIN: > Enter new password: > Enter it again: > Password changed. > > > I guess that's the point of snapshotting 'KRB done' state (can I be wrong?) > > DNS for all the hosts involved is similar to: > root at client2:/etc# nslookup ipa > Server: 192.168.137.29 > Address: 192.168.137.29#53 > > Name: ipa.localdomain > Address: 192.168.137.13 > > root at client2:/etc# nslookup 192.168.137.13 > Server: 192.168.137.29 > Address: 192.168.137.29#53 > > 13.137.168.192.in-addr.arpa name = ipa.localdomain. > > Now I guess it's time for certificates, where I do have some doubts... > > I've added the SSH host keys via web interface, now the cert part: > > having generated the CSR afte creating the new database: > > certutil -R -d . -a -g 2048 -s 'CN=client2.localdomain,O=LOCALDOMAIN' > (in the /etc/pki dir) I paste the CSR and Issue the certificate for host > > (/etc/pi contains newly created cert8.db key3.db secmod.db ) > > Which of those should be used to add the cert to? > > (like certutil -A -d /etc/pki/nssdb -n "IPA CA" -t CT,C,C -a -i */path/to/ > *ca.crt) > > All of the tries result in: > root at client2:/etc/pki# certutil -A -d /etc/pki/cert8.db -n "IPA CA" -t > CT,C,C -a -i ./ca.crt > certutil: function failed: security library: bad database. > root at client2:/etc/pki# certutil -A -d /etc/pki/secmod.db -n "IPA CA" -t > CT,C,C -a -i ./ca.crt > certutil: function failed: security library: bad database. > root at client2:/etc/pki# certutil -A -d /etc/pki/key3.db -n "IPA CA" -t > CT,C,C -a -i ./ca.crt > certutil: function failed: security library: bad database. > > Could someone show me my mistake? > > Regards > Michal > > > > On Thu, Aug 29, 2013 at 9:00 PM, Micha? Dwu?nik wrote: > >> As for now I have set up a 'known good' client on RH based distro, to get >> the feeling how the config files >> look like when configured correctly. >> >> Thanks for the nice reference >> >> M. >> >> >> On Thu, Aug 29, 2013 at 7:56 PM, Rob Crittenden wrote: >> >>> Micha? Dwu?nik wrote: >>> >>>> Hi folks, >>>> >>>> did anyone succeed in connecting such an old thing recently to freeipa >>>> server? >>>> >>>> Is there a document (or an archive post) about connecting a 'non ipa >>>> aware' client step by step? >>>> I got as far as woing Kerberos with no issues, hit a wall with ldap >>>> part.. >>>> >>> >>> You might try this: http://docs.fedoraproject.org/** >>> en-US/Fedora/17/html/FreeIPA_**Guide/linux-manual.html >>> >>> rob >>> >>> >> >> >> -- >> Michal Dwuznik >> > > > > -- > Michal Dwuznik > -- Michal Dwuznik -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Aug 30 02:04:43 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 29 Aug 2013 22:04:43 -0400 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: References: <521F8B4F.7000204@redhat.com> Message-ID: <521FFDBB.1070205@redhat.com> Micha? Dwu?nik wrote: > Sorry for quick continuation... > > Certificate added to nss DB in /etc/pki > certutil -A -d /etc/pki/ -n "IPA CA" -t CT,C,C -a -i pki/ca.crt > > sssd configured according to > http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/linux-manual.html > > How do I test now, before changing PAM options that the pieces fit together? Perhaps exercise nss with: % id admin % getent passwd admin % getent group admin You can substitute admin for any IPA user or group. And really you can skip the cert step if you want. Unless you have something that will use it we put a cert on the system as a convenience right now. There isn't currently anything using it by default. rob > > > (Sorry for being a bit too tired...) > > M. > > > On Fri, Aug 30, 2013 at 1:49 AM, Micha? Dwu?nik > > wrote: > > Ok, going step by step I did the following on squeeze: > > set up ntp, time synced with ipa server > > test setup is done on > ipa.localdomain (server) > client.localdomain > (client on Scientific Linux 6.4, looks ok after ipa-client-install, > ssh works for test users tester and tester2) > > client2.localdomain is the Debian Squeeze client > > added host client2.localdomain on IPA server, added 'managedby', got > the keytab and put the 'client2.keytab' in /etc/krb5.keytab on client2 > > most important part of /etc/krb5.conf: > > [realms] > LOCALDOMAIN = { > kdc = ipa.localdomain > admin_server = ipa.localdomain > } > > [domain_realm] > .localdomain = LOCALDOMAIN > localdomain = LOCALDOMAIN > default_domain = localdomain > > [libdefaults] > default_realm = LOCALDOMAIN > > > The following lets me think the KRB5 part of the setup is done > correctly: > > root at client2:/etc# kinit admin > Password for admin at LOCALDOMAIN: > root at client2:/etc# kdestroy > root at client2:/etc# kinit tester > Password for tester at LOCALDOMAIN: > root at client2:/etc# klis > -su: klis: command not found > root at client2:/etc# klist > Ticket cache: FILE:/tmp/krb5cc_0 > Default principal: tester at LOCALDOMAIN > > Valid starting Expires Service principal > 08/30/13 00:35:50 08/31/13 00:35:47 krbtgt/LOCALDOMAIN at LOCALDOMAIN > > > root at client2:/etc# kpasswd tester > Password for tester at LOCALDOMAIN: > Enter new password: > Enter it again: > Password changed. > > > I guess that's the point of snapshotting 'KRB done' state (can I be > wrong?) > > DNS for all the hosts involved is similar to: > root at client2:/etc# nslookup ipa > Server: 192.168.137.29 > Address: 192.168.137.29#53 > > Name: ipa.localdomain > Address: 192.168.137.13 > > root at client2:/etc# nslookup 192.168.137.13 > Server: 192.168.137.29 > Address: 192.168.137.29#53 > > 13.137.168.192.in-addr.arpa name = ipa.localdomain. > > Now I guess it's time for certificates, where I do have some doubts... > > I've added the SSH host keys via web interface, now the cert part: > > having generated the CSR afte creating the new database: > > certutil -R -d . -a -g 2048 -s 'CN=client2.localdomain,O=LOCALDOMAIN' > (in the /etc/pki dir) I paste the CSR and Issue the certificate for host > > (/etc/pi contains newly created cert8.db key3.db secmod.db ) > > Which of those should be used to add the cert to? > > (like certutil -A -d /etc/pki/nssdb -n "IPA CA" -t CT,C,C -a -i > /|/path/to/|/ca.crt) > > All of the tries result in: > root at client2:/etc/pki# certutil -A -d /etc/pki/cert8.db -n "IPA CA" > -t CT,C,C -a -i ./ca.crt > certutil: function failed: security library: bad database. > root at client2:/etc/pki# certutil -A -d /etc/pki/secmod.db -n "IPA CA" > -t CT,C,C -a -i ./ca.crt > certutil: function failed: security library: bad database. > root at client2:/etc/pki# certutil -A -d /etc/pki/key3.db -n "IPA CA" > -t CT,C,C -a -i ./ca.crt > certutil: function failed: security library: bad database. > > Could someone show me my mistake? > > Regards > Michal > > > > On Thu, Aug 29, 2013 at 9:00 PM, Micha? Dwu?nik > > wrote: > > As for now I have set up a 'known good' client on RH based > distro, to get the feeling how the config files > look like when configured correctly. > > Thanks for the nice reference > > M. > > > On Thu, Aug 29, 2013 at 7:56 PM, Rob Crittenden > > wrote: > > Micha? Dwu?nik wrote: > > Hi folks, > > did anyone succeed in connecting such an old thing > recently to freeipa > server? > > Is there a document (or an archive post) about > connecting a 'non ipa > aware' client step by step? > I got as far as woing Kerberos with no issues, hit a > wall with ldap part.. > > > You might try this: > http://docs.fedoraproject.org/__en-US/Fedora/17/html/FreeIPA___Guide/linux-manual.html > > > rob > > > > > -- > Michal Dwuznik > > > > > -- > Michal Dwuznik > > > > > -- > Michal Dwuznik > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > From bret.wortman at damascusgrp.com Fri Aug 30 08:23:35 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Fri, 30 Aug 2013 04:23:35 -0400 Subject: [Freeipa-users] Fwd: Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> <521F99FE.6060205@redhat.com> Message-ID: Morning update. I made the change Rob suggested to /etc/ipa/default.conf, which appeared to work, but didn't quite. It asked me to back out the whole server installation and start over: [ipamaster2]# ipa-ca-install --skip-conncheck replica-info-ipamaster2.foo.net.gpg Directory Manager (existing master) password: COnfiguring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds [1/16]: creating certificate server user [2/16]: configuring certificate server instance ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exit status 1 Your system may be partly configured. Run/usr/sbin/ipa-server-install --uninstall to clean up. Configuration of CA failed. [ipamaster2]# Which uninstallation & cleanup I did. Now, when trying to re-install the replica file: [ipamaster2]# ipa-replica-install --setup-dns --no-forwarders --setup-ca /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg Directory manager (existing master) password: Run connection check to master Check connection from replica to remote master 'ipamaster.foo.net': Directory Service: Unsecure port (389): OK Directory Service: Secure port (686): OK Kerberos KDC: TCP (88): OK Kerberos Kpasswd: TCP (464): OK HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK The followign list of ports use UDP protocol and would need to be checked manually: Kerberos KDC: UDP (88): SKIPPED Kerberos Kpasswd: UDP (464): SKIPPED Connection from replica to master is OK. Start listening on required ports for remote master check Get credentials to log in to remote master admin at FOO.NET password: Check SSH connection to remote master Execute check on remote master Check connection from master to remote replica 'ipamaster2.foo.net': Directory Service: Unsecure port (389): OK Directory Service: Secure port (686): OK Kerberos KDC: TCP (88): OK Kerberos KDC: UDP (88): OK Kerberos Kpasswd: TCP (464): OK Kerberos Kpasswd: UDP (464): OK HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK Connection from master to replica is OK. Connection check OK The host ipamaster2.foo.net already exists on the master server. You should remove it before proceeding: % ipa host-del ipamaster2.foo.net ipa : ERROR Could not resolve hostname ipamaster.foo.net using DNS Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.) Continue? [no]: *yes* [ipamaster2]# host ipamaster.foo.net ipamaster.foo.net has address 1.2.3.4 No matter what answer I give to the "Continue?" prompt, it just exits. "nslookup" returns the same value, and I have three different nameservers configured for this host (including ipamaster and two of the older replicas). And this message is the one that has prompted me to want to delete hosts before installing in the past, Simo. Any thoughts on how best to proceed now? *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Thu, Aug 29, 2013 at 2:59 PM, Rob Crittenden wrote: > Bret Wortman wrote: > >> Okay, I got the cacert.p12 (turns out it was taking my passphrase, but >> the messages looked like errors to my addled eyes). This system is on a >> different network, so getting the file transferred would take me about >> 24 hours. Is there something I can get that'll tell you what you need >> but is plaintext? >> > > Ok, that's fine. > > Try this. Set ra_plugin to dogtag in /etc/ipa/default.conf. This will let > it get past the error and it should install a CA. I'm trying to think worst > case scenario what it might do and I'm not coming up with anything. I think > the worst that happens is that adding a CA fails later. > > rob > > >> I tried this and hope this subset of information is helpful: >> >> # openssl pkcs12 -in cacert.p12 -out cacert.pem.bdw -cacerts -nokeys >> # cat cacert.pem.bdw >> Bag Attributes: >> subject=/O=FOO.NET/CN=**Certificate < >> http://FOO.NET/CN=Certificate**> Authority/ >> issuer=/O=FOO.NET/CN=**Certificate < >> http://FOO.NET/CN=Certificate**> Authority >> >> -----BEGIN CERTIFICATE----- >> MIIDgzCCA... >> ...Iwk4r >> -----END CERTIFICATE----- >> # openssl pkcs12 -in cacert.p12 -out cert.pem.bdw -clcerts -nokeys >> # cat cert.pem.bdw >> Bag Attributes: >> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE 6C A7 2E 53 >> 88 >> friendlyName: caSigningCert cert-pki-ca >> subject=/O=FOO.NET/CN=**Certificate < >> http://FOO.NET/CN=Certificate**> Authority >> issuer=/O=FOO.NET/CN=**Certificate < >> http://FOO.NET/CN=Certificate**> Authority >> >> -----BEGIN CERTIFICATE----- >> MIIDgzCCA... >> ...Iwk4r >> -----END CERTIFICATE----- >> Bag Attributes: >> localKeyID: 88 BF DF 56 30 BB A9 47 12 D4 5F 7B AE 39 DC BF CF F5 92 >> 22 >> friendlyName: ocspSigningCert cert-pki-ca >> subject=/O=FOO.NET/CN=OCSP Subsystem >> issuer=/O=FOO.NET/CN=**Certificate < >> http://FOO.NET/CN=Certificate**> Authority >> >> -----BEGIN CERTIFICATE----- >> MIIDYTCCA... >> ...wlr4Q= >> -----END CERTIFICATE----- >> Bag Attributes: >> localKeyID: B5 3B 27 CC 57 72 45 E2 8D 46 C9 5E E1 C0 50 DF 2D 11 62 >> 0E >> friendlyName: subsystemCert cert-pki-ca >> subject=/O=FOO.NET/CN=CA Subsystem >> issuer=/O=FOO.NET/CN=**Certificate < >> http://FOO.NET/CN=Certificate**> Authority >> >> -----BEGIN CERTIFICATE----- >> MIIDaTCCA... >> ...BxqqA== >> -----END CERTIFICATE----- >> Bag Attributes: >> localKeyID: 1F 69 62 C7 88 D8 95 2A B1 7D 61 F9 10 87 14 D0 76 Ba B9 >> 44 >> friendlyName: auditSigningCert cert-pki-ca >> subject=/O=FOO.NET/CN=CA Audit >> issuer=/O=FOO.NET/CN=**CertificateAUthority >> >> > >> >> -----BEGIN CERTIFICATE----- >> MIIDRDCCA... >> ...EAd+Ug7 >> -----END CERTIFICATE----- >> # openssl pkcs12 -in cacert.p12 -out key.pem.bdw -nocerts >> # cat key.pem.bdw >> Bag Attributes >> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE 6C A7 2E 53 >> 88 >> friendlyName: CN=Certificate Authority,O=FOO.NET >> >> Key Attributes: >> -----BEGIN ENCRYPTED PRIVATE KEY----- >> MIIFDjBAB... >> ...XLtoD8= >> -----END ENCRYPTED PRIVATE KEY----- >> Bag Attributes >> localKeyID: >> friendlyName: CN=OCSP Subsystem,O=FOO.NET >> >> Key Attributes: >> -----BEGIN ENCRYPTED PRIVATE KEY----- >> : >> -----END ENCRYPTED PRIVATE KEY----- >> Bag Attributes >> localKeyID: >> friendlyName: CN=CA Subsystem,O=FOO.NET >> >> Key Attributes: >> Bag Attributes >> localKeyID: >> friendlyName: CN=CA Audit,O=FOO.NET >> >> Key Attributes: >> >> If you need to see anything else, please let me know. >> >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden > > wrote: >> >> Bret Wortman wrote: >> >> On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden >> >> >>** >> __wrote: >> >> Bret Wortman wrote: >> >> On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden >> >> > >> > > >>**> wrote: >> >> Bret Wortman wrote: >> >> A bit of googling has led me to understand >> that we must >> have >> created the >> original server with --selfsign, and that >> locked us into >> something bad >> which is now causing us problems. I'm not sure >> how this >> happened, since >> we actually created our original instance on a >> different server, >> created >> ipamaster as a replica of that one, then ran >> ipa-ca-install on >> ipamaster >> to make it the new CA. How did it end up in >> this state? >> >> Anyway, is there ANY way around this? Can I >> simply >> ignore this, >> break >> the replication agreement as Simo suggested, >> rebuild >> ipamaster, >> replicate ipamaster2 to the new ipamaster, and >> then >> somehow make >> ipamaster be a CA using Dogtag? Will that >> screw up all >> the clients? >> >> >> I think we should pause and take a look at your >> installation. >> >> I'd check all your current masters, whether they >> are currently >> working or not. Look at the value of ra_plugin in >> /etc/ipa/default.conf. That controls what IPA >> thinks the CA is. >> >> on ipamaster: ra_plugin=dogtag >> >> and either that same value or the ra_plugin doesn't >> exist on the >> replicas. On ipamaster2, the one I just installed, >> there is no >> ra_plugin >> in the file. >> >> Then check to see if you have dogtag running on >> any of these >> systems. This will include a 2nd 389-ds instance, >> /etc/dirsrv/slapd-PKI-IPA and, depending on your >> distro, a PKI >> service like pki-tomcatd at pki-tomcat.______** >> service. >> >> You can >> >> optionally >> >> see if /etc/pki/pki-tomcat exists. >> >> ipamaster definitely has a /etc/dirsrv/slapd-PKI-IPA >> directory, with >> files updated fairly recently (within the past 30 >> minutes - >> lse.ldif and >> lse.ldif.bak, others updated yesterday). I also have a >> pki-tomcatd at .service file and a pki-tomcatd.target. no >> /etc/pki/pki-tomcat. >> >> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. It does >> have >> pki-tomcatd.target and pki-tomcatd at .service. No >> /etc/pki/pki-tomcat. >> >> >> Ok. When you created the replica file for ipamaster2, did >> you create >> it on ipamaster? Only a replica that is a CA can create a >> replica >> with a CA. >> >> Yes. So I'm not sure what went askew. >> >> >> Ok. I think we need to see what's in the prepared file. It is just a >> gpg-encrypted tarball. Can you do something like: >> >> gpg -d replica-info-pacer.greyoak.__**com.gpg |tar xf - >> >> >> This will create a realm_info subdirectory. The file cacert.p12 >> should be in there. >> >> rob >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Fri Aug 30 08:36:34 2013 From: jhrozek at redhat.com (Jakub Hrozek) Date: Fri, 30 Aug 2013 10:36:34 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: <521FFDBB.1070205@redhat.com> References: <521F8B4F.7000204@redhat.com> <521FFDBB.1070205@redhat.com> Message-ID: <20130830083634.GG5929@hendrix.brq.redhat.com> On Thu, Aug 29, 2013 at 10:04:43PM -0400, Rob Crittenden wrote: > Micha? Dwu?nik wrote: > >Sorry for quick continuation... > > > >Certificate added to nss DB in /etc/pki > >certutil -A -d /etc/pki/ -n "IPA CA" -t CT,C,C -a -i pki/ca.crt > > > >sssd configured according to > >http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/linux-manual.html > > > >How do I test now, before changing PAM options that the pieces fit together? > > Perhaps exercise nss with: > > % id admin > % getent passwd admin > % getent group admin > > You can substitute admin for any IPA user or group. > > And really you can skip the cert step if you want. Unless you have > something that will use it we put a cert on the system as a > convenience right now. There isn't currently anything using it by > default. > > rob On the client, one piece of functionality where you need the cert are password migrations from LDAP to IPA. I don't think that's your case, though. From pviktori at redhat.com Fri Aug 30 09:03:49 2013 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 30 Aug 2013 11:03:49 +0200 Subject: [Freeipa-users] Fwd: Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> <521F99FE.6060205@redhat.com> Message-ID: <52205FF5.4040506@redhat.com> On 08/30/2013 10:23 AM, Bret Wortman wrote: > Morning update. I made the change Rob suggested to > /etc/ipa/default.conf, which appeared to work, but didn't quite. It > asked me to back out the whole server installation and start over: > > [ipamaster2]# ipa-ca-install --skip-conncheck > replica-info-ipamaster2.foo.net.gpg > Directory Manager (existing master) password: > > COnfiguring certificate server (pki-tomcatd): Estimated time 3 minutes > 30 seconds > [1/16]: creating certificate server user > [2/16]: configuring certificate server instance > ipa : CRITICAL failed to configure ca instance Command > '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exit status 1 > > Your system may be partly configured. > Run/usr/sbin/ipa-server-install --uninstall to clean up. Can you look into /var/log/ipareplica-ca-install.log? It should have more information on what caused pkispawn to fail. > Configuration of CA failed. > [ipamaster2]# > > Which uninstallation & cleanup I did. > > Now, when trying to re-install the > replica file: > > [ipamaster2]# ipa-replica-install --setup-dns --no-forwarders --setup-ca > /var/lib/ipa/replica-info-ipamaster2.foo.net.gpg > Directory manager (existing master) password: > > Run connection check to master > Check connection from replica to remote master 'ipamaster.foo.net > ': > Directory Service: Unsecure port (389): OK > Directory Service: Secure port (686): OK > Kerberos KDC: TCP (88): OK > Kerberos Kpasswd: TCP (464): OK > HTTP Server: Unsecure port (80): OK > HTTP Server: Secure port (443): OK > > The followign list of ports use UDP protocol and would need to be > checked manually: > Kerberos KDC: UDP (88): SKIPPED > Kerberos Kpasswd: UDP (464): SKIPPED > > Connection from replica to master is OK. > Start listening on required ports for remote master check > Get credentials to log in to remote master > admin at FOO.NET password: > > Check SSH connection to remote master > Execute check on remote master > Check connection from master to remote replica 'ipamaster2.foo.net > ': > Directory Service: Unsecure port (389): OK > Directory Service: Secure port (686): OK > Kerberos KDC: TCP (88): OK > Kerberos KDC: UDP (88): OK > Kerberos Kpasswd: TCP (464): OK > Kerberos Kpasswd: UDP (464): OK > HTTP Server: Unsecure port (80): OK > HTTP Server: Secure port (443): OK > > Connection from master to replica is OK. > > Connection check OK > The host ipamaster2.foo.net already exists > on the master server. > You should remove it before proceeding: > % ipa host-del ipamaster2.foo.net > ipa : ERROR Could not resolve hostname ipamaster.foo.net > using DNS Clients may not function properly. > Please check your DNS setup. (Note that this check queries IPA DNS > directly and ignores /etc/hosts.) > Continue? [no]: *yes* > [ipamaster2]# host ipamaster.foo.net > ipamaster.foo.net has address 1.2.3.4 > > No matter what answer I give to the "Continue?" prompt, it just exits. > "nslookup" returns the same value, and I have three different > nameservers configured for this host (including ipamaster and two of the > older replicas). The error that caused the installation to fail is that ipamaster2.foo.net already exists on the master server. The DNS warning and its "Continue?" prompt is unrelated, but the order of the output is very confusing. I've filed ticket 3889 for this. Anyway, to do this DNS resolution check you'd need to explicitly ask for the IPA server: $ dig @ipamaster.foo.net ipamaster2.foo.net > And this message is the one that has prompted me to want to delete hosts > before installing in the past, Simo. > > Any thoughts on how best to proceed now? I believe you do need to delete he host at this point, but I'd rather have Rob or Simo confirm. > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Thu, Aug 29, 2013 at 2:59 PM, Rob Crittenden > wrote: > > Bret Wortman wrote: > > Okay, I got the cacert.p12 (turns out it was taking my > passphrase, but > the messages looked like errors to my addled eyes). This system > is on a > different network, so getting the file transferred would take me > about > 24 hours. Is there something I can get that'll tell you what you > need > but is plaintext? > > > Ok, that's fine. > > Try this. Set ra_plugin to dogtag in /etc/ipa/default.conf. This > will let it get past the error and it should install a CA. I'm > trying to think worst case scenario what it might do and I'm not > coming up with anything. I think the worst that happens is that > adding a CA fails later. > > rob > > > I tried this and hope this subset of information is helpful: > > # openssl pkcs12 -in cacert.p12 -out cacert.pem.bdw -cacerts -nokeys > # cat cacert.pem.bdw > Bag Attributes: > subject=/O=FOO.NET/CN=__Certificate > > Authority/ > issuer=/O=FOO.NET/CN=__Certificate > > Authority > > -----BEGIN CERTIFICATE----- > MIIDgzCCA... > ...Iwk4r > -----END CERTIFICATE----- > # openssl pkcs12 -in cacert.p12 -out cert.pem.bdw -clcerts -nokeys > # cat cert.pem.bdw > Bag Attributes: > localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE > 6C A7 2E 53 88 > friendlyName: caSigningCert cert-pki-ca > subject=/O=FOO.NET/CN=__Certificate > > Authority > issuer=/O=FOO.NET/CN=__Certificate > > Authority > > -----BEGIN CERTIFICATE----- > MIIDgzCCA... > ...Iwk4r > -----END CERTIFICATE----- > Bag Attributes: > localKeyID: 88 BF DF 56 30 BB A9 47 12 D4 5F 7B AE 39 DC > BF CF F5 92 22 > friendlyName: ocspSigningCert cert-pki-ca > subject=/O=FOO.NET/CN=OCSP > Subsystem > issuer=/O=FOO.NET/CN=__Certificate > > Authority > > -----BEGIN CERTIFICATE----- > MIIDYTCCA... > ...wlr4Q= > -----END CERTIFICATE----- > Bag Attributes: > localKeyID: B5 3B 27 CC 57 72 45 E2 8D 46 C9 5E E1 C0 50 > DF 2D 11 62 0E > friendlyName: subsystemCert cert-pki-ca > subject=/O=FOO.NET/CN=CA > Subsystem > issuer=/O=FOO.NET/CN=__Certificate > > Authority > > -----BEGIN CERTIFICATE----- > MIIDaTCCA... > ...BxqqA== > -----END CERTIFICATE----- > Bag Attributes: > localKeyID: 1F 69 62 C7 88 D8 95 2A B1 7D 61 F9 10 87 14 > D0 76 Ba B9 44 > friendlyName: auditSigningCert cert-pki-ca > subject=/O=FOO.NET/CN=CA > Audit > issuer=/O=FOO.NET/CN=__CertificateAUthority > > > > > -----BEGIN CERTIFICATE----- > MIIDRDCCA... > ...EAd+Ug7 > -----END CERTIFICATE----- > # openssl pkcs12 -in cacert.p12 -out key.pem.bdw -nocerts > # cat key.pem.bdw > Bag Attributes > localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE > 6C A7 2E 53 88 > friendlyName: CN=Certificate Authority,O=FOO.NET > > > Key Attributes: > -----BEGIN ENCRYPTED PRIVATE KEY----- > MIIFDjBAB... > ...XLtoD8= > -----END ENCRYPTED PRIVATE KEY----- > Bag Attributes > localKeyID: > friendlyName: CN=OCSP Subsystem,O=FOO.NET > > > Key Attributes: > -----BEGIN ENCRYPTED PRIVATE KEY----- > : > -----END ENCRYPTED PRIVATE KEY----- > Bag Attributes > localKeyID: > friendlyName: CN=CA Subsystem,O=FOO.NET > > > Key Attributes: > Bag Attributes > localKeyID: > friendlyName: CN=CA Audit,O=FOO.NET > > > Key Attributes: > > If you need to see anything else, please let me know. > > > _ > _ > *Bret Wortman* > > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden > > >> wrote: > > Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden > > > > >>>____wrote: > > Bret Wortman wrote: > > On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden > > > >> > > > > >>>__> wrote: > > Bret Wortman wrote: > > A bit of googling has led me to > understand > that we must > have > created the > original server with --selfsign, and that > locked us into > something bad > which is now causing us problems. I'm > not sure > how this > happened, since > we actually created our original > instance on a > different server, > created > ipamaster as a replica of that one, > then ran > ipa-ca-install on > ipamaster > to make it the new CA. How did it end > up in > this state? > > Anyway, is there ANY way around this? > Can I simply > ignore this, > break > the replication agreement as Simo > suggested, > rebuild > ipamaster, > replicate ipamaster2 to the new > ipamaster, and > then > somehow make > ipamaster be a CA using Dogtag? Will that > screw up all > the clients? > > > I think we should pause and take a look > at your > installation. > > I'd check all your current masters, > whether they > are currently > working or not. Look at the value of > ra_plugin in > /etc/ipa/default.conf. That controls what IPA > thinks the CA is. > > on ipamaster: ra_plugin=dogtag > > and either that same value or the ra_plugin > doesn't > exist on the > replicas. On ipamaster2, the one I just installed, > there is no > ra_plugin > in the file. > > Then check to see if you have dogtag > running on > any of these > systems. This will include a 2nd 389-ds > instance, > /etc/dirsrv/slapd-PKI-IPA and, depending > on your > distro, a PKI > service like > pki-tomcatd at pki-tomcat.________service. > > You can > > optionally > > see if /etc/pki/pki-tomcat exists. > > ipamaster definitely has a > /etc/dirsrv/slapd-PKI-IPA > directory, with > files updated fairly recently (within the past > 30 minutes - > lse.ldif and > lse.ldif.bak, others updated yesterday). I > also have a > pki-tomcatd at .service file and a > pki-tomcatd.target. no > /etc/pki/pki-tomcat. > > ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. > It does have > pki-tomcatd.target and pki-tomcatd at .service. No > /etc/pki/pki-tomcat. > > > Ok. When you created the replica file for > ipamaster2, did > you create > it on ipamaster? Only a replica that is a CA can > create a > replica > with a CA. > > Yes. So I'm not sure what went askew. > > > Ok. I think we need to see what's in the prepared file. It > is just a > gpg-encrypted tarball. Can you do something like: > > gpg -d replica-info-pacer.greyoak.____com.gpg |tar xf - > > > This will create a realm_info subdirectory. The file cacert.p12 > should be in there. > > rob > > > > > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users > -- Petr? From bret.wortman at damascusgrp.com Fri Aug 30 09:17:55 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Fri, 30 Aug 2013 05:17:55 -0400 Subject: [Freeipa-users] Fwd: Fwd: Fwd: Scorched earth In-Reply-To: <52205FF5.4040506@redhat.com> References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> <521F99FE.6060205@redhat.com> <52205FF5.4040506@redhat.com> Message-ID: On Fri, Aug 30, 2013 at 5:03 AM, Petr Viktorin wrote: > On 08/30/2013 10:23 AM, Bret Wortman wrote: > >> Morning update. I made the change Rob suggested to >> /etc/ipa/default.conf, which appeared to work, but didn't quite. It >> asked me to back out the whole server installation and start over: >> >> [ipamaster2]# ipa-ca-install --skip-conncheck >> replica-info-ipamaster2.foo.**net.gpg >> Directory Manager (existing master) password: >> >> COnfiguring certificate server (pki-tomcatd): Estimated time 3 minutes >> 30 seconds >> [1/16]: creating certificate server user >> [2/16]: configuring certificate server instance >> ipa : CRITICAL failed to configure ca instance Command >> '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exit >> status 1 >> >> Your system may be partly configured. >> Run/usr/sbin/ipa-server-**install --uninstall to clean up. >> > > Can you look into /var/log/ipareplica-ca-**install.log? It should have > more information on what caused pkispawn to fail. > > Here's what it looks like: Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. Installation failed. 2013-08-30T07:37:24Z DEBUG stderr=pkispawn : WARNING ...... unable to validate security domain user/password through REST interface. Interface not available pkispawn : ERROR ...... Exception from Java Configuration Servlet: Failed to obtain installation token from security domain: java.lang.NullPointerException 2013-08-30T07:37:24Z CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exist status 1 2013-08-30T07:37:24Z INFO File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 619, in run_script return_value = main_function() File "/usr/sbin/ipa-ca-install", line 182, in main config, dogtag_master_ds_port, postinstall=True) : File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 744, in __spawn_instance raise RuntimeError('Configuration of CA failed') 2013-08-30T07:37:24Z INFO The ipa-ca-install command failed, exception: RuntimeError: Configuration of CA Failed > Configuration of CA failed. >> [ipamaster2]# >> >> Which uninstallation & cleanup I did. >> >> Now, when trying to re-install the >> replica file: >> >> [ipamaster2]# ipa-replica-install --setup-dns --no-forwarders --setup-ca >> /var/lib/ipa/replica-info-**ipamaster2.foo.net.gpg >> Directory manager (existing master) password: >> >> Run connection check to master >> Check connection from replica to remote master 'ipamaster.foo.net >> ': >> >> Directory Service: Unsecure port (389): OK >> Directory Service: Secure port (686): OK >> Kerberos KDC: TCP (88): OK >> Kerberos Kpasswd: TCP (464): OK >> HTTP Server: Unsecure port (80): OK >> HTTP Server: Secure port (443): OK >> >> The followign list of ports use UDP protocol and would need to be >> checked manually: >> Kerberos KDC: UDP (88): SKIPPED >> Kerberos Kpasswd: UDP (464): SKIPPED >> >> Connection from replica to master is OK. >> Start listening on required ports for remote master check >> Get credentials to log in to remote master >> admin at FOO.NET password: >> >> >> Check SSH connection to remote master >> Execute check on remote master >> Check connection from master to remote replica 'ipamaster2.foo.net >> ': >> >> Directory Service: Unsecure port (389): OK >> Directory Service: Secure port (686): OK >> Kerberos KDC: TCP (88): OK >> Kerberos KDC: UDP (88): OK >> Kerberos Kpasswd: TCP (464): OK >> Kerberos Kpasswd: UDP (464): OK >> HTTP Server: Unsecure port (80): OK >> HTTP Server: Secure port (443): OK >> >> Connection from master to replica is OK. >> >> Connection check OK >> The host ipamaster2.foo.net already exists >> >> on the master server. >> You should remove it before proceeding: >> % ipa host-del ipamaster2.foo.net >> >> ipa : ERROR Could not resolve hostname ipamaster.foo.net >> using DNS Clients may not function properly. >> >> Please check your DNS setup. (Note that this check queries IPA DNS >> directly and ignores /etc/hosts.) >> Continue? [no]: *yes* >> [ipamaster2]# host ipamaster.foo.net >> ipamaster.foo.net has address 1.2.3.4 >> >> >> No matter what answer I give to the "Continue?" prompt, it just exits. >> "nslookup" returns the same value, and I have three different >> nameservers configured for this host (including ipamaster and two of the >> older replicas). >> > > The error that caused the installation to fail is that ipamaster2.foo.netalready exists on the master server. > > The DNS warning and its "Continue?" prompt is unrelated, but the order of > the output is very confusing. I've filed ticket 3889 for this. > Anyway, to do this DNS resolution check you'd need to explicitly ask for > the IPA server: > $ dig @ipamaster.foo.net ipamaster2.foo.net > > Right, and that's a problem because the main reason for needing to clone this box was that the master server has got its tentacles in a wad and can't actually delete any replication agreements. It's missing some who have died and left the configuration, and others who are uncommunicative. Any new attempts to remove replicas just get stuck behind this logjam and add to the fun. > And this message is the one that has prompted me to want to delete hosts >> before installing in the past, Simo. >> >> Any thoughts on how best to proceed now? >> > > I believe you do need to delete he host at this point, but I'd rather have > Rob or Simo confirm. > > Instead, I believe I'll change this system's name and let it become yet another new replica instead. That should take less time in the long run, as long as I don't end up running out of IP addresses on this subnet. ;-) I dug into this job's log file earlier and found that the DNS problem was the script trying to resolve against a nonexistent replica (see prior comment). And the exit even when I tell it to continue appears to be an inline sys.exit(3) call on line 629 of the ipa-replica-install script, which exits if a replication agreement already exists. The script is supposed to print a message to that effect but for some reason, that message isn't getting to stdout before the sys.exit(3) occurs. Another reason to give this temporary box a new identity and try again. Thanks for your help! > *Bret Wortman* >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Thu, Aug 29, 2013 at 2:59 PM, Rob Crittenden > > wrote: >> >> Bret Wortman wrote: >> >> Okay, I got the cacert.p12 (turns out it was taking my >> passphrase, but >> the messages looked like errors to my addled eyes). This system >> is on a >> different network, so getting the file transferred would take me >> about >> 24 hours. Is there something I can get that'll tell you what you >> need >> but is plaintext? >> >> >> Ok, that's fine. >> >> Try this. Set ra_plugin to dogtag in /etc/ipa/default.conf. This >> will let it get past the error and it should install a CA. I'm >> trying to think worst case scenario what it might do and I'm not >> coming up with anything. I think the worst that happens is that >> adding a CA fails later. >> >> rob >> >> >> I tried this and hope this subset of information is helpful: >> >> # openssl pkcs12 -in cacert.p12 -out cacert.pem.bdw -cacerts >> -nokeys >> # cat cacert.pem.bdw >> Bag Attributes: >> subject=/O=FOO.NET/CN=__**Certificate >> >> > >> Authority/ >> issuer=/O=FOO.NET/CN=__**Certificate >> >> > >> Authority >> >> >> -----BEGIN CERTIFICATE----- >> MIIDgzCCA... >> ...Iwk4r >> -----END CERTIFICATE----- >> # openssl pkcs12 -in cacert.p12 -out cert.pem.bdw -clcerts -nokeys >> # cat cert.pem.bdw >> Bag Attributes: >> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE >> 6C A7 2E 53 88 >> friendlyName: caSigningCert cert-pki-ca >> subject=/O=FOO.NET/CN=__**Certificate >> >> > >> Authority >> issuer=/O=FOO.NET/CN=__**Certificate >> >> > >> Authority >> >> >> -----BEGIN CERTIFICATE----- >> MIIDgzCCA... >> ...Iwk4r >> -----END CERTIFICATE----- >> Bag Attributes: >> localKeyID: 88 BF DF 56 30 BB A9 47 12 D4 5F 7B AE 39 DC >> BF CF F5 92 22 >> friendlyName: ocspSigningCert cert-pki-ca >> subject=/O=FOO.NET/CN=OCSP >> Subsystem >> issuer=/O=FOO.NET/CN=__**Certificate >> >> > >> Authority >> >> >> -----BEGIN CERTIFICATE----- >> MIIDYTCCA... >> ...wlr4Q= >> -----END CERTIFICATE----- >> Bag Attributes: >> localKeyID: B5 3B 27 CC 57 72 45 E2 8D 46 C9 5E E1 C0 50 >> DF 2D 11 62 0E >> friendlyName: subsystemCert cert-pki-ca >> subject=/O=FOO.NET/CN=CA >> Subsystem >> issuer=/O=FOO.NET/CN=__**Certificate >> >> > >> Authority >> >> >> -----BEGIN CERTIFICATE----- >> MIIDaTCCA... >> ...BxqqA== >> -----END CERTIFICATE----- >> Bag Attributes: >> localKeyID: 1F 69 62 C7 88 D8 95 2A B1 7D 61 F9 10 87 14 >> D0 76 Ba B9 44 >> friendlyName: auditSigningCert cert-pki-ca >> subject=/O=FOO.NET/CN=CA >> Audit >> issuer=/O=FOO.NET/CN=__**CertificateAUthority >> >> > >> >> >> >> >> >> >> -----BEGIN CERTIFICATE----- >> MIIDRDCCA... >> ...EAd+Ug7 >> -----END CERTIFICATE----- >> # openssl pkcs12 -in cacert.p12 -out key.pem.bdw -nocerts >> # cat key.pem.bdw >> Bag Attributes >> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE >> 6C A7 2E 53 88 >> friendlyName: CN=Certificate Authority,O=FOO.NET >> >> >> >> Key Attributes: >> -----BEGIN ENCRYPTED PRIVATE KEY----- >> MIIFDjBAB... >> ...XLtoD8= >> -----END ENCRYPTED PRIVATE KEY----- >> Bag Attributes >> localKeyID: >> friendlyName: CN=OCSP Subsystem,O=FOO.NET >> >> >> Key Attributes: >> -----BEGIN ENCRYPTED PRIVATE KEY----- >> : >> -----END ENCRYPTED PRIVATE KEY----- >> Bag Attributes >> localKeyID: >> friendlyName: CN=CA Subsystem,O=FOO.NET >> >> >> Key Attributes: >> Bag Attributes >> localKeyID: >> friendlyName: CN=CA Audit,O=FOO.NET >> >> >> Key Attributes: >> >> If you need to see anything else, please let me know. >> >> >> _ >> _ >> *Bret Wortman* >> >> >> http://damascusgrp.com/ >> http://about.me/wortmanbret >> >> >> On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden >> >> >> wrote: >> >> Bret Wortman wrote: >> >> On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden >> >> > >> > > >>**>____wrote: >> >> >> Bret Wortman wrote: >> >> On Thu, Aug 29, 2013 at 11:10 AM, Rob Crittenden >> > > > >> > > >> >> > >> > > > >> > >>**>__> wrote: >> >> Bret Wortman wrote: >> >> A bit of googling has led me to >> understand >> that we must >> have >> created the >> original server with --selfsign, and >> that >> locked us into >> something bad >> which is now causing us problems. I'm >> not sure >> how this >> happened, since >> we actually created our original >> instance on a >> different server, >> created >> ipamaster as a replica of that one, >> then ran >> ipa-ca-install on >> ipamaster >> to make it the new CA. How did it end >> up in >> this state? >> >> Anyway, is there ANY way around this? >> Can I simply >> ignore this, >> break >> the replication agreement as Simo >> suggested, >> rebuild >> ipamaster, >> replicate ipamaster2 to the new >> ipamaster, and >> then >> somehow make >> ipamaster be a CA using Dogtag? Will >> that >> screw up all >> the clients? >> >> >> I think we should pause and take a look >> at your >> installation. >> >> I'd check all your current masters, >> whether they >> are currently >> working or not. Look at the value of >> ra_plugin in >> /etc/ipa/default.conf. That controls what >> IPA >> thinks the CA is. >> >> on ipamaster: ra_plugin=dogtag >> >> and either that same value or the ra_plugin >> doesn't >> exist on the >> replicas. On ipamaster2, the one I just >> installed, >> there is no >> ra_plugin >> in the file. >> >> Then check to see if you have dogtag >> running on >> any of these >> systems. This will include a 2nd 389-ds >> instance, >> /etc/dirsrv/slapd-PKI-IPA and, depending >> on your >> distro, a PKI >> service like >> pki-tomcatd at pki-tomcat._______**_service. >> >> >> You can >> >> optionally >> >> see if /etc/pki/pki-tomcat exists. >> >> ipamaster definitely has a >> /etc/dirsrv/slapd-PKI-IPA >> directory, with >> files updated fairly recently (within the past >> 30 minutes - >> lse.ldif and >> lse.ldif.bak, others updated yesterday). I >> also have a >> pki-tomcatd at .service file and a >> pki-tomcatd.target. no >> /etc/pki/pki-tomcat. >> >> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. >> It does have >> pki-tomcatd.target and pki-tomcatd at .service. No >> /etc/pki/pki-tomcat. >> >> >> Ok. When you created the replica file for >> ipamaster2, did >> you create >> it on ipamaster? Only a replica that is a CA can >> create a >> replica >> with a CA. >> >> Yes. So I'm not sure what went askew. >> >> >> Ok. I think we need to see what's in the prepared file. It >> is just a >> gpg-encrypted tarball. Can you do something like: >> >> gpg -d replica-info-pacer.greyoak.___**_com.gpg |tar xf - >> >> >> >> This will create a realm_info subdirectory. The file >> cacert.p12 >> should be in there. >> >> rob >> >> >> >> >> >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> >> > > -- > Petr? > > > ______________________________**_________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/**mailman/listinfo/freeipa-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Fri Aug 30 10:08:17 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Fri, 30 Aug 2013 06:08:17 -0400 Subject: [Freeipa-users] Fwd: Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> <521F99FE.6060205@redhat.com> <52205FF5.4040506@redhat.com> Message-ID: Now, look. Maybe I'm being thick, or maybe my master is really messed up. But when I take a replica file to a freshly-created replica with a virgin OS and install that replica file as the first act on that system and it fails with the same error, basically that "a replication agreement for this host already exists," I have to say, "of *course* a replication agreement already exists, because I *just bleeding set it up!*" Obviously, I am not understanding something. I asked the master to create a file for a new replica. I take that file to a new replica. Installation fails because the agreement exists. So I have to go back to the master to delete the replication agreement before I can add the replica? Now that I think of it, this has been what I've had to do for every replica along the way, but until the master got hosed, it hasn't been a show-stopper. Given that I can't remove agreements from my server (which is the root problem I'm trying to solve by creating one replica that can hold my actual data long enough for me to nuke that master and restore it from the temporary replica), is there any way to make this work? Sorry. It's been a long ten or eleven days. I'm not upset, just punchy. I woke up at 1AM with an idea that hasn't panned out, so it's also been a long morning already. * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Fri, Aug 30, 2013 at 5:17 AM, Bret Wortman wrote: > On Fri, Aug 30, 2013 at 5:03 AM, Petr Viktorin wrote: > >> On 08/30/2013 10:23 AM, Bret Wortman wrote: >> >>> Morning update. I made the change Rob suggested to >>> /etc/ipa/default.conf, which appeared to work, but didn't quite. It >>> asked me to back out the whole server installation and start over: >>> >>> [ipamaster2]# ipa-ca-install --skip-conncheck >>> replica-info-ipamaster2.foo.**net.gpg >>> Directory Manager (existing master) password: >>> >>> COnfiguring certificate server (pki-tomcatd): Estimated time 3 minutes >>> 30 seconds >>> [1/16]: creating certificate server user >>> [2/16]: configuring certificate server instance >>> ipa : CRITICAL failed to configure ca instance Command >>> '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exit >>> status 1 >>> >>> Your system may be partly configured. >>> Run/usr/sbin/ipa-server-**install --uninstall to clean up. >>> >> >> Can you look into /var/log/ipareplica-ca-**install.log? It should have >> more information on what caused pkispawn to fail. >> >> Here's what it looks like: > > Storing deployment configuration into > /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. > Installation failed. > > > 2013-08-30T07:37:24Z DEBUG stderr=pkispawn : WARNING ...... unable to > validate security domain user/password through REST interface. Interface > not available > pkispawn : ERROR ...... Exception from Java Configuration Servlet: > Failed to obtain installation token from security domain: > java.lang.NullPointerException > > 2013-08-30T07:37:24Z CRITICAL failed to configure ca instance Command > '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exist status > 1 > 2013-08-30T07:37:24Z INFO File > "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line > 619, in run_script > return_value = main_function() > > File "/usr/sbin/ipa-ca-install", line 182, in main > config, dogtag_master_ds_port, postinstall=True) > : > File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", > line 744, in __spawn_instance > raise RuntimeError('Configuration of CA failed') > 2013-08-30T07:37:24Z INFO The ipa-ca-install command failed, exception: > RuntimeError: Configuration of CA Failed > > >> Configuration of CA failed. >>> [ipamaster2]# >>> >>> Which uninstallation & cleanup I did. >>> >>> Now, when trying to re-install the >>> replica file: >>> >>> [ipamaster2]# ipa-replica-install --setup-dns --no-forwarders --setup-ca >>> /var/lib/ipa/replica-info-**ipamaster2.foo.net.gpg >>> Directory manager (existing master) password: >>> >>> Run connection check to master >>> Check connection from replica to remote master 'ipamaster.foo.net >>> ': >>> >>> Directory Service: Unsecure port (389): OK >>> Directory Service: Secure port (686): OK >>> Kerberos KDC: TCP (88): OK >>> Kerberos Kpasswd: TCP (464): OK >>> HTTP Server: Unsecure port (80): OK >>> HTTP Server: Secure port (443): OK >>> >>> The followign list of ports use UDP protocol and would need to be >>> checked manually: >>> Kerberos KDC: UDP (88): SKIPPED >>> Kerberos Kpasswd: UDP (464): SKIPPED >>> >>> Connection from replica to master is OK. >>> Start listening on required ports for remote master check >>> Get credentials to log in to remote master >>> admin at FOO.NET password: >>> >>> >>> Check SSH connection to remote master >>> Execute check on remote master >>> Check connection from master to remote replica 'ipamaster2.foo.net >>> ': >>> >>> Directory Service: Unsecure port (389): OK >>> Directory Service: Secure port (686): OK >>> Kerberos KDC: TCP (88): OK >>> Kerberos KDC: UDP (88): OK >>> Kerberos Kpasswd: TCP (464): OK >>> Kerberos Kpasswd: UDP (464): OK >>> HTTP Server: Unsecure port (80): OK >>> HTTP Server: Secure port (443): OK >>> >>> Connection from master to replica is OK. >>> >>> Connection check OK >>> The host ipamaster2.foo.net already exists >>> >>> on the master server. >>> You should remove it before proceeding: >>> % ipa host-del ipamaster2.foo.net >>> >>> ipa : ERROR Could not resolve hostname ipamaster.foo.net >>> using DNS Clients may not function properly. >>> >>> Please check your DNS setup. (Note that this check queries IPA DNS >>> directly and ignores /etc/hosts.) >>> Continue? [no]: *yes* >>> [ipamaster2]# host ipamaster.foo.net >>> ipamaster.foo.net has address 1.2.3.4 >>> >>> >>> No matter what answer I give to the "Continue?" prompt, it just exits. >>> "nslookup" returns the same value, and I have three different >>> nameservers configured for this host (including ipamaster and two of the >>> older replicas). >>> >> >> The error that caused the installation to fail is that ipamaster2.foo.netalready exists on the master server. >> >> The DNS warning and its "Continue?" prompt is unrelated, but the order of >> the output is very confusing. I've filed ticket 3889 for this. >> Anyway, to do this DNS resolution check you'd need to explicitly ask for >> the IPA server: >> $ dig @ipamaster.foo.net ipamaster2.foo.net >> >> Right, and that's a problem because the main reason for needing to clone > this box was that the master server has got its tentacles in a wad and > can't actually delete any replication agreements. It's missing some who > have died and left the configuration, and others who are uncommunicative. > Any new attempts to remove replicas just get stuck behind this logjam and > add to the fun. > > >> And this message is the one that has prompted me to want to delete hosts >>> before installing in the past, Simo. >>> >>> Any thoughts on how best to proceed now? >>> >> >> I believe you do need to delete he host at this point, but I'd rather >> have Rob or Simo confirm. >> >> Instead, I believe I'll change this system's name and let it become yet > another new replica instead. That should take less time in the long run, as > long as I don't end up running out of IP addresses on this subnet. ;-) > > I dug into this job's log file earlier and found that the DNS problem was > the script trying to resolve against a nonexistent replica (see prior > comment). And the exit even when I tell it to continue appears to be an > inline sys.exit(3) call on line 629 of the ipa-replica-install script, > which exits if a replication agreement already exists. The script is > supposed to print a message to that effect but for some reason, that > message isn't getting to stdout before the sys.exit(3) occurs. > > Another reason to give this temporary box a new identity and try again. > > Thanks for your help! > > >> *Bret Wortman* >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Thu, Aug 29, 2013 at 2:59 PM, Rob Crittenden >> > wrote: >>> >>> Bret Wortman wrote: >>> >>> Okay, I got the cacert.p12 (turns out it was taking my >>> passphrase, but >>> the messages looked like errors to my addled eyes). This system >>> is on a >>> different network, so getting the file transferred would take me >>> about >>> 24 hours. Is there something I can get that'll tell you what you >>> need >>> but is plaintext? >>> >>> >>> Ok, that's fine. >>> >>> Try this. Set ra_plugin to dogtag in /etc/ipa/default.conf. This >>> will let it get past the error and it should install a CA. I'm >>> trying to think worst case scenario what it might do and I'm not >>> coming up with anything. I think the worst that happens is that >>> adding a CA fails later. >>> >>> rob >>> >>> >>> I tried this and hope this subset of information is helpful: >>> >>> # openssl pkcs12 -in cacert.p12 -out cacert.pem.bdw -cacerts >>> -nokeys >>> # cat cacert.pem.bdw >>> Bag Attributes: >>> subject=/O=FOO.NET/CN=__**Certificate >>> >>> > >>> Authority/ >>> issuer=/O=FOO.NET/CN=__**Certificate >>> >>> > >>> Authority >>> >>> >>> -----BEGIN CERTIFICATE----- >>> MIIDgzCCA... >>> ...Iwk4r >>> -----END CERTIFICATE----- >>> # openssl pkcs12 -in cacert.p12 -out cert.pem.bdw -clcerts >>> -nokeys >>> # cat cert.pem.bdw >>> Bag Attributes: >>> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE >>> 6C A7 2E 53 88 >>> friendlyName: caSigningCert cert-pki-ca >>> subject=/O=FOO.NET/CN=__**Certificate >>> >>> > >>> Authority >>> issuer=/O=FOO.NET/CN=__**Certificate >>> >>> > >>> Authority >>> >>> >>> -----BEGIN CERTIFICATE----- >>> MIIDgzCCA... >>> ...Iwk4r >>> -----END CERTIFICATE----- >>> Bag Attributes: >>> localKeyID: 88 BF DF 56 30 BB A9 47 12 D4 5F 7B AE 39 DC >>> BF CF F5 92 22 >>> friendlyName: ocspSigningCert cert-pki-ca >>> subject=/O=FOO.NET/CN=OCSP >>> Subsystem >>> issuer=/O=FOO.NET/CN=__**Certificate >>> >>> > >>> Authority >>> >>> >>> -----BEGIN CERTIFICATE----- >>> MIIDYTCCA... >>> ...wlr4Q= >>> -----END CERTIFICATE----- >>> Bag Attributes: >>> localKeyID: B5 3B 27 CC 57 72 45 E2 8D 46 C9 5E E1 C0 50 >>> DF 2D 11 62 0E >>> friendlyName: subsystemCert cert-pki-ca >>> subject=/O=FOO.NET/CN=CA >>> Subsystem >>> issuer=/O=FOO.NET/CN=__**Certificate >>> >>> > >>> Authority >>> >>> >>> -----BEGIN CERTIFICATE----- >>> MIIDaTCCA... >>> ...BxqqA== >>> -----END CERTIFICATE----- >>> Bag Attributes: >>> localKeyID: 1F 69 62 C7 88 D8 95 2A B1 7D 61 F9 10 87 14 >>> D0 76 Ba B9 44 >>> friendlyName: auditSigningCert cert-pki-ca >>> subject=/O=FOO.NET/CN=CA >>> Audit >>> issuer=/O=FOO.NET/CN=__**CertificateAUthority >>> >>> > >>> >>> >>> >>> >> >>> >>> -----BEGIN CERTIFICATE----- >>> MIIDRDCCA... >>> ...EAd+Ug7 >>> -----END CERTIFICATE----- >>> # openssl pkcs12 -in cacert.p12 -out key.pem.bdw -nocerts >>> # cat key.pem.bdw >>> Bag Attributes >>> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE >>> 6C A7 2E 53 88 >>> friendlyName: CN=Certificate Authority,O=FOO.NET >>> >>> >>> >>> Key Attributes: >>> -----BEGIN ENCRYPTED PRIVATE KEY----- >>> MIIFDjBAB... >>> ...XLtoD8= >>> -----END ENCRYPTED PRIVATE KEY----- >>> Bag Attributes >>> localKeyID: >>> friendlyName: CN=OCSP Subsystem,O=FOO.NET >>> >>> >>> Key Attributes: >>> -----BEGIN ENCRYPTED PRIVATE KEY----- >>> : >>> -----END ENCRYPTED PRIVATE KEY----- >>> Bag Attributes >>> localKeyID: >>> friendlyName: CN=CA Subsystem,O=FOO.NET >>> >>> >>> Key Attributes: >>> Bag Attributes >>> localKeyID: >>> friendlyName: CN=CA Audit,O=FOO.NET >>> >>> >>> Key Attributes: >>> >>> If you need to see anything else, please let me know. >>> >>> >>> _ >>> _ >>> *Bret Wortman* >>> >>> >>> http://damascusgrp.com/ >>> http://about.me/wortmanbret >>> >>> >>> On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden >>> >>> >> >>> wrote: >>> >>> Bret Wortman wrote: >>> >>> On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden >>> >>> > >>> >> >> >>**>____wrote: >>> >>> >>> Bret Wortman wrote: >>> >>> On Thu, Aug 29, 2013 at 11:10 AM, Rob >>> Crittenden >>> >> >> > >>> >> >> >> >>> >> >>> >> > >> >>> >> >>**>__> wrote: >>> >>> Bret Wortman wrote: >>> >>> A bit of googling has led me to >>> understand >>> that we must >>> have >>> created the >>> original server with --selfsign, and >>> that >>> locked us into >>> something bad >>> which is now causing us problems. I'm >>> not sure >>> how this >>> happened, since >>> we actually created our original >>> instance on a >>> different server, >>> created >>> ipamaster as a replica of that one, >>> then ran >>> ipa-ca-install on >>> ipamaster >>> to make it the new CA. How did it end >>> up in >>> this state? >>> >>> Anyway, is there ANY way around this? >>> Can I simply >>> ignore this, >>> break >>> the replication agreement as Simo >>> suggested, >>> rebuild >>> ipamaster, >>> replicate ipamaster2 to the new >>> ipamaster, and >>> then >>> somehow make >>> ipamaster be a CA using Dogtag? Will >>> that >>> screw up all >>> the clients? >>> >>> >>> I think we should pause and take a look >>> at your >>> installation. >>> >>> I'd check all your current masters, >>> whether they >>> are currently >>> working or not. Look at the value of >>> ra_plugin in >>> /etc/ipa/default.conf. That controls what >>> IPA >>> thinks the CA is. >>> >>> on ipamaster: ra_plugin=dogtag >>> >>> and either that same value or the ra_plugin >>> doesn't >>> exist on the >>> replicas. On ipamaster2, the one I just >>> installed, >>> there is no >>> ra_plugin >>> in the file. >>> >>> Then check to see if you have dogtag >>> running on >>> any of these >>> systems. This will include a 2nd 389-ds >>> instance, >>> /etc/dirsrv/slapd-PKI-IPA and, depending >>> on your >>> distro, a PKI >>> service like >>> pki-tomcatd at pki-tomcat._______**_service. >>> >>> >>> You can >>> >>> optionally >>> >>> see if /etc/pki/pki-tomcat exists. >>> >>> ipamaster definitely has a >>> /etc/dirsrv/slapd-PKI-IPA >>> directory, with >>> files updated fairly recently (within the past >>> 30 minutes - >>> lse.ldif and >>> lse.ldif.bak, others updated yesterday). I >>> also have a >>> pki-tomcatd at .service file and a >>> pki-tomcatd.target. no >>> /etc/pki/pki-tomcat. >>> >>> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. >>> It does have >>> pki-tomcatd.target and pki-tomcatd at .service. >>> No >>> /etc/pki/pki-tomcat. >>> >>> >>> Ok. When you created the replica file for >>> ipamaster2, did >>> you create >>> it on ipamaster? Only a replica that is a CA can >>> create a >>> replica >>> with a CA. >>> >>> Yes. So I'm not sure what went askew. >>> >>> >>> Ok. I think we need to see what's in the prepared file. It >>> is just a >>> gpg-encrypted tarball. Can you do something like: >>> >>> gpg -d replica-info-pacer.greyoak.___**_com.gpg |tar xf - >>> >>> >>> >>> This will create a realm_info subdirectory. The file >>> cacert.p12 >>> should be in there. >>> >>> rob >>> >>> >>> >>> >>> >>> >>> >>> ______________________________**_________________ >>> Freeipa-users mailing list >>> Freeipa-users at redhat.com >>> https://www.redhat.com/**mailman/listinfo/freeipa-users >>> >>> >> >> -- >> Petr? >> >> >> ______________________________**_________________ >> Freeipa-users mailing list >> Freeipa-users at redhat.com >> https://www.redhat.com/**mailman/listinfo/freeipa-users >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bret.wortman at damascusgrp.com Fri Aug 30 11:09:05 2013 From: bret.wortman at damascusgrp.com (Bret Wortman) Date: Fri, 30 Aug 2013 07:09:05 -0400 Subject: [Freeipa-users] Fwd: Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <1377741685734.9f79c198@Nodemailer> <1377781758.2917.43.camel@willson.li.ssimo.org> <1377782510.2917.48.camel@willson.li.ssimo.org> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> <521F99FE.6060205@redhat.com> <52205FF5.4040506@redhat.com> Message-ID: Still odder ... I went ahead and tried to delete the agreement: [ipamaster]# ipa-replica-manage del ipamaster3.foo.net --force 'ipamaster.foo.net' has no replication agreement for 'ipamaster3.foo.net' [ipamaster]# Dug back into the script and realized upon further reading (and widening my read to more of the code) that found was being set True elsewhere -- where it was complaining about how ipamaster knew about ipamaster3 already. Fair enough. So I hopped on over there and removed it. Which worked. And now the script proceeds much better. Guess the third cup of coffee helped. CA configuration still failed, though, at the same place as before (though executed as part of ipa-replica-install --setup-ca this time): [2/17]: configuring certificate server instance ipa : CRITICAL failed to configure ca instance Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpnq_J4d' returned non-zero exit status 1 Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Configuration of CA failed. *This* time, I'm not going to run the --uninstall command until someone on the team tells me to do so.... * * *Bret Wortman* http://damascusgrp.com/ http://about.me/wortmanbret On Fri, Aug 30, 2013 at 6:08 AM, Bret Wortman wrote: > Now, look. Maybe I'm being thick, or maybe my master is really messed up. > But when I take a replica file to a freshly-created replica with a virgin > OS and install that replica file as the first act on that system and it > fails with the same error, basically that "a replication agreement for this > host already exists," I have to say, "of *course* a replication agreement > already exists, because I *just bleeding set it up!*" > > Obviously, I am not understanding something. I asked the master to create > a file for a new replica. I take that file to a new replica. Installation > fails because the agreement exists. So I have to go back to the master to > delete the replication agreement before I can add the replica? Now that I > think of it, this has been what I've had to do for every replica along the > way, but until the master got hosed, it hasn't been a show-stopper. > > Given that I can't remove agreements from my server (which is the root > problem I'm trying to solve by creating one replica that can hold my actual > data long enough for me to nuke that master and restore it from the > temporary replica), is there any way to make this work? > > Sorry. It's been a long ten or eleven days. I'm not upset, just punchy. I > woke up at 1AM with an idea that hasn't panned out, so it's also been a > long morning already. > > > * > * > *Bret Wortman* > > http://damascusgrp.com/ > http://about.me/wortmanbret > > > On Fri, Aug 30, 2013 at 5:17 AM, Bret Wortman < > bret.wortman at damascusgrp.com> wrote: > >> On Fri, Aug 30, 2013 at 5:03 AM, Petr Viktorin wrote: >> >>> On 08/30/2013 10:23 AM, Bret Wortman wrote: >>> >>>> Morning update. I made the change Rob suggested to >>>> /etc/ipa/default.conf, which appeared to work, but didn't quite. It >>>> asked me to back out the whole server installation and start over: >>>> >>>> [ipamaster2]# ipa-ca-install --skip-conncheck >>>> replica-info-ipamaster2.foo.**net.gpg >>>> Directory Manager (existing master) password: >>>> >>>> COnfiguring certificate server (pki-tomcatd): Estimated time 3 minutes >>>> 30 seconds >>>> [1/16]: creating certificate server user >>>> [2/16]: configuring certificate server instance >>>> ipa : CRITICAL failed to configure ca instance Command >>>> '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exit >>>> status 1 >>>> >>>> Your system may be partly configured. >>>> Run/usr/sbin/ipa-server-**install --uninstall to clean up. >>>> >>> >>> Can you look into /var/log/ipareplica-ca-**install.log? It should have >>> more information on what caused pkispawn to fail. >>> >>> Here's what it looks like: >> >> Storing deployment configuration into >> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg. >> Installation failed. >> >> >> 2013-08-30T07:37:24Z DEBUG stderr=pkispawn : WARNING ...... unable to >> validate security domain user/password through REST interface. Interface >> not available >> pkispawn : ERROR ...... Exception from Java Configuration Servlet: >> Failed to obtain installation token from security domain: >> java.lang.NullPointerException >> >> 2013-08-30T07:37:24Z CRITICAL failed to configure ca instance Command >> '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exist status >> 1 >> 2013-08-30T07:37:24Z INFO File >> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line >> 619, in run_script >> return_value = main_function() >> >> File "/usr/sbin/ipa-ca-install", line 182, in main >> config, dogtag_master_ds_port, postinstall=True) >> : >> File >> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line >> 744, in __spawn_instance >> raise RuntimeError('Configuration of CA failed') >> 2013-08-30T07:37:24Z INFO The ipa-ca-install command failed, exception: >> RuntimeError: Configuration of CA Failed >> >> >>> Configuration of CA failed. >>>> [ipamaster2]# >>>> >>>> Which uninstallation & cleanup I did. >>>> >>>> Now, when trying to re-install the >>>> replica file: >>>> >>>> [ipamaster2]# ipa-replica-install --setup-dns --no-forwarders --setup-ca >>>> /var/lib/ipa/replica-info-**ipamaster2.foo.net.gpg >>>> Directory manager (existing master) password: >>>> >>>> Run connection check to master >>>> Check connection from replica to remote master 'ipamaster.foo.net >>>> ': >>>> >>>> Directory Service: Unsecure port (389): OK >>>> Directory Service: Secure port (686): OK >>>> Kerberos KDC: TCP (88): OK >>>> Kerberos Kpasswd: TCP (464): OK >>>> HTTP Server: Unsecure port (80): OK >>>> HTTP Server: Secure port (443): OK >>>> >>>> The followign list of ports use UDP protocol and would need to be >>>> checked manually: >>>> Kerberos KDC: UDP (88): SKIPPED >>>> Kerberos Kpasswd: UDP (464): SKIPPED >>>> >>>> Connection from replica to master is OK. >>>> Start listening on required ports for remote master check >>>> Get credentials to log in to remote master >>>> admin at FOO.NET password: >>>> >>>> >>>> Check SSH connection to remote master >>>> Execute check on remote master >>>> Check connection from master to remote replica 'ipamaster2.foo.net >>>> ': >>>> >>>> Directory Service: Unsecure port (389): OK >>>> Directory Service: Secure port (686): OK >>>> Kerberos KDC: TCP (88): OK >>>> Kerberos KDC: UDP (88): OK >>>> Kerberos Kpasswd: TCP (464): OK >>>> Kerberos Kpasswd: UDP (464): OK >>>> HTTP Server: Unsecure port (80): OK >>>> HTTP Server: Secure port (443): OK >>>> >>>> Connection from master to replica is OK. >>>> >>>> Connection check OK >>>> The host ipamaster2.foo.net already exists >>>> >>>> on the master server. >>>> You should remove it before proceeding: >>>> % ipa host-del ipamaster2.foo.net >>>> >>>> ipa : ERROR Could not resolve hostname ipamaster.foo.net >>>> using DNS Clients may not function properly. >>>> >>>> Please check your DNS setup. (Note that this check queries IPA DNS >>>> directly and ignores /etc/hosts.) >>>> Continue? [no]: *yes* >>>> [ipamaster2]# host ipamaster.foo.net >>>> ipamaster.foo.net has address 1.2.3.4 >>>> >>>> >>>> No matter what answer I give to the "Continue?" prompt, it just exits. >>>> "nslookup" returns the same value, and I have three different >>>> nameservers configured for this host (including ipamaster and two of the >>>> older replicas). >>>> >>> >>> The error that caused the installation to fail is that >>> ipamaster2.foo.net already exists on the master server. >>> >>> The DNS warning and its "Continue?" prompt is unrelated, but the order >>> of the output is very confusing. I've filed ticket 3889 for this. >>> Anyway, to do this DNS resolution check you'd need to explicitly ask for >>> the IPA server: >>> $ dig @ipamaster.foo.net ipamaster2.foo.net >>> >>> Right, and that's a problem because the main reason for needing to clone >> this box was that the master server has got its tentacles in a wad and >> can't actually delete any replication agreements. It's missing some who >> have died and left the configuration, and others who are uncommunicative. >> Any new attempts to remove replicas just get stuck behind this logjam and >> add to the fun. >> >> >>> And this message is the one that has prompted me to want to delete hosts >>>> before installing in the past, Simo. >>>> >>>> Any thoughts on how best to proceed now? >>>> >>> >>> I believe you do need to delete he host at this point, but I'd rather >>> have Rob or Simo confirm. >>> >>> Instead, I believe I'll change this system's name and let it become yet >> another new replica instead. That should take less time in the long run, as >> long as I don't end up running out of IP addresses on this subnet. ;-) >> >> I dug into this job's log file earlier and found that the DNS problem was >> the script trying to resolve against a nonexistent replica (see prior >> comment). And the exit even when I tell it to continue appears to be an >> inline sys.exit(3) call on line 629 of the ipa-replica-install script, >> which exits if a replication agreement already exists. The script is >> supposed to print a message to that effect but for some reason, that >> message isn't getting to stdout before the sys.exit(3) occurs. >> >> Another reason to give this temporary box a new identity and try again. >> >> Thanks for your help! >> >> >>> *Bret Wortman* >>>> >>>> http://damascusgrp.com/ >>>> http://about.me/wortmanbret >>>> >>>> >>>> On Thu, Aug 29, 2013 at 2:59 PM, Rob Crittenden >>> > wrote: >>>> >>>> Bret Wortman wrote: >>>> >>>> Okay, I got the cacert.p12 (turns out it was taking my >>>> passphrase, but >>>> the messages looked like errors to my addled eyes). This system >>>> is on a >>>> different network, so getting the file transferred would take me >>>> about >>>> 24 hours. Is there something I can get that'll tell you what you >>>> need >>>> but is plaintext? >>>> >>>> >>>> Ok, that's fine. >>>> >>>> Try this. Set ra_plugin to dogtag in /etc/ipa/default.conf. This >>>> will let it get past the error and it should install a CA. I'm >>>> trying to think worst case scenario what it might do and I'm not >>>> coming up with anything. I think the worst that happens is that >>>> adding a CA fails later. >>>> >>>> rob >>>> >>>> >>>> I tried this and hope this subset of information is helpful: >>>> >>>> # openssl pkcs12 -in cacert.p12 -out cacert.pem.bdw -cacerts >>>> -nokeys >>>> # cat cacert.pem.bdw >>>> Bag Attributes: >>>> subject=/O=FOO.NET/CN=__**Certificate >>>> >>>> > >>>> Authority/ >>>> issuer=/O=FOO.NET/CN=__**Certificate >>>> >>>> > >>>> Authority >>>> >>>> >>>> -----BEGIN CERTIFICATE----- >>>> MIIDgzCCA... >>>> ...Iwk4r >>>> -----END CERTIFICATE----- >>>> # openssl pkcs12 -in cacert.p12 -out cert.pem.bdw -clcerts >>>> -nokeys >>>> # cat cert.pem.bdw >>>> Bag Attributes: >>>> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE >>>> 6C A7 2E 53 88 >>>> friendlyName: caSigningCert cert-pki-ca >>>> subject=/O=FOO.NET/CN=__**Certificate >>>> >>>> > >>>> Authority >>>> issuer=/O=FOO.NET/CN=__**Certificate >>>> >>>> > >>>> Authority >>>> >>>> >>>> -----BEGIN CERTIFICATE----- >>>> MIIDgzCCA... >>>> ...Iwk4r >>>> -----END CERTIFICATE----- >>>> Bag Attributes: >>>> localKeyID: 88 BF DF 56 30 BB A9 47 12 D4 5F 7B AE 39 DC >>>> BF CF F5 92 22 >>>> friendlyName: ocspSigningCert cert-pki-ca >>>> subject=/O=FOO.NET/CN=OCSP >>>> Subsystem >>>> issuer=/O=FOO.NET/CN=__**Certificate >>>> >>>> > >>>> Authority >>>> >>>> >>>> -----BEGIN CERTIFICATE----- >>>> MIIDYTCCA... >>>> ...wlr4Q= >>>> -----END CERTIFICATE----- >>>> Bag Attributes: >>>> localKeyID: B5 3B 27 CC 57 72 45 E2 8D 46 C9 5E E1 C0 50 >>>> DF 2D 11 62 0E >>>> friendlyName: subsystemCert cert-pki-ca >>>> subject=/O=FOO.NET/CN=CA >>>> Subsystem >>>> issuer=/O=FOO.NET/CN=__**Certificate >>>> >>>> > >>>> Authority >>>> >>>> >>>> -----BEGIN CERTIFICATE----- >>>> MIIDaTCCA... >>>> ...BxqqA== >>>> -----END CERTIFICATE----- >>>> Bag Attributes: >>>> localKeyID: 1F 69 62 C7 88 D8 95 2A B1 7D 61 F9 10 87 14 >>>> D0 76 Ba B9 44 >>>> friendlyName: auditSigningCert cert-pki-ca >>>> subject=/O=FOO.NET/CN=CA >>>> Audit >>>> issuer=/O=FOO.NET/CN=__**CertificateAUthority >>>> >>>> > >>>> >>>> >>>> >>>> >> >>>> >>>> -----BEGIN CERTIFICATE----- >>>> MIIDRDCCA... >>>> ...EAd+Ug7 >>>> -----END CERTIFICATE----- >>>> # openssl pkcs12 -in cacert.p12 -out key.pem.bdw -nocerts >>>> # cat key.pem.bdw >>>> Bag Attributes >>>> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE >>>> 6C A7 2E 53 88 >>>> friendlyName: CN=Certificate Authority,O=FOO.NET >>>> >>>> >>>> >>>> Key Attributes: >>>> -----BEGIN ENCRYPTED PRIVATE KEY----- >>>> MIIFDjBAB... >>>> ...XLtoD8= >>>> -----END ENCRYPTED PRIVATE KEY----- >>>> Bag Attributes >>>> localKeyID: >>>> friendlyName: CN=OCSP Subsystem,O=FOO.NET >>> > >>>> >>>> >>>> Key Attributes: >>>> -----BEGIN ENCRYPTED PRIVATE KEY----- >>>> : >>>> -----END ENCRYPTED PRIVATE KEY----- >>>> Bag Attributes >>>> localKeyID: >>>> friendlyName: CN=CA Subsystem,O=FOO.NET >>>> >>>> >>>> Key Attributes: >>>> Bag Attributes >>>> localKeyID: >>>> friendlyName: CN=CA Audit,O=FOO.NET >>>> >>>> >>>> Key Attributes: >>>> >>>> If you need to see anything else, please let me know. >>>> >>>> >>>> _ >>>> _ >>>> *Bret Wortman* >>>> >>>> >>>> http://damascusgrp.com/ >>>> http://about.me/wortmanbret >>>> >>>> >>>> On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden >>>> >>>> >> >>>> wrote: >>>> >>>> Bret Wortman wrote: >>>> >>>> On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden >>>> >>>> > >>>> >>> >>> >>**>____wrote: >>>> >>>> >>>> Bret Wortman wrote: >>>> >>>> On Thu, Aug 29, 2013 at 11:10 AM, Rob >>>> Crittenden >>>> >>> >>> > >>>> >>> >>> >> >>>> >>> >>>> >>> > >>> >>>> >>> >>**>__> wrote: >>>> >>>> Bret Wortman wrote: >>>> >>>> A bit of googling has led me to >>>> understand >>>> that we must >>>> have >>>> created the >>>> original server with --selfsign, and >>>> that >>>> locked us into >>>> something bad >>>> which is now causing us problems. I'm >>>> not sure >>>> how this >>>> happened, since >>>> we actually created our original >>>> instance on a >>>> different server, >>>> created >>>> ipamaster as a replica of that one, >>>> then ran >>>> ipa-ca-install on >>>> ipamaster >>>> to make it the new CA. How did it end >>>> up in >>>> this state? >>>> >>>> Anyway, is there ANY way around this? >>>> Can I simply >>>> ignore this, >>>> break >>>> the replication agreement as Simo >>>> suggested, >>>> rebuild >>>> ipamaster, >>>> replicate ipamaster2 to the new >>>> ipamaster, and >>>> then >>>> somehow make >>>> ipamaster be a CA using Dogtag? Will >>>> that >>>> screw up all >>>> the clients? >>>> >>>> >>>> I think we should pause and take a look >>>> at your >>>> installation. >>>> >>>> I'd check all your current masters, >>>> whether they >>>> are currently >>>> working or not. Look at the value of >>>> ra_plugin in >>>> /etc/ipa/default.conf. That controls >>>> what IPA >>>> thinks the CA is. >>>> >>>> on ipamaster: ra_plugin=dogtag >>>> >>>> and either that same value or the ra_plugin >>>> doesn't >>>> exist on the >>>> replicas. On ipamaster2, the one I just >>>> installed, >>>> there is no >>>> ra_plugin >>>> in the file. >>>> >>>> Then check to see if you have dogtag >>>> running on >>>> any of these >>>> systems. This will include a 2nd 389-ds >>>> instance, >>>> /etc/dirsrv/slapd-PKI-IPA and, depending >>>> on your >>>> distro, a PKI >>>> service like >>>> pki-tomcatd at pki-tomcat._______**_service. >>>> >>>> >>>> You can >>>> >>>> optionally >>>> >>>> see if /etc/pki/pki-tomcat exists. >>>> >>>> ipamaster definitely has a >>>> /etc/dirsrv/slapd-PKI-IPA >>>> directory, with >>>> files updated fairly recently (within the past >>>> 30 minutes - >>>> lse.ldif and >>>> lse.ldif.bak, others updated yesterday). I >>>> also have a >>>> pki-tomcatd at .service file and a >>>> pki-tomcatd.target. no >>>> /etc/pki/pki-tomcat. >>>> >>>> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET. >>>> It does have >>>> pki-tomcatd.target and pki-tomcatd at .service. >>>> No >>>> /etc/pki/pki-tomcat. >>>> >>>> >>>> Ok. When you created the replica file for >>>> ipamaster2, did >>>> you create >>>> it on ipamaster? Only a replica that is a CA can >>>> create a >>>> replica >>>> with a CA. >>>> >>>> Yes. So I'm not sure what went askew. >>>> >>>> >>>> Ok. I think we need to see what's in the prepared file. It >>>> is just a >>>> gpg-encrypted tarball. Can you do something like: >>>> >>>> gpg -d replica-info-pacer.greyoak.___**_com.gpg |tar xf - >>>> >>>> >>>> >>>> This will create a realm_info subdirectory. The file >>>> cacert.p12 >>>> should be in there. >>>> >>>> rob >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> ______________________________**_________________ >>>> Freeipa-users mailing list >>>> Freeipa-users at redhat.com >>>> https://www.redhat.com/**mailman/listinfo/freeipa-users >>>> >>>> >>> >>> -- >>> Petr? >>> >>> >>> ______________________________**_________________ >>> Freeipa-users mailing list >>> Freeipa-users at redhat.com >>> https://www.redhat.com/**mailman/listinfo/freeipa-users >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Aug 30 12:47:56 2013 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 30 Aug 2013 08:47:56 -0400 Subject: [Freeipa-users] Fwd: Fwd: Fwd: Scorched earth In-Reply-To: References: <521E9BF8.6050302@redhat.com> <521F6473.9040605@redhat.com> <521F6B71.1080906@redhat.com> <521F6F98.3070909@redhat.com> <521F99FE.6060205@redhat.com> <52205FF5.4040506@redhat.com> Message-ID: <5220947C.1020101@redhat.com> Bret Wortman wrote: > Still odder ... I went ahead and tried to delete the agreement: > > [ipamaster]# ipa-replica-manage del ipamaster3.foo.net > --force > 'ipamaster.foo.net ' has no replication > agreement for 'ipamaster3.foo.net ' > [ipamaster]# > > Dug back into the script and realized upon further reading (and widening > my read to more of the code) that found was being set True elsewhere -- > where it was complaining about how ipamaster knew about ipamaster3 > already. Fair enough. So I hopped on over there and removed it. Which > worked. And now the script proceeds much better. > > Guess the third cup of coffee helped. > > CA configuration still failed, though, at the same place as before > (though executed as part of ipa-replica-install --setup-ca this time): > > [2/17]: configuring certificate server instance > ipa : CRITICAL failed to configure ca instance Command > '/usr/sbin/pkispawn -s CA -f /tmp/tmpnq_J4d' returned non-zero exit status 1 > > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > Configuration of CA failed. > > /This/ time, I'm not going to run the --uninstall command until someone > on the team tells me to do so.... Ok. What we'll need to see is the full /var/log/ipareplica-install.log and the CA debug log from /var/log/pki/pki-tomcat/ca/debug. The CA team sometimes wants the debug log from the master you're cloning from too. You can send these to me out of band if you'd like, the debug logs in particular tend to be humongous. rob From michal.dwuznik at gmail.com Fri Aug 30 13:54:54 2013 From: michal.dwuznik at gmail.com (=?UTF-8?B?TWljaGHFgiBEd3XFvG5paw==?=) Date: Fri, 30 Aug 2013 15:54:54 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: <20130830083634.GG5929@hendrix.brq.redhat.com> References: <521F8B4F.7000204@redhat.com> <521FFDBB.1070205@redhat.com> <20130830083634.GG5929@hendrix.brq.redhat.com> Message-ID: Ok, I somehow assumed certs are very much needed for ldaps... In the meantime, I set up a debian wheezy machine to try the freeipa-client from debs. I managed to get working ipa-client (with a few quirks...- default nss database needed to be created) with packages from deb http://apt.numeezy.fr wheezy main deb-src http://apt.numeezy.fr wheezy main. So now I have a ready set of debian-like configs for wheezy, making it work with squeeze seems easier now (it comes with learning, too...) I must admit ipa-client debug option is lovely as a step-by-step guide for trying by hand :> Going back to thinking whether to try getting ipa on squeeze or getting the legacy software working with squeeze... (some of the scientists seem to be the happiest if the system is totally unchanged for some 20 years...). Regards Michal PS:I do see hope for rooting out the last instance of NIS on the campus :> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhrozek at redhat.com Fri Aug 30 14:48:17 2013 From: jhrozek at redhat.com (Jakub Hrozek) Date: Fri, 30 Aug 2013 16:48:17 +0200 Subject: [Freeipa-users] setting up a client on Debian squeeze In-Reply-To: References: <521F8B4F.7000204@redhat.com> <521FFDBB.1070205@redhat.com> <20130830083634.GG5929@hendrix.brq.redhat.com> Message-ID: <20130830144817.GL5929@hendrix.brq.redhat.com> On Fri, Aug 30, 2013 at 03:54:54PM +0200, Micha? Dwu?nik wrote: > Ok, I somehow assumed certs are very much needed for ldaps... > Well, for most operations the SSSD uses GSSAPI authentication. Only when passwords are migrated, we do an LDAP bind with StartTLS. > In the meantime, I set up a debian wheezy machine to try the freeipa-client > from debs. > > I managed to get working ipa-client (with a few quirks...- default nss > database needed to be created) with packages from > deb http://apt.numeezy.fr wheezy main > deb-src http://apt.numeezy.fr wheezy main. > So now I have a ready set of debian-like configs for wheezy, making it work > with squeeze seems easier now (it comes with learning, too...) > > I must admit ipa-client debug option is lovely as a step-by-step guide for > trying by hand :> > > Going back to thinking whether to try getting ipa on squeeze or getting the > legacy software working with squeeze... > (some of the scientists seem to be the happiest if the system is totally > unchanged for some 20 years...). > > > Regards > Michal > > PS:I do see hope for rooting out the last instance of NIS on the campus :> Terminate it with extreme prejudice :-) From john.moyer at digitalreasoning.com Fri Aug 30 19:31:22 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 15:31:22 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <521E1A03.1060904@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> Message-ID: Rob or anyone else, So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. So just to double check what I'm thinking. I need to create three indexes 1. objectclass pres 2. objecclass eq 3. uid pres Please let me know if I'm reading this correctly or if I'm way off? 7337 (objectclass=inetorgperson) 4597 (objectclass=*) 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) 307 (objectclass=krbticketpolicyaux) 292 (uid=*) Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: > John Moyer wrote: >> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? > > I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes > > Note that you'll want to create the same index on all hosts. This configuration is not replicated. > > You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update > > rob > >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> Digital Reasoning Systems, Inc. >> John.Moyer at digitalreasoning.com >> Office: 703.678.2311 >> Mobile: 240.460.0023 >> Fax: 703.678.2312 >> www.digitalreasoning.com >> >> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >> >>> John Moyer wrote: >>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>> >>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>> >>> rob >>> >>>> >>>> >>>> >>>> Start of Log: 27/Aug/2013:02:36:08 >>>> End of Log: 27/Aug/2013:12:17:15 >>>> >>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>> >>>> Restarts: 2 >>>> Total Connections: 45224 >>>> SSL Connections: 44735 >>>> Peak Concurrent Connections: 76 >>>> Total Operations: 132568 >>>> Total Results: 132737 >>>> Overall Performance: 100.0% >>>> >>>> Searches: 61318 (1.76/sec) (105.52/min) >>>> Modifications: 277 (0.01/sec) (0.48/min) >>>> Adds: 10 (0.00/sec) (0.02/min) >>>> Deletes: 12 (0.00/sec) (0.02/min) >>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>> Compares: 0 (0.00/sec) (0.00/min) >>>> Binds: 62143 (1.78/sec) (106.94/min) >>>> >>>> Proxied Auth Operations: 0 >>>> Persistent Searches: 3 >>>> Internal Operations: 0 >>>> Entry Operations: 0 >>>> Extended Operations: 8808 >>>> Abandoned Requests: 0 >>>> Smart Referrals Received: 0 >>>> >>>> VLV Operations: 0 >>>> VLV Unindexed Searches: 0 >>>> SORT Operations: 353 >>>> >>>> Entire Search Base Queries: 106 >>>> Unindexed Searches: 319 >>>> >>>> FDs Taken: 45262 >>>> FDs Returned: 45210 >>>> Highest FD Taken: 139 >>>> >>>> Broken Pipes: 0 >>>> Connections Reset By Peer: 0 >>>> Resource Unavailable: 0 >>>> >>>> Binds: 62143 >>>> Unbinds: 44539 >>>> >>>> LDAP v2 Binds: 2 >>>> LDAP v3 Binds: 62141 >>>> SSL Client Binds: 0 >>>> Failed SSL Client Binds: 0 >>>> SASL Binds: 1466 >>>> 1458 GSSAPI >>>> 8 EXTERNAL >>>> >>>> Directory Manager Binds: 10 >>>> Anonymous Binds: 1476 >>>> Other Binds: 60657 >>>> >>>> >>>> >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>> >>>>> John Moyer wrote: >>>>>> Is there any way to see what fields are index'ed? >>>>> >>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>> >>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>> >>>>> rob >>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> Digital Reasoning Systems, Inc. >>>>>> John.Moyer at digitalreasoning.com >>>>>> Office: 703.678.2311 >>>>>> Mobile: 240.460.0023 >>>>>> Fax: 703.678.2312 >>>>>> www.digitalreasoning.com >>>>>> >>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>> >>>>>>> That looks like the output I just got shown below: >>>>>>> >>>>>>> >>>>>>> dn: cn=mapping tree,cn=config >>>>>>> >>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>> >>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>> >>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> >>>>>>> >>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>> >>>>>>>> John Moyer wrote: >>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>> >>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>> everything down wait times were around 25%+ >>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>> adding to the mess >>>>>>>>> >>>>>>>>> >>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm running the following: >>>>>>>>> >>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>> >>>>>>>>> >>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>> user instead of IPA). >>>>>>>>> >>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>> well. >>>>>>>>> >>>>>>>>> Any help would be appreciated. >>>>>>>>> >>>>>>>> >>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>> >>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>> >>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>> >>>>>>>> They should look like: >>>>>>>> >>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>> >>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rmeggins at redhat.com Fri Aug 30 19:41:59 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Fri, 30 Aug 2013 13:41:59 -0600 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> Message-ID: <5220F587.6000808@redhat.com> On 08/30/2013 01:31 PM, John Moyer wrote: > Rob or anyone else, > > So while struggling along on this server I just grabbed the logs off > it and ran that log program with the options you suggested. There > are a lot of unindexed requests. These are the top issues I've > removed the one username that showed up. > > So just to double check what I'm thinking. I need to create three > indexes > 1. objectclass pres No, do not create this one > 2. objectclass eq This should already be indexed > 3. uid pres I suppose the UI might be doing this search? > > Please let me know if I'm reading this correctly or if I'm way off? > > > 7337 (objectclass=inetorgperson) > 4597 (objectclass=*) > 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) > 307 (objectclass=krbticketpolicyaux) > 292 (uid=*) > > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > *Digital Reasoning Systems, Inc.* > John.Moyer at digitalreasoning.com > Office:703.678.2311 > Mobile:240.460.0023 > Fax:703.678.2312 > www.digitalreasoning.com > > On Aug 28, 2013, at 11:40 AM, Rob Crittenden > wrote: > >> John Moyer wrote: >>> So this method of search logs is great, and it shows some indexes >>> that would likely highly increase efficiency with my usage. So, >>> are there instructions how to do that? or do you know off hand how >>> to do that? >> >> I'd start with >> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >> >> Note that you'll want to create the same index on all hosts. This >> configuration is not replicated. >> >> You can see the ones we create in /usr/share/ipa/indices.ldif and >> /usr/share/ipa/updates/20-indices.update >> >> rob >> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> Digital Reasoning Systems, Inc. >>> John.Moyer at digitalreasoning.com >>> Office:703.678.2311 >>> Mobile:240.460.0023 >>> Fax:703.678.2312 >>> www.digitalreasoning.com >>> >>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>> >>>> John Moyer wrote: >>>>> Wow, this is quite insightful, this is the output from that, it >>>>> looks like there aren't many unindexed searches (319 doesn't seem >>>>> like a lot to me at least). Do you have any suggestions from this >>>>> output? >>>> >>>> There are a slew of options you can provide to logconv.pl. I >>>> typically use logconv.pl -ula >>>> /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>> >>>> rob >>>> >>>>> >>>>> >>>>> >>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>> End of Log: 27/Aug/2013:12:17:15 >>>>> >>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>> >>>>> Restarts: 2 >>>>> Total Connections: 45224 >>>>> SSL Connections: 44735 >>>>> Peak Concurrent Connections: 76 >>>>> Total Operations: 132568 >>>>> Total Results: 132737 >>>>> Overall Performance: 100.0% >>>>> >>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>> >>>>> Proxied Auth Operations: 0 >>>>> Persistent Searches: 3 >>>>> Internal Operations: 0 >>>>> Entry Operations: 0 >>>>> Extended Operations: 8808 >>>>> Abandoned Requests: 0 >>>>> Smart Referrals Received: 0 >>>>> >>>>> VLV Operations: 0 >>>>> VLV Unindexed Searches: 0 >>>>> SORT Operations: 353 >>>>> >>>>> Entire Search Base Queries: 106 >>>>> Unindexed Searches: 319 >>>>> >>>>> FDs Taken: 45262 >>>>> FDs Returned: 45210 >>>>> Highest FD Taken: 139 >>>>> >>>>> Broken Pipes: 0 >>>>> Connections Reset By Peer: 0 >>>>> Resource Unavailable: 0 >>>>> >>>>> Binds: 62143 >>>>> Unbinds: 44539 >>>>> >>>>> LDAP v2 Binds: 2 >>>>> LDAP v3 Binds: 62141 >>>>> SSL Client Binds: 0 >>>>> Failed SSL Client Binds: 0 >>>>> SASL Binds: 1466 >>>>> 1458 GSSAPI >>>>> 8 EXTERNAL >>>>> >>>>> Directory Manager Binds: 10 >>>>> Anonymous Binds: 1476 >>>>> Other Binds: 60657 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden >>>>> wrote: >>>>> >>>>>> John Moyer wrote: >>>>>>> Is there any way to see what fields are index'ed? >>>>>> >>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b >>>>>> 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>> >>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>> >>>>>> rob >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> Digital Reasoning Systems, Inc. >>>>>>> John.Moyer at digitalreasoning.com >>>>>>> Office:703.678.2311 >>>>>>> Mobile:240.460.0023 >>>>>>> Fax:703.678.2312 >>>>>>> www.digitalreasoning.com >>>>>>> >>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer >>>>>>> wrote: >>>>>>> >>>>>>>> That looks like the output I just got shown below: >>>>>>>> >>>>>>>> >>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>> >>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>> >>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>> >>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>> memberof idnssoaserial >>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>> krbloginfailedcount >>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE >>>>>>>> entryusn krblasts >>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> _____________________________________________________ >>>>>>>> John Moyer >>>>>>>> Director, IT Operations >>>>>>>> >>>>>>>> >>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden >>>>>>>> wrote: >>>>>>>> >>>>>>>>> John Moyer wrote: >>>>>>>>>> Ok, so we tried to implement this again, and as soon as we >>>>>>>>>> put on a >>>>>>>>>> server that authenticates heavily the IPA came to it's knees >>>>>>>>>> again. >>>>>>>>>> This time I was able to watch it closely and try to >>>>>>>>>> troubleshoot a lot >>>>>>>>>> more, and also know exactly what server caused it (Mercurial >>>>>>>>>> with help >>>>>>>>>> of bamboo). This runs fine on a normal old openldap >>>>>>>>>> servers. The >>>>>>>>>> user is logging in very quickly and each time it logs in I >>>>>>>>>> can see in >>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or >>>>>>>>>> whatever it is >>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling >>>>>>>>>> VERY >>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>> >>>>>>>>>> Issue 1: This is causing severe disk latency which obviously >>>>>>>>>> slows >>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>> Issue 2: These changes need to be replicated to my slave >>>>>>>>>> server thus >>>>>>>>>> adding to the mess >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> My question is, why does the IPA server fail to keep up with >>>>>>>>>> the load >>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I'm running the following: >>>>>>>>>> >>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> So I've implemented this server anyway (against my better >>>>>>>>>> judgement with >>>>>>>>>> these issues and just made the user that logs into mercurial >>>>>>>>>> a local >>>>>>>>>> user instead of IPA). >>>>>>>>>> >>>>>>>>>> Also note before I did that for fun I implemented a RAM disk >>>>>>>>>> to put the >>>>>>>>>> change logs on, and that dropped the wait time to 0 (except >>>>>>>>>> bursts where >>>>>>>>>> it would raise to 30 to write the access log) but the CPU >>>>>>>>>> drove to 100% >>>>>>>>>> trying to keep up with the load. I have also killed the >>>>>>>>>> replication as >>>>>>>>>> well. >>>>>>>>>> >>>>>>>>>> Any help would be appreciated. >>>>>>>>>> >>>>>>>>> >>>>>>>>> krblastsuccessfulauth should be excluded from replication, >>>>>>>>> though I guess that doesn't prevent it from ending up in the >>>>>>>>> changelog. >>>>>>>>> >>>>>>>>> You can confirm that they are excluded by searching the >>>>>>>>> agreements: >>>>>>>>> >>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D >>>>>>>>> 'cn=directory manager' -W nsDS5ReplicatedAttributeList >>>>>>>>> nsDS5ReplicatedAttributeListTotal >>>>>>>>> >>>>>>>>> They should look like: >>>>>>>>> >>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>>> memberof idnssoaserial entryusn krblastsuccessfulauth >>>>>>>>> krblastfailedauth krbloginfailedcount >>>>>>>>> >>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE >>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>>> krbloginfailedcount >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.moyer at digitalreasoning.com Fri Aug 30 19:49:24 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 15:49:24 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> Message-ID: <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> I'm sorry that was my top unique filter list not my unindexed list. Please disregard my last email. Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 30, 2013, at 3:47 PM, John Moyer wrote: > If objectclass eq is already indexed how are these on my top unindexed list? Wouldn't objectclass eq cover this (objectclass=inetorgperson)? and the third and fourth entry? I apologize if I'm way off as I am new to the intricacies of LDAP indexing. > > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > > On Aug 30, 2013, at 3:41 PM, Rich Megginson wrote: > >> On 08/30/2013 01:31 PM, John Moyer wrote: >>> Rob or anyone else, >>> >>> So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. >>> >>> So just to double check what I'm thinking. I need to create three indexes >>> 1. objectclass pres >> No, do not create this one >>> 2. objectclass eq >> This should already be indexed >>> 3. uid pres >> I suppose the UI might be doing this search? >>> >>> Please let me know if I'm reading this correctly or if I'm way off? >>> >>> >>> 7337 (objectclass=inetorgperson) >>> 4597 (objectclass=*) >>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>> 307 (objectclass=krbticketpolicyaux) >>> 292 (uid=*) >>> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> Digital Reasoning Systems, Inc. >>> John.Moyer at digitalreasoning.com >>> Office: 703.678.2311 >>> Mobile: 240.460.0023 >>> Fax: 703.678.2312 >>> www.digitalreasoning.com >>> >>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: >>> >>>> John Moyer wrote: >>>>> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? >>>> >>>> I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>> >>>> Note that you'll want to create the same index on all hosts. This configuration is not replicated. >>>> >>>> You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update >>>> >>>> rob >>>> >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> Digital Reasoning Systems, Inc. >>>>> John.Moyer at digitalreasoning.com >>>>> Office: 703.678.2311 >>>>> Mobile: 240.460.0023 >>>>> Fax: >>>>> 703.678.2312 >>>>> www.digitalreasoning.com >>>>> >>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>>>> >>>>>> John Moyer wrote: >>>>>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>>>>> >>>>>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>>>> >>>>>> rob >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>> >>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>> >>>>>>> Restarts: 2 >>>>>>> Total Connections: 45224 >>>>>>> SSL Connections: 44735 >>>>>>> Peak Concurrent Connections: 76 >>>>>>> Total Operations: 132568 >>>>>>> Total Results: 132737 >>>>>>> Overall Performance: 100.0% >>>>>>> >>>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>>> >>>>>>> Proxied Auth Operations: 0 >>>>>>> Persistent Searches: 3 >>>>>>> Internal Operations: 0 >>>>>>> Entry Operations: 0 >>>>>>> Extended Operations: 8808 >>>>>>> Abandoned Requests: 0 >>>>>>> Smart Referrals Received: 0 >>>>>>> >>>>>>> VLV Operations: 0 >>>>>>> VLV Unindexed Searches: 0 >>>>>>> SORT Operations: 353 >>>>>>> >>>>>>> Entire Search Base Queries: 106 >>>>>>> Unindexed Searches: 319 >>>>>>> >>>>>>> FDs Taken: 45262 >>>>>>> FDs Returned: 45210 >>>>>>> Highest FD Taken: 139 >>>>>>> >>>>>>> Broken Pipes: 0 >>>>>>> Connections Reset By Peer: 0 >>>>>>> Resource Unavailable: 0 >>>>>>> >>>>>>> Binds: 62143 >>>>>>> Unbinds: 44539 >>>>>>> >>>>>>> LDAP v2 Binds: 2 >>>>>>> LDAP v3 Binds: 62141 >>>>>>> SSL Client Binds: 0 >>>>>>> Failed SSL Client Binds: 0 >>>>>>> SASL Binds: 1466 >>>>>>> 1458 GSSAPI >>>>>>> 8 EXTERNAL >>>>>>> >>>>>>> Directory Manager Binds: 10 >>>>>>> Anonymous Binds: 1476 >>>>>>> Other Binds: 60657 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>>>>> >>>>>>>> John Moyer wrote: >>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>> >>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>> >>>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> _____________________________________________________ >>>>>>>>> John Moyer >>>>>>>>> Director, IT Operations >>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>> Office: 703.678.2311 >>>>>>>>> Mobile: 240.460.0023 >>>>>>>>> Fax: >>>>>>>>> 703.678.2312 >>>>>>>>> www.digitalreasoning.com >>>>>>>>> >>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>>>>> >>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>> >>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>> >>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>> >>>>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> _____________________________________________________ >>>>>>>>>> John Moyer >>>>>>>>>> Director, IT Operations >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>>>>> >>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>> >>>>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>>>>> adding to the mess >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'm running the following: >>>>>>>>>>>> >>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>> >>>>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>>>>> well. >>>>>>>>>>>> >>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>>>>> >>>>>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>>>>> >>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>>>>> >>>>>>>>>>> They should look like: >>>>>>>>>>> >>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>> >>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From john.moyer at digitalreasoning.com Fri Aug 30 19:47:16 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 15:47:16 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <5220F587.6000808@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> Message-ID: If objectclass eq is already indexed how are these on my top unindexed list? Wouldn't objectclass eq cover this (objectclass=inetorgperson)? and the third and fourth entry? I apologize if I'm way off as I am new to the intricacies of LDAP indexing. Thanks, _____________________________________________________ John Moyer Director, IT Operations On Aug 30, 2013, at 3:41 PM, Rich Megginson wrote: > On 08/30/2013 01:31 PM, John Moyer wrote: >> Rob or anyone else, >> >> So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. >> >> So just to double check what I'm thinking. I need to create three indexes >> 1. objectclass pres > No, do not create this one >> 2. objectclass eq > This should already be indexed >> 3. uid pres > I suppose the UI might be doing this search? >> >> Please let me know if I'm reading this correctly or if I'm way off? >> >> >> 7337 (objectclass=inetorgperson) >> 4597 (objectclass=*) >> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >> 307 (objectclass=krbticketpolicyaux) >> 292 (uid=*) >> >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> Digital Reasoning Systems, Inc. >> John.Moyer at digitalreasoning.com >> Office: 703.678.2311 >> Mobile: 240.460.0023 >> Fax: 703.678.2312 >> www.digitalreasoning.com >> >> On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: >> >>> John Moyer wrote: >>>> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? >>> >>> I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>> >>> Note that you'll want to create the same index on all hosts. This configuration is not replicated. >>> >>> You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update >>> >>> rob >>> >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> Digital Reasoning Systems, Inc. >>>> John.Moyer at digitalreasoning.com >>>> Office: 703.678.2311 >>>> Mobile: 240.460.0023 >>>> Fax: >>>> 703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>>> >>>>> John Moyer wrote: >>>>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>>>> >>>>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>>> >>>>> rob >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>> >>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>> >>>>>> Restarts: 2 >>>>>> Total Connections: 45224 >>>>>> SSL Connections: 44735 >>>>>> Peak Concurrent Connections: 76 >>>>>> Total Operations: 132568 >>>>>> Total Results: 132737 >>>>>> Overall Performance: 100.0% >>>>>> >>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>> >>>>>> Proxied Auth Operations: 0 >>>>>> Persistent Searches: 3 >>>>>> Internal Operations: 0 >>>>>> Entry Operations: 0 >>>>>> Extended Operations: 8808 >>>>>> Abandoned Requests: 0 >>>>>> Smart Referrals Received: 0 >>>>>> >>>>>> VLV Operations: 0 >>>>>> VLV Unindexed Searches: 0 >>>>>> SORT Operations: 353 >>>>>> >>>>>> Entire Search Base Queries: 106 >>>>>> Unindexed Searches: 319 >>>>>> >>>>>> FDs Taken: 45262 >>>>>> FDs Returned: 45210 >>>>>> Highest FD Taken: 139 >>>>>> >>>>>> Broken Pipes: 0 >>>>>> Connections Reset By Peer: 0 >>>>>> Resource Unavailable: 0 >>>>>> >>>>>> Binds: 62143 >>>>>> Unbinds: 44539 >>>>>> >>>>>> LDAP v2 Binds: 2 >>>>>> LDAP v3 Binds: 62141 >>>>>> SSL Client Binds: 0 >>>>>> Failed SSL Client Binds: 0 >>>>>> SASL Binds: 1466 >>>>>> 1458 GSSAPI >>>>>> 8 EXTERNAL >>>>>> >>>>>> Directory Manager Binds: 10 >>>>>> Anonymous Binds: 1476 >>>>>> Other Binds: 60657 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>>>> >>>>>>> John Moyer wrote: >>>>>>>> Is there any way to see what fields are index'ed? >>>>>>> >>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>> >>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>> >>>>>>> rob >>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> _____________________________________________________ >>>>>>>> John Moyer >>>>>>>> Director, IT Operations >>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>> Office: 703.678.2311 >>>>>>>> Mobile: 240.460.0023 >>>>>>>> Fax: >>>>>>>> 703.678.2312 >>>>>>>> www.digitalreasoning.com >>>>>>>> >>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>>>> >>>>>>>>> That looks like the output I just got shown below: >>>>>>>>> >>>>>>>>> >>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>> >>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>> >>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>> >>>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> _____________________________________________________ >>>>>>>>> John Moyer >>>>>>>>> Director, IT Operations >>>>>>>>> >>>>>>>>> >>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>>>> >>>>>>>>>> John Moyer wrote: >>>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>> >>>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>>>> adding to the mess >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I'm running the following: >>>>>>>>>>> >>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>>>> user instead of IPA). >>>>>>>>>>> >>>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>>>> well. >>>>>>>>>>> >>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>>>> >>>>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>>>> >>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>>>> >>>>>>>>>> They should look like: >>>>>>>>>> >>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>> >>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From john.moyer at digitalreasoning.com Fri Aug 30 20:10:47 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 16:10:47 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> Message-ID: So I'm trying to focus on just one use case having the performance issues. I figured out my JIRA server does a sync against this which brings it to it's knees. It takes it 900 seconds to complete the sync and seems to go through every group and user on the machine. I have the base for the user group set to cn=users,cn=accounts,dc=example,dc=com and the groups to cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 seconds off my openldap server I had before this. Any suggestions what to look at? Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 30, 2013, at 3:49 PM, John Moyer wrote: > I'm sorry that was my top unique filter list not my unindexed list. Please disregard my last email. > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > Digital Reasoning Systems, Inc. > John.Moyer at digitalreasoning.com > Office: 703.678.2311 > Mobile: 240.460.0023 > Fax: 703.678.2312 > www.digitalreasoning.com > > On Aug 30, 2013, at 3:47 PM, John Moyer wrote: > >> If objectclass eq is already indexed how are these on my top unindexed list? Wouldn't objectclass eq cover this (objectclass=inetorgperson)? and the third and fourth entry? I apologize if I'm way off as I am new to the intricacies of LDAP indexing. >> >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> >> On Aug 30, 2013, at 3:41 PM, Rich Megginson wrote: >> >>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>> Rob or anyone else, >>>> >>>> So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. >>>> >>>> So just to double check what I'm thinking. I need to create three indexes >>>> 1. objectclass pres >>> No, do not create this one >>>> 2. objectclass eq >>> This should already be indexed >>>> 3. uid pres >>> I suppose the UI might be doing this search? >>>> >>>> Please let me know if I'm reading this correctly or if I'm way off? >>>> >>>> >>>> 7337 (objectclass=inetorgperson) >>>> 4597 (objectclass=*) >>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>> 307 (objectclass=krbticketpolicyaux) >>>> 292 (uid=*) >>>> >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> Digital Reasoning Systems, Inc. >>>> John.Moyer at digitalreasoning.com >>>> Office: 703.678.2311 >>>> Mobile: 240.460.0023 >>>> Fax: 703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: >>>> >>>>> John Moyer wrote: >>>>>> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? >>>>> >>>>> I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>> >>>>> Note that you'll want to create the same index on all hosts. This configuration is not replicated. >>>>> >>>>> You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update >>>>> >>>>> rob >>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> Digital Reasoning Systems, Inc. >>>>>> John.Moyer at digitalreasoning.com >>>>>> Office: 703.678.2311 >>>>>> Mobile: 240.460.0023 >>>>>> Fax: >>>>>> 703.678.2312 >>>>>> www.digitalreasoning.com >>>>>> >>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>>>>> >>>>>>> John Moyer wrote: >>>>>>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>>>>>> >>>>>>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>>>>> >>>>>>> rob >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>> >>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>> >>>>>>>> Restarts: 2 >>>>>>>> Total Connections: 45224 >>>>>>>> SSL Connections: 44735 >>>>>>>> Peak Concurrent Connections: 76 >>>>>>>> Total Operations: 132568 >>>>>>>> Total Results: 132737 >>>>>>>> Overall Performance: 100.0% >>>>>>>> >>>>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>>>> >>>>>>>> Proxied Auth Operations: 0 >>>>>>>> Persistent Searches: 3 >>>>>>>> Internal Operations: 0 >>>>>>>> Entry Operations: 0 >>>>>>>> Extended Operations: 8808 >>>>>>>> Abandoned Requests: 0 >>>>>>>> Smart Referrals Received: 0 >>>>>>>> >>>>>>>> VLV Operations: 0 >>>>>>>> VLV Unindexed Searches: 0 >>>>>>>> SORT Operations: 353 >>>>>>>> >>>>>>>> Entire Search Base Queries: 106 >>>>>>>> Unindexed Searches: 319 >>>>>>>> >>>>>>>> FDs Taken: 45262 >>>>>>>> FDs Returned: 45210 >>>>>>>> Highest FD Taken: 139 >>>>>>>> >>>>>>>> Broken Pipes: 0 >>>>>>>> Connections Reset By Peer: 0 >>>>>>>> Resource Unavailable: 0 >>>>>>>> >>>>>>>> Binds: 62143 >>>>>>>> Unbinds: 44539 >>>>>>>> >>>>>>>> LDAP v2 Binds: 2 >>>>>>>> LDAP v3 Binds: 62141 >>>>>>>> SSL Client Binds: 0 >>>>>>>> Failed SSL Client Binds: 0 >>>>>>>> SASL Binds: 1466 >>>>>>>> 1458 GSSAPI >>>>>>>> 8 EXTERNAL >>>>>>>> >>>>>>>> Directory Manager Binds: 10 >>>>>>>> Anonymous Binds: 1476 >>>>>>>> Other Binds: 60657 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> _____________________________________________________ >>>>>>>> John Moyer >>>>>>>> Director, IT Operations >>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>>>>>> >>>>>>>>> John Moyer wrote: >>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>> >>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>> >>>>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> _____________________________________________________ >>>>>>>>>> John Moyer >>>>>>>>>> Director, IT Operations >>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>> Office: 703.678.2311 >>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>> Fax: >>>>>>>>>> 703.678.2312 >>>>>>>>>> www.digitalreasoning.com >>>>>>>>>> >>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>>>>>> >>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>> >>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>> >>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>> >>>>>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> _____________________________________________________ >>>>>>>>>>> John Moyer >>>>>>>>>>> Director, IT Operations >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>>>>>> >>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>> >>>>>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>> >>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>> >>>>>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>>>>>> well. >>>>>>>>>>>>> >>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>>>>>> >>>>>>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>>>>>> >>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>> >>>>>>>>>>>> They should look like: >>>>>>>>>>>> >>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>> >>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rmeggins at redhat.com Fri Aug 30 20:25:38 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Fri, 30 Aug 2013 14:25:38 -0600 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> Message-ID: <5220FFC2.90207@redhat.com> On 08/30/2013 02:10 PM, John Moyer wrote: > So I'm trying to focus on just one use case having the performance > issues. I figured out my JIRA server does a sync against this which > brings it to it's knees. It takes it 900 seconds to complete the > sync and seems to go through every group and user on the machine. I > have the base for the user group set to > cn=users,cn=accounts,dc=example,dc=com and the groups to > cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 > seconds off my openldap server I had before this. Any suggestions > what to look at? The access log should have an operation which has etime=900 (or at least a very high etime) If you do logconv.pl -V it will print out a lot of information, including the highest etimes. You can then use this report to look at the access logs to find out exactly which operations had these high etimes. > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > *Digital Reasoning Systems, Inc.* > John.Moyer at digitalreasoning.com > Office:703.678.2311 > Mobile:240.460.0023 > Fax:703.678.2312 > www.digitalreasoning.com > > On Aug 30, 2013, at 3:49 PM, John Moyer > > wrote: > >> I'm sorry that was my top unique filter list not my unindexed list. >> Please disregard my last email. >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> *Digital Reasoning Systems, Inc.* >> John.Moyer at digitalreasoning.com >> Office:703.678.2311 >> Mobile:240.460.0023 >> Fax:703.678.2312 >> www.digitalreasoning.com >> >> On Aug 30, 2013, at 3:47 PM, John Moyer >> > > wrote: >> >>> If objectclass eq is already indexed how are these on my top >>> unindexed list? Wouldn't objectclass eq cover this >>> (objectclass=inetorgperson)? and the third and fourth entry? I >>> apologize if I'm way off as I am new to the intricacies of LDAP >>> indexing. >>> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> >>> On Aug 30, 2013, at 3:41 PM, Rich Megginson >> > wrote: >>> >>>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>>> Rob or anyone else, >>>>> >>>>> So while struggling along on this server I just grabbed the logs >>>>> off it and ran that log program with the options you suggested. >>>>> There are a lot of unindexed requests. These are the top issues >>>>> I've removed the one username that showed up. >>>>> >>>>> So just to double check what I'm thinking. I need to create >>>>> three indexes >>>>> 1. objectclass pres >>>> No, do not create this one >>>>> 2. objectclass eq >>>> This should already be indexed >>>>> 3. uid pres >>>> I suppose the UI might be doing this search? >>>>> >>>>> Please let me know if I'm reading this correctly or if I'm way off? >>>>> >>>>> >>>>> 7337 (objectclass=inetorgperson) >>>>> 4597 (objectclass=*) >>>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>>> 307 (objectclass=krbticketpolicyaux) >>>>> 292 (uid=*) >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> *Digital Reasoning Systems, Inc.* >>>>> John.Moyer at digitalreasoning.com >>>>> >>>>> Office:703.678.2311 >>>>> Mobile:240.460.0023 >>>>> Fax:703.678.2312 >>>>> www.digitalreasoning.com >>>>> >>>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden >>>> > wrote: >>>>> >>>>>> John Moyer wrote: >>>>>>> So this method of search logs is great, and it shows some >>>>>>> indexes that would likely highly increase efficiency with my >>>>>>> usage. So, are there instructions how to do that? or do you >>>>>>> know off hand how to do that? >>>>>> >>>>>> I'd start with >>>>>> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>>> >>>>>> Note that you'll want to create the same index on all hosts. This >>>>>> configuration is not replicated. >>>>>> >>>>>> You can see the ones we create in /usr/share/ipa/indices.ldif and >>>>>> /usr/share/ipa/updates/20-indices.update >>>>>> >>>>>> rob >>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> Digital Reasoning Systems, Inc. >>>>>>> John.Moyer at digitalreasoning.com >>>>>>> >>>>>>> Office:703.678.2311 >>>>>>> Mobile:240.460.0023 >>>>>>> Fax:703.678.2312 >>>>>>> www.digitalreasoning.com >>>>>>> >>>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden >>>>>>> wrote: >>>>>>> >>>>>>>> John Moyer wrote: >>>>>>>>> Wow, this is quite insightful, this is the output from that, >>>>>>>>> it looks like there aren't many unindexed searches (319 >>>>>>>>> doesn't seem like a lot to me at least). Do you have any >>>>>>>>> suggestions from this output? >>>>>>>> >>>>>>>> There are a slew of options you can provide to logconv.pl. I >>>>>>>> typically use logconv.pl -ula >>>>>>>> /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search >>>>>>>> analysis. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>>> >>>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>>> >>>>>>>>> Restarts: 2 >>>>>>>>> Total Connections: 45224 >>>>>>>>> SSL Connections: 44735 >>>>>>>>> Peak Concurrent Connections: 76 >>>>>>>>> Total Operations: 132568 >>>>>>>>> Total Results: 132737 >>>>>>>>> Overall Performance: 100.0% >>>>>>>>> >>>>>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>>>>> >>>>>>>>> Proxied Auth Operations: 0 >>>>>>>>> Persistent Searches: 3 >>>>>>>>> Internal Operations: 0 >>>>>>>>> Entry Operations: 0 >>>>>>>>> Extended Operations: 8808 >>>>>>>>> Abandoned Requests: 0 >>>>>>>>> Smart Referrals Received: 0 >>>>>>>>> >>>>>>>>> VLV Operations: 0 >>>>>>>>> VLV Unindexed Searches: 0 >>>>>>>>> SORT Operations: 353 >>>>>>>>> >>>>>>>>> Entire Search Base Queries: 106 >>>>>>>>> Unindexed Searches: 319 >>>>>>>>> >>>>>>>>> FDs Taken: 45262 >>>>>>>>> FDs Returned: 45210 >>>>>>>>> Highest FD Taken: 139 >>>>>>>>> >>>>>>>>> Broken Pipes: 0 >>>>>>>>> Connections Reset By Peer: 0 >>>>>>>>> Resource Unavailable: 0 >>>>>>>>> >>>>>>>>> Binds: 62143 >>>>>>>>> Unbinds: 44539 >>>>>>>>> >>>>>>>>> LDAP v2 Binds: 2 >>>>>>>>> LDAP v3 Binds: 62141 >>>>>>>>> SSL Client Binds: 0 >>>>>>>>> Failed SSL Client Binds: 0 >>>>>>>>> SASL Binds: 1466 >>>>>>>>> 1458 GSSAPI >>>>>>>>> 8 EXTERNAL >>>>>>>>> >>>>>>>>> Directory Manager Binds: 10 >>>>>>>>> Anonymous Binds: 1476 >>>>>>>>> Other Binds: 60657 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> _____________________________________________________ >>>>>>>>> John Moyer >>>>>>>>> Director, IT Operations >>>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> John Moyer wrote: >>>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>>> >>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b >>>>>>>>>> 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>>> >>>>>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> _____________________________________________________ >>>>>>>>>>> John Moyer >>>>>>>>>>> Director, IT Operations >>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>> Office:703.678.2311 >>>>>>>>>>> Mobile:240.460.0023 >>>>>>>>>>> Fax:703.678.2312 >>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>> >>>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>>> >>>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>> >>>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>> tree,cn=config >>>>>>>>>>>> >>>>>>>>>>>> dn: cn=meToipa2.example.com >>>>>>>>>>>> ,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>>>>>> memberof idnssoaserial >>>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>>>>>> krbloginfailedcount >>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ >>>>>>>>>>>> EXCLUDE entryusn krblasts >>>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>> John Moyer >>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as >>>>>>>>>>>>>> we put on a >>>>>>>>>>>>>> server that authenticates heavily the IPA came to it's >>>>>>>>>>>>>> knees again. >>>>>>>>>>>>>> This time I was able to watch it closely and try to >>>>>>>>>>>>>> troubleshoot a lot >>>>>>>>>>>>>> more, and also know exactly what server caused it >>>>>>>>>>>>>> (Mercurial with help >>>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap >>>>>>>>>>>>>> servers. The >>>>>>>>>>>>>> user is logging in very quickly and each time it logs in >>>>>>>>>>>>>> I can see in >>>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or >>>>>>>>>>>>>> whatever it is >>>>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are >>>>>>>>>>>>>> filling VERY >>>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue 1: This is causing severe disk latency which >>>>>>>>>>>>>> obviously slows >>>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>>> Issue 2: These changes need to be replicated to my slave >>>>>>>>>>>>>> server thus >>>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> My question is, why does the IPA server fail to keep up >>>>>>>>>>>>>> with the load >>>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>>> >>>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> So I've implemented this server anyway (against my better >>>>>>>>>>>>>> judgement with >>>>>>>>>>>>>> these issues and just made the user that logs into >>>>>>>>>>>>>> mercurial a local >>>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also note before I did that for fun I implemented a RAM >>>>>>>>>>>>>> disk to put the >>>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 >>>>>>>>>>>>>> (except bursts where >>>>>>>>>>>>>> it would raise to 30 to write the access log) but the CPU >>>>>>>>>>>>>> drove to 100% >>>>>>>>>>>>>> trying to keep up with the load. I have also killed the >>>>>>>>>>>>>> replication as >>>>>>>>>>>>>> well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> krblastsuccessfulauth should be excluded from replication, >>>>>>>>>>>>> though I guess that doesn't prevent it from ending up in >>>>>>>>>>>>> the changelog. >>>>>>>>>>>>> >>>>>>>>>>>>> You can confirm that they are excluded by searching the >>>>>>>>>>>>> agreements: >>>>>>>>>>>>> >>>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D >>>>>>>>>>>>> 'cn=directory manager' -W nsDS5ReplicatedAttributeList >>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>>> >>>>>>>>>>>>> They should look like: >>>>>>>>>>>>> >>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>>>>>>> memberof idnssoaserial entryusn krblastsuccessfulauth >>>>>>>>>>>>> krblastfailedauth krbloginfailedcount >>>>>>>>>>>>> >>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ >>>>>>>>>>>>> EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>>>>>>> krbloginfailedcount >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmeggins at redhat.com Fri Aug 30 20:48:35 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Fri, 30 Aug 2013 14:48:35 -0600 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <2C999C19-A8A2-435E-B54F-E6F9F7F964AF@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> <5220FFC2.90207@redhat.com> <2C999C19-A8A2-435E-B54F-E6F9F7F964AF@digitalreasoning.com> Message-ID: <52210523.7030403@redhat.com> On 08/30/2013 02:34 PM, John Moyer wrote: > I put the output in a log and did some command line magic. It doesn't > look bad at all. Highest Etime was 6. > > cat access_output | grep Etime | sort -n | uniq -c > 225 - Etime: 0 > 67 - Etime: 1 > 1 - Etime: 11 > 138 - Etime: 2 > 74 - Etime: 3 > 35 - Etime: 4 > 7 - Etime: 5 > 4 - Etime: 6 > Then I don't understand what exactly is taking 900 seconds? > > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > *Digital Reasoning Systems, Inc.* > John.Moyer at digitalreasoning.com > Office:703.678.2311 > Mobile:240.460.0023 > Fax:703.678.2312 > www.digitalreasoning.com > > On Aug 30, 2013, at 4:25 PM, Rich Megginson > wrote: > >> On 08/30/2013 02:10 PM, John Moyer wrote: >>> So I'm trying to focus on just one use case having the performance >>> issues. I figured out my JIRA server does a sync against this which >>> brings it to it's knees. It takes it 900 seconds to complete the >>> sync and seems to go through every group and user on the machine. I >>> have the base for the user group set to >>> cn=users,cn=accounts,dc=example,dc=com and the groups to >>> cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 >>> seconds off my openldap server I had before this. Any suggestions >>> what to look at? >> >> The access log should have an operation which has etime=900 (or at >> least a very high etime) >> >> If you do logconv.pl -V it will print out a lot of information, >> including the highest etimes. You can then use this report to look >> at the access logs to find out exactly which operations had these >> high etimes. >> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> *Digital Reasoning Systems, Inc.* >>> John.Moyer at digitalreasoning.com >>> Office:703.678.2311 >>> Mobile:240.460.0023 >>> Fax:703.678.2312 >>> www.digitalreasoning.com >>> >>> On Aug 30, 2013, at 3:49 PM, John Moyer >>> >> > wrote: >>> >>>> I'm sorry that was my top unique filter list not my unindexed list. >>>> Please disregard my last email. >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> *Digital Reasoning Systems, Inc.* >>>> John.Moyer at digitalreasoning.com >>>> >>>> Office:703.678.2311 >>>> Mobile:240.460.0023 >>>> Fax:703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Aug 30, 2013, at 3:47 PM, John Moyer >>>> >>> > wrote: >>>> >>>>> If objectclass eq is already indexed how are these on my top >>>>> unindexed list? Wouldn't objectclass eq cover this >>>>> (objectclass=inetorgperson)? and the third and fourth entry? I >>>>> apologize if I'm way off as I am new to the intricacies of LDAP >>>>> indexing. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> >>>>> On Aug 30, 2013, at 3:41 PM, Rich Megginson >>>> > wrote: >>>>> >>>>>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>>>>> Rob or anyone else, >>>>>>> >>>>>>> So while struggling along on this server I just grabbed the logs >>>>>>> off it and ran that log program with the options you suggested. >>>>>>> There are a lot of unindexed requests. These are the top >>>>>>> issues I've removed the one username that showed up. >>>>>>> >>>>>>> So just to double check what I'm thinking. I need to create >>>>>>> three indexes >>>>>>> 1. objectclass pres >>>>>> No, do not create this one >>>>>>> 2. objectclass eq >>>>>> This should already be indexed >>>>>>> 3. uid pres >>>>>> I suppose the UI might be doing this search? >>>>>>> >>>>>>> Please let me know if I'm reading this correctly or if I'm way off? >>>>>>> >>>>>>> >>>>>>> 7337 (objectclass=inetorgperson) >>>>>>> 4597 (objectclass=*) >>>>>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>>>>> 307 (objectclass=krbticketpolicyaux) >>>>>>> 292 (uid=*) >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> *Digital Reasoning Systems, Inc.* >>>>>>> John.Moyer at digitalreasoning.com >>>>>>> >>>>>>> Office:703.678.2311 >>>>>>> Mobile:240.460.0023 >>>>>>> Fax:703.678.2312 >>>>>>> www.digitalreasoning.com >>>>>>> >>>>>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden >>>>>>> > wrote: >>>>>>> >>>>>>>> John Moyer wrote: >>>>>>>>> So this method of search logs is great, and it shows some >>>>>>>>> indexes that would likely highly increase efficiency with my >>>>>>>>> usage. So, are there instructions how to do that? or do you >>>>>>>>> know off hand how to do that? >>>>>>>> >>>>>>>> I'd start with >>>>>>>> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>>>>> >>>>>>>> Note that you'll want to create the same index on all hosts. >>>>>>>> This configuration is not replicated. >>>>>>>> >>>>>>>> You can see the ones we create in /usr/share/ipa/indices.ldif >>>>>>>> and /usr/share/ipa/updates/20-indices.update >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> _____________________________________________________ >>>>>>>>> John Moyer >>>>>>>>> Director, IT Operations >>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>> >>>>>>>>> Office:703.678.2311 >>>>>>>>> Mobile:240.460.0023 >>>>>>>>> Fax:703.678.2312 >>>>>>>>> www.digitalreasoning.com >>>>>>>>> >>>>>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> John Moyer wrote: >>>>>>>>>>> Wow, this is quite insightful, this is the output from that, >>>>>>>>>>> it looks like there aren't many unindexed searches (319 >>>>>>>>>>> doesn't seem like a lot to me at least). Do you have any >>>>>>>>>>> suggestions from this output? >>>>>>>>>> >>>>>>>>>> There are a slew of options you can provide to logconv.pl. I >>>>>>>>>> typically use logconv.pl -ula >>>>>>>>>> /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search >>>>>>>>>> analysis. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>>>>> >>>>>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>>>>> >>>>>>>>>>> Restarts: 2 >>>>>>>>>>> Total Connections: 45224 >>>>>>>>>>> SSL Connections: 44735 >>>>>>>>>>> Peak Concurrent Connections: 76 >>>>>>>>>>> Total Operations: 132568 >>>>>>>>>>> Total Results: 132737 >>>>>>>>>>> Overall Performance: 100.0% >>>>>>>>>>> >>>>>>>>>>> Searches: 61318 (1.76/sec) >>>>>>>>>>> (105.52/min) >>>>>>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>>>>>> Binds: 62143 (1.78/sec) >>>>>>>>>>> (106.94/min) >>>>>>>>>>> >>>>>>>>>>> Proxied Auth Operations: 0 >>>>>>>>>>> Persistent Searches: 3 >>>>>>>>>>> Internal Operations: 0 >>>>>>>>>>> Entry Operations: 0 >>>>>>>>>>> Extended Operations: 8808 >>>>>>>>>>> Abandoned Requests: 0 >>>>>>>>>>> Smart Referrals Received: 0 >>>>>>>>>>> >>>>>>>>>>> VLV Operations: 0 >>>>>>>>>>> VLV Unindexed Searches: 0 >>>>>>>>>>> SORT Operations: 353 >>>>>>>>>>> >>>>>>>>>>> Entire Search Base Queries: 106 >>>>>>>>>>> Unindexed Searches: 319 >>>>>>>>>>> >>>>>>>>>>> FDs Taken: 45262 >>>>>>>>>>> FDs Returned: 45210 >>>>>>>>>>> Highest FD Taken: 139 >>>>>>>>>>> >>>>>>>>>>> Broken Pipes: 0 >>>>>>>>>>> Connections Reset By Peer: 0 >>>>>>>>>>> Resource Unavailable: 0 >>>>>>>>>>> >>>>>>>>>>> Binds: 62143 >>>>>>>>>>> Unbinds: 44539 >>>>>>>>>>> >>>>>>>>>>> LDAP v2 Binds: 2 >>>>>>>>>>> LDAP v3 Binds: 62141 >>>>>>>>>>> SSL Client Binds: 0 >>>>>>>>>>> Failed SSL Client Binds: 0 >>>>>>>>>>> SASL Binds: 1466 >>>>>>>>>>> 1458 GSSAPI >>>>>>>>>>> 8 EXTERNAL >>>>>>>>>>> >>>>>>>>>>> Directory Manager Binds: 10 >>>>>>>>>>> Anonymous Binds: 1476 >>>>>>>>>>> Other Binds: 60657 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> _____________________________________________________ >>>>>>>>>>> John Moyer >>>>>>>>>>> Director, IT Operations >>>>>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>>>>> >>>>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b >>>>>>>>>>>> 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>>>>> >>>>>>>>>>>> Your best bet is to use the logconv.pl tool to examine your >>>>>>>>>>>> logs. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>> John Moyer >>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>>>> Office:703.678.2311 >>>>>>>>>>>>> Mobile:240.460.0023 >>>>>>>>>>>>> Fax:703.678.2312 >>>>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>>>> >>>>>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>>>>> >>>>>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>> >>>>>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> tree,cn=config >>>>>>>>>>>>>> >>>>>>>>>>>>>> dn: cn=meToipa2.example.com >>>>>>>>>>>>>> ,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>>>>>>>> memberof idnssoaserial >>>>>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>>>>>>>> krbloginfailedcount >>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ >>>>>>>>>>>>>> EXCLUDE entryusn krblasts >>>>>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as >>>>>>>>>>>>>>>> we put on a >>>>>>>>>>>>>>>> server that authenticates heavily the IPA came to it's >>>>>>>>>>>>>>>> knees again. >>>>>>>>>>>>>>>> This time I was able to watch it closely and try to >>>>>>>>>>>>>>>> troubleshoot a lot >>>>>>>>>>>>>>>> more, and also know exactly what server caused it >>>>>>>>>>>>>>>> (Mercurial with help >>>>>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap >>>>>>>>>>>>>>>> servers. The >>>>>>>>>>>>>>>> user is logging in very quickly and each time it logs >>>>>>>>>>>>>>>> in I can see in >>>>>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or >>>>>>>>>>>>>>>> whatever it is >>>>>>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are >>>>>>>>>>>>>>>> filling VERY >>>>>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Issue 1: This is causing severe disk latency which >>>>>>>>>>>>>>>> obviously slows >>>>>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>>>>> Issue 2: These changes need to be replicated to my >>>>>>>>>>>>>>>> slave server thus >>>>>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My question is, why does the IPA server fail to keep up >>>>>>>>>>>>>>>> with the load >>>>>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So I've implemented this server anyway (against my >>>>>>>>>>>>>>>> better judgement with >>>>>>>>>>>>>>>> these issues and just made the user that logs into >>>>>>>>>>>>>>>> mercurial a local >>>>>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also note before I did that for fun I implemented a RAM >>>>>>>>>>>>>>>> disk to put the >>>>>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 >>>>>>>>>>>>>>>> (except bursts where >>>>>>>>>>>>>>>> it would raise to 30 to write the access log) but the >>>>>>>>>>>>>>>> CPU drove to 100% >>>>>>>>>>>>>>>> trying to keep up with the load. I have also killed >>>>>>>>>>>>>>>> the replication as >>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> krblastsuccessfulauth should be excluded from >>>>>>>>>>>>>>> replication, though I guess that doesn't prevent it from >>>>>>>>>>>>>>> ending up in the changelog. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You can confirm that they are excluded by searching the >>>>>>>>>>>>>>> agreements: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D >>>>>>>>>>>>>>> 'cn=directory manager' -W nsDS5ReplicatedAttributeList >>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> They should look like: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>>>>>>>>> memberof idnssoaserial entryusn krblastsuccessfulauth >>>>>>>>>>>>>>> krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ >>>>>>>>>>>>>>> EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>>>>>>>>> krbloginfailedcount >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rob >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.moyer at digitalreasoning.com Fri Aug 30 20:54:16 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 16:54:16 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <52210523.7030403@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> <5220FFC2.90207@redhat.com> <2C999C19-A8A2-435E-B54F-E6F9F7F964AF@digitalreasoning.com> <52210523.7030403@redhat.com> Message-ID: <6D4BA0C3-5003-46AD-842A-E95194B865E2@digitalreasoning.com> The full sync of all the users in IPA to JIRA. It's not one query it's many many queries that it hammers the server with. However, the old LDAP server did it in 60 seconds and might have hit 10% CPU while it was happening. This server with more CPU takes 900 seconds while max'ed out at 100% CPU. Could it be uid=* searches? On a clone of this machine I just added a pres index to UID and it dropped to 175 seconds for the sync (but without the rest of my production systems on it) so it's hard to know if that change is from the index or the lack of other queries. Thanks, _____________________________________________________ John Moyer Director, IT Operations On Aug 30, 2013, at 4:48 PM, Rich Megginson wrote: > On 08/30/2013 02:34 PM, John Moyer wrote: >> I put the output in a log and did some command line magic. It doesn't look bad at all. Highest Etime was 6. >> >> cat access_output | grep Etime | sort -n | uniq -c >> 225 - Etime: 0 >> 67 - Etime: 1 >> 1 - Etime: 11 >> 138 - Etime: 2 >> 74 - Etime: 3 >> 35 - Etime: 4 >> 7 - Etime: 5 >> 4 - Etime: 6 >> > > Then I don't understand what exactly is taking 900 seconds? > >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> Digital Reasoning Systems, Inc. >> John.Moyer at digitalreasoning.com >> Office: 703.678.2311 >> Mobile: 240.460.0023 >> Fax: 703.678.2312 >> www.digitalreasoning.com >> >> On Aug 30, 2013, at 4:25 PM, Rich Megginson wrote: >> >>> On 08/30/2013 02:10 PM, John Moyer wrote: >>>> So I'm trying to focus on just one use case having the performance issues. I figured out my JIRA server does a sync against this which brings it to it's knees. It takes it 900 seconds to complete the sync and seems to go through every group and user on the machine. I have the base for the user group set to cn=users,cn=accounts,dc=example,dc=com and the groups to cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 seconds off my openldap server I had before this. Any suggestions what to look at? >>> >>> The access log should have an operation which has etime=900 (or at least a very high etime) >>> >>> If you do logconv.pl -V it will print out a lot of information, including the highest etimes. You can then use this report to look at the access logs to find out exactly which operations had these high etimes. >>> >>>> >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> Digital Reasoning Systems, Inc. >>>> John.Moyer at digitalreasoning.com >>>> Office: 703.678.2311 >>>> Mobile: 240.460.0023 >>>> Fax: 703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Aug 30, 2013, at 3:49 PM, John Moyer wrote: >>>> >>>>> I'm sorry that was my top unique filter list not my unindexed list. Please disregard my last email. >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> Digital Reasoning Systems, Inc. >>>>> John.Moyer at digitalreasoning.com >>>>> Office: 703.678.2311 >>>>> Mobile: 240.460.0023 >>>>> Fax: 703.678.2312 >>>>> www.digitalreasoning.com >>>>> >>>>> On Aug 30, 2013, at 3:47 PM, John Moyer wrote: >>>>> >>>>>> If objectclass eq is already indexed how are these on my top unindexed list? Wouldn't objectclass eq cover this (objectclass=inetorgperson)? and the third and fourth entry? I apologize if I'm way off as I am new to the intricacies of LDAP indexing. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> >>>>>> On Aug 30, 2013, at 3:41 PM, Rich Megginson wrote: >>>>>> >>>>>>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>>>>>> Rob or anyone else, >>>>>>>> >>>>>>>> So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. >>>>>>>> >>>>>>>> So just to double check what I'm thinking. I need to create three indexes >>>>>>>> 1. objectclass pres >>>>>>> No, do not create this one >>>>>>>> 2. objectclass eq >>>>>>> This should already be indexed >>>>>>>> 3. uid pres >>>>>>> I suppose the UI might be doing this search? >>>>>>>> >>>>>>>> Please let me know if I'm reading this correctly or if I'm way off? >>>>>>>> >>>>>>>> >>>>>>>> 7337 (objectclass=inetorgperson) >>>>>>>> 4597 (objectclass=*) >>>>>>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>>>>>> 307 (objectclass=krbticketpolicyaux) >>>>>>>> 292 (uid=*) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> _____________________________________________________ >>>>>>>> John Moyer >>>>>>>> Director, IT Operations >>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>> Office: 703.678.2311 >>>>>>>> Mobile: 240.460.0023 >>>>>>>> Fax: 703.678.2312 >>>>>>>> www.digitalreasoning.com >>>>>>>> >>>>>>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: >>>>>>>> >>>>>>>>> John Moyer wrote: >>>>>>>>>> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? >>>>>>>>> >>>>>>>>> I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>>>>>> >>>>>>>>> Note that you'll want to create the same index on all hosts. This configuration is not replicated. >>>>>>>>> >>>>>>>>> You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> _____________________________________________________ >>>>>>>>>> John Moyer >>>>>>>>>> Director, IT Operations >>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>> Office: 703.678.2311 >>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>> Fax: >>>>>>>>>> 703.678.2312 >>>>>>>>>> www.digitalreasoning.com >>>>>>>>>> >>>>>>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>>>>>>>>> >>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>>>>>>>>>> >>>>>>>>>>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>>>>>> >>>>>>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>>>>>> >>>>>>>>>>>> Restarts: 2 >>>>>>>>>>>> Total Connections: 45224 >>>>>>>>>>>> SSL Connections: 44735 >>>>>>>>>>>> Peak Concurrent Connections: 76 >>>>>>>>>>>> Total Operations: 132568 >>>>>>>>>>>> Total Results: 132737 >>>>>>>>>>>> Overall Performance: 100.0% >>>>>>>>>>>> >>>>>>>>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>>>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>>>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>>>>>>>> >>>>>>>>>>>> Proxied Auth Operations: 0 >>>>>>>>>>>> Persistent Searches: 3 >>>>>>>>>>>> Internal Operations: 0 >>>>>>>>>>>> Entry Operations: 0 >>>>>>>>>>>> Extended Operations: 8808 >>>>>>>>>>>> Abandoned Requests: 0 >>>>>>>>>>>> Smart Referrals Received: 0 >>>>>>>>>>>> >>>>>>>>>>>> VLV Operations: 0 >>>>>>>>>>>> VLV Unindexed Searches: 0 >>>>>>>>>>>> SORT Operations: 353 >>>>>>>>>>>> >>>>>>>>>>>> Entire Search Base Queries: 106 >>>>>>>>>>>> Unindexed Searches: 319 >>>>>>>>>>>> >>>>>>>>>>>> FDs Taken: 45262 >>>>>>>>>>>> FDs Returned: 45210 >>>>>>>>>>>> Highest FD Taken: 139 >>>>>>>>>>>> >>>>>>>>>>>> Broken Pipes: 0 >>>>>>>>>>>> Connections Reset By Peer: 0 >>>>>>>>>>>> Resource Unavailable: 0 >>>>>>>>>>>> >>>>>>>>>>>> Binds: 62143 >>>>>>>>>>>> Unbinds: 44539 >>>>>>>>>>>> >>>>>>>>>>>> LDAP v2 Binds: 2 >>>>>>>>>>>> LDAP v3 Binds: 62141 >>>>>>>>>>>> SSL Client Binds: 0 >>>>>>>>>>>> Failed SSL Client Binds: 0 >>>>>>>>>>>> SASL Binds: 1466 >>>>>>>>>>>> 1458 GSSAPI >>>>>>>>>>>> 8 EXTERNAL >>>>>>>>>>>> >>>>>>>>>>>> Directory Manager Binds: 10 >>>>>>>>>>>> Anonymous Binds: 1476 >>>>>>>>>>>> Other Binds: 60657 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>> John Moyer >>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>>>>>>>>>> >>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>>>>>> >>>>>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>>>>>> >>>>>>>>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>>>>> Office: 703.678.2311 >>>>>>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>>>>>> Fax: >>>>>>>>>>>>>> 703.678.2312 >>>>>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>>>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>>>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>>>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>>>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> They should look like: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rmeggins at redhat.com Fri Aug 30 21:04:09 2013 From: rmeggins at redhat.com (Rich Megginson) Date: Fri, 30 Aug 2013 15:04:09 -0600 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <6D4BA0C3-5003-46AD-842A-E95194B865E2@digitalreasoning.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> <5220FFC2.90207@redhat.com> <2C999C19-A8A2-435E-B54F-E6F9F7F964AF@digitalreasoning.com> <52210523.7030403@redhat.com> <6D4BA0C3-5003-46AD-842A-E95194B865E2@digitalreasonin! g.com> Message-ID: <522108C9.7030506@redhat.com> On 08/30/2013 02:54 PM, John Moyer wrote: > The full sync of all the users in IPA to JIRA. It's not one query > it's many many queries that it hammers the server with. However, the > old LDAP server did it in 60 seconds and might have hit 10% CPU while > it was happening. This server with more CPU takes 900 seconds while > max'ed out at 100% CPU. Could it be uid=* searches? Could be. How many entries match uid=*? How many entries in your database? > > On a clone of this machine I just added a pres index to UID and it > dropped to 175 seconds for the sync (but without the rest of my > production systems on it) so it's hard to know if that change is from > the index or the lack of other queries. 175 is still >> 60. When the sync happens from IPA to JIRA, is the client the JIRA machine? If so, does this machine hit IPA for any other reason during the sync? If not, then we could use the existing access logs and exclude operations that originate from other IP addresses. > > Thanks, > _____________________________________________________ > John Moyer > Director, IT Operations > > On Aug 30, 2013, at 4:48 PM, Rich Megginson > wrote: > >> On 08/30/2013 02:34 PM, John Moyer wrote: >>> I put the output in a log and did some command line magic. It >>> doesn't look bad at all. Highest Etime was 6. >>> >>> cat access_output | grep Etime | sort -n | uniq -c >>> 225 - Etime: 0 >>> 67 - Etime: 1 >>> 1 - Etime: 11 >>> 138 - Etime: 2 >>> 74 - Etime: 3 >>> 35 - Etime: 4 >>> 7 - Etime: 5 >>> 4 - Etime: 6 >>> >> >> Then I don't understand what exactly is taking 900 seconds? >> >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> *Digital Reasoning Systems, Inc.* >>> John.Moyer at digitalreasoning.com >>> Office:703.678.2311 >>> Mobile:240.460.0023 >>> Fax:703.678.2312 >>> www.digitalreasoning.com >>> >>> On Aug 30, 2013, at 4:25 PM, Rich Megginson >> > wrote: >>> >>>> On 08/30/2013 02:10 PM, John Moyer wrote: >>>>> So I'm trying to focus on just one use case having the performance >>>>> issues. I figured out my JIRA server does a sync against this >>>>> which brings it to it's knees. It takes it 900 seconds to >>>>> complete the sync and seems to go through every group and user on >>>>> the machine. I have the base for the user group set to >>>>> cn=users,cn=accounts,dc=example,dc=com and the groups to >>>>> cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 >>>>> seconds off my openldap server I had before this. Any suggestions >>>>> what to look at? >>>> >>>> The access log should have an operation which has etime=900 (or at >>>> least a very high etime) >>>> >>>> If you do logconv.pl -V it will print out a lot of information, >>>> including the highest etimes. You can then use this report to look >>>> at the access logs to find out exactly which operations had these >>>> high etimes. >>>> >>>>> >>>>> >>>>> Thanks, >>>>> _____________________________________________________ >>>>> John Moyer >>>>> Director, IT Operations >>>>> *Digital Reasoning Systems, Inc.* >>>>> John.Moyer at digitalreasoning.com >>>>> >>>>> Office:703.678.2311 >>>>> Mobile:240.460.0023 >>>>> Fax:703.678.2312 >>>>> www.digitalreasoning.com >>>>> >>>>> On Aug 30, 2013, at 3:49 PM, John Moyer >>>>> >>>> > wrote: >>>>> >>>>>> I'm sorry that was my top unique filter list not my unindexed >>>>>> list. Please disregard my last email. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> *Digital Reasoning Systems, Inc.* >>>>>> John.Moyer at digitalreasoning.com >>>>>> >>>>>> Office:703.678.2311 >>>>>> Mobile:240.460.0023 >>>>>> Fax:703.678.2312 >>>>>> www.digitalreasoning.com >>>>>> >>>>>> On Aug 30, 2013, at 3:47 PM, John Moyer >>>>>> >>>>> > wrote: >>>>>> >>>>>>> If objectclass eq is already indexed how are these on my top >>>>>>> unindexed list? Wouldn't objectclass eq cover this >>>>>>> (objectclass=inetorgperson)? and the third and fourth entry? I >>>>>>> apologize if I'm way off as I am new to the intricacies of LDAP >>>>>>> indexing. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> >>>>>>> On Aug 30, 2013, at 3:41 PM, Rich Megginson >>>>>> > wrote: >>>>>>> >>>>>>>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>>>>>>> Rob or anyone else, >>>>>>>>> >>>>>>>>> So while struggling along on this server I just grabbed the >>>>>>>>> logs off it and ran that log program with the options you >>>>>>>>> suggested. There are a lot of unindexed requests. These are >>>>>>>>> the top issues I've removed the one username that showed up. >>>>>>>>> >>>>>>>>> So just to double check what I'm thinking. I need to create >>>>>>>>> three indexes >>>>>>>>> 1. objectclass pres >>>>>>>> No, do not create this one >>>>>>>>> 2. objectclass eq >>>>>>>> This should already be indexed >>>>>>>>> 3. uid pres >>>>>>>> I suppose the UI might be doing this search? >>>>>>>>> >>>>>>>>> Please let me know if I'm reading this correctly or if I'm way >>>>>>>>> off? >>>>>>>>> >>>>>>>>> >>>>>>>>> 7337 (objectclass=inetorgperson) >>>>>>>>> 4597 (objectclass=*) >>>>>>>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>>>>>>> 307 (objectclass=krbticketpolicyaux) >>>>>>>>> 292 (uid=*) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> _____________________________________________________ >>>>>>>>> John Moyer >>>>>>>>> Director, IT Operations >>>>>>>>> *Digital Reasoning Systems, Inc.* >>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>> >>>>>>>>> Office:703.678.2311 >>>>>>>>> Mobile:240.460.0023 >>>>>>>>> Fax:703.678.2312 >>>>>>>>> www.digitalreasoning.com >>>>>>>>> >>>>>>>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> John Moyer wrote: >>>>>>>>>>> So this method of search logs is great, and it shows some >>>>>>>>>>> indexes that would likely highly increase efficiency with my >>>>>>>>>>> usage. So, are there instructions how to do that? or do >>>>>>>>>>> you know off hand how to do that? >>>>>>>>>> >>>>>>>>>> I'd start with >>>>>>>>>> https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>>>>>>> >>>>>>>>>> Note that you'll want to create the same index on all hosts. >>>>>>>>>> This configuration is not replicated. >>>>>>>>>> >>>>>>>>>> You can see the ones we create in /usr/share/ipa/indices.ldif >>>>>>>>>> and /usr/share/ipa/updates/20-indices.update >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> _____________________________________________________ >>>>>>>>>>> John Moyer >>>>>>>>>>> Director, IT Operations >>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>> >>>>>>>>>>> Office:703.678.2311 >>>>>>>>>>> Mobile:240.460.0023 >>>>>>>>>>> Fax:703.678.2312 >>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>> >>>>>>>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>> Wow, this is quite insightful, this is the output from >>>>>>>>>>>>> that, it looks like there aren't many unindexed searches >>>>>>>>>>>>> (319 doesn't seem like a lot to me at least). Do you have >>>>>>>>>>>>> any suggestions from this output? >>>>>>>>>>>> >>>>>>>>>>>> There are a slew of options you can provide to logconv.pl. >>>>>>>>>>>> I typically use logconv.pl -ula >>>>>>>>>>>> /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search >>>>>>>>>>>> analysis. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>>>>>>> >>>>>>>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>>>>>>> >>>>>>>>>>>>> Restarts: 2 >>>>>>>>>>>>> Total Connections: 45224 >>>>>>>>>>>>> SSL Connections: 44735 >>>>>>>>>>>>> Peak Concurrent Connections: 76 >>>>>>>>>>>>> Total Operations: 132568 >>>>>>>>>>>>> Total Results: 132737 >>>>>>>>>>>>> Overall Performance: 100.0% >>>>>>>>>>>>> >>>>>>>>>>>>> Searches: 61318 (1.76/sec) >>>>>>>>>>>>> (105.52/min) >>>>>>>>>>>>> Modifications: 277 (0.01/sec) >>>>>>>>>>>>> (0.48/min) >>>>>>>>>>>>> Adds: 10 (0.00/sec) >>>>>>>>>>>>> (0.02/min) >>>>>>>>>>>>> Deletes: 12 (0.00/sec) >>>>>>>>>>>>> (0.02/min) >>>>>>>>>>>>> Mod RDNs: 0 (0.00/sec) >>>>>>>>>>>>> (0.00/min) >>>>>>>>>>>>> Compares: 0 (0.00/sec) >>>>>>>>>>>>> (0.00/min) >>>>>>>>>>>>> Binds: 62143 (1.78/sec) >>>>>>>>>>>>> (106.94/min) >>>>>>>>>>>>> >>>>>>>>>>>>> Proxied Auth Operations: 0 >>>>>>>>>>>>> Persistent Searches: 3 >>>>>>>>>>>>> Internal Operations: 0 >>>>>>>>>>>>> Entry Operations: 0 >>>>>>>>>>>>> Extended Operations: 8808 >>>>>>>>>>>>> Abandoned Requests: 0 >>>>>>>>>>>>> Smart Referrals Received: 0 >>>>>>>>>>>>> >>>>>>>>>>>>> VLV Operations: 0 >>>>>>>>>>>>> VLV Unindexed Searches: 0 >>>>>>>>>>>>> SORT Operations: 353 >>>>>>>>>>>>> >>>>>>>>>>>>> Entire Search Base Queries: 106 >>>>>>>>>>>>> Unindexed Searches: 319 >>>>>>>>>>>>> >>>>>>>>>>>>> FDs Taken: 45262 >>>>>>>>>>>>> FDs Returned: 45210 >>>>>>>>>>>>> Highest FD Taken: 139 >>>>>>>>>>>>> >>>>>>>>>>>>> Broken Pipes: 0 >>>>>>>>>>>>> Connections Reset By Peer: 0 >>>>>>>>>>>>> Resource Unavailable: 0 >>>>>>>>>>>>> >>>>>>>>>>>>> Binds: 62143 >>>>>>>>>>>>> Unbinds: 44539 >>>>>>>>>>>>> >>>>>>>>>>>>> LDAP v2 Binds: 2 >>>>>>>>>>>>> LDAP v3 Binds: 62141 >>>>>>>>>>>>> SSL Client Binds: 0 >>>>>>>>>>>>> Failed SSL Client Binds: 0 >>>>>>>>>>>>> SASL Binds: 1466 >>>>>>>>>>>>> 1458 GSSAPI >>>>>>>>>>>>> 8 EXTERNAL >>>>>>>>>>>>> >>>>>>>>>>>>> Directory Manager Binds: 10 >>>>>>>>>>>>> Anonymous Binds: 1476 >>>>>>>>>>>>> Other Binds: 60657 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>> John Moyer >>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>>>>>>> >>>>>>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b >>>>>>>>>>>>>> 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>>>>>>> >>>>>>>>>>>>>> Your best bet is to use the logconv.pl tool to examine >>>>>>>>>>>>>> your logs. >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>>>>>> Office:703.678.2311 >>>>>>>>>>>>>>> Mobile:240.460.0023 >>>>>>>>>>>>>>> Fax:703.678.2312 >>>>>>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>>> tree,cn=config >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dn: cn=meToipa2.example.com >>>>>>>>>>>>>>>> ,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE >>>>>>>>>>>>>>>> memberof idnssoaserial >>>>>>>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth >>>>>>>>>>>>>>>> krbloginfailedcount >>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ >>>>>>>>>>>>>>>> EXCLUDE entryusn krblasts >>>>>>>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon >>>>>>>>>>>>>>>>>> as we put on a >>>>>>>>>>>>>>>>>> server that authenticates heavily the IPA came to >>>>>>>>>>>>>>>>>> it's knees again. >>>>>>>>>>>>>>>>>> This time I was able to watch it closely and try to >>>>>>>>>>>>>>>>>> troubleshoot a lot >>>>>>>>>>>>>>>>>> more, and also know exactly what server caused it >>>>>>>>>>>>>>>>>> (Mercurial with help >>>>>>>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap >>>>>>>>>>>>>>>>>> servers. The >>>>>>>>>>>>>>>>>> user is logging in very quickly and each time it logs >>>>>>>>>>>>>>>>>> in I can see in >>>>>>>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter >>>>>>>>>>>>>>>>>> (or whatever it is >>>>>>>>>>>>>>>>>> called) is updated over and over and over in the >>>>>>>>>>>>>>>>>> changelog >>>>>>>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are >>>>>>>>>>>>>>>>>> filling VERY >>>>>>>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Issue 1: This is causing severe disk latency which >>>>>>>>>>>>>>>>>> obviously slows >>>>>>>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>>>>>>> Issue 2: These changes need to be replicated to my >>>>>>>>>>>>>>>>>> slave server thus >>>>>>>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> My question is, why does the IPA server fail to keep >>>>>>>>>>>>>>>>>> up with the load >>>>>>>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So I've implemented this server anyway (against my >>>>>>>>>>>>>>>>>> better judgement with >>>>>>>>>>>>>>>>>> these issues and just made the user that logs into >>>>>>>>>>>>>>>>>> mercurial a local >>>>>>>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also note before I did that for fun I implemented a >>>>>>>>>>>>>>>>>> RAM disk to put the >>>>>>>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 >>>>>>>>>>>>>>>>>> (except bursts where >>>>>>>>>>>>>>>>>> it would raise to 30 to write the access log) but the >>>>>>>>>>>>>>>>>> CPU drove to 100% >>>>>>>>>>>>>>>>>> trying to keep up with the load. I have also killed >>>>>>>>>>>>>>>>>> the replication as >>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> krblastsuccessfulauth should be excluded from >>>>>>>>>>>>>>>>> replication, though I guess that doesn't prevent it >>>>>>>>>>>>>>>>> from ending up in the changelog. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You can confirm that they are excluded by searching >>>>>>>>>>>>>>>>> the agreements: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D >>>>>>>>>>>>>>>>> 'cn=directory manager' -W nsDS5ReplicatedAttributeList >>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> They should look like: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ >>>>>>>>>>>>>>>>> EXCLUDE memberof idnssoaserial entryusn >>>>>>>>>>>>>>>>> krblastsuccessfulauth krblastfailedauth >>>>>>>>>>>>>>>>> krbloginfailedcount >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ >>>>>>>>>>>>>>>>> EXCLUDE entryusn krblastsuccessfulauth >>>>>>>>>>>>>>>>> krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dpal at redhat.com Fri Aug 30 21:04:15 2013 From: dpal at redhat.com (Dmitri Pal) Date: Fri, 30 Aug 2013 17:04:15 -0400 Subject: [Freeipa-users] FreeIPA on Debian Message-ID: <522108CF.8040707@redhat.com> Hello, Sorry for cross posting to 4 different lists but it seems that this is the best way to include most of people who might be interested in this discussion. The question of "When FreeIPA will be available on Debian?" has been coming up periodically on the list(s) without any resolution. However it is clear that it would be beneficial for the community and the project. May be it is time to try again? Let us see why it yet has not happened? 1) Some components need to be ported to Debian especially Dogtag and a slew of its new RESTEasy dependencies. This requires time and quite an effort from someone familiar with the domain. 2) The code needs to be changed in installer and potentially in other places as it might have had some Fedorizms blended in 3) Someone needs to own packages in Debian and maintain them, someone with good knowledge of the distro and time to take ownership of about 50 packages. Can we pull it off together this time? Say we plan for some Dogtag and IPA domain experts to work on the port during Nov 13 - Feb 14 and address 1) and 2). Would there be any interest to join forces with them? Would there be anyone to take on item 3) from the list above? -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. ------------------------------- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ From john.moyer at digitalreasoning.com Fri Aug 30 21:08:16 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 17:08:16 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <522108C9.7030506@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> <5220FFC2.90207@redhat.com> <2C999C19-A8A2-435E-B54F-E6F9F7F964AF@digitalreasoning.com> <52210523.7030403@redhat.com> <6D4BA0C3-5003-46AD-842A-E95194B865E2@digitalreasonin! ! g.com> <522108C9.7030506@redhat.com> Message-ID: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log: 30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections: 824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results: 1805 Overall Performance: 99.9% Searches: 968 (4.02/sec) (241.00/min) Modifications: 5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds: 833 (3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 30, 2013, at 5:04 PM, Rich Megginson wrote: > On 08/30/2013 02:54 PM, John Moyer wrote: >> The full sync of all the users in IPA to JIRA. It's not one query it's many many queries that it hammers the server with. However, the old LDAP server did it in 60 seconds and might have hit 10% CPU while it was happening. This server with more CPU takes 900 seconds while max'ed out at 100% CPU. Could it be uid=* searches? > > Could be. How many entries match uid=*? How many entries in your database? > >> >> On a clone of this machine I just added a pres index to UID and it dropped to 175 seconds for the sync (but without the rest of my production systems on it) so it's hard to know if that change is from the index or the lack of other queries. > > 175 is still >> 60. > > When the sync happens from IPA to JIRA, is the client the JIRA machine? If so, does this machine hit IPA for any other reason during the sync? If not, then we could use the existing access logs and exclude operations that originate from other IP addresses. > >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> >> On Aug 30, 2013, at 4:48 PM, Rich Megginson wrote: >> >>> On 08/30/2013 02:34 PM, John Moyer wrote: >>>> I put the output in a log and did some command line magic. It doesn't look bad at all. Highest Etime was 6. >>>> >>>> cat access_output | grep Etime | sort -n | uniq -c >>>> 225 - Etime: 0 >>>> 67 - Etime: 1 >>>> 1 - Etime: 11 >>>> 138 - Etime: 2 >>>> 74 - Etime: 3 >>>> 35 - Etime: 4 >>>> 7 - Etime: 5 >>>> 4 - Etime: 6 >>>> >>> >>> Then I don't understand what exactly is taking 900 seconds? >>> >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> Digital Reasoning Systems, Inc. >>>> John.Moyer at digitalreasoning.com >>>> Office: 703.678.2311 >>>> Mobile: 240.460.0023 >>>> Fax: 703.678.2312 >>>> www.digitalreasoning.com >>>> >>>> On Aug 30, 2013, at 4:25 PM, Rich Megginson wrote: >>>> >>>>> On 08/30/2013 02:10 PM, John Moyer wrote: >>>>>> So I'm trying to focus on just one use case having the performance issues. I figured out my JIRA server does a sync against this which brings it to it's knees. It takes it 900 seconds to complete the sync and seems to go through every group and user on the machine. I have the base for the user group set to cn=users,cn=accounts,dc=example,dc=com and the groups to cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 seconds off my openldap server I had before this. Any suggestions what to look at? >>>>> >>>>> The access log should have an operation which has etime=900 (or at least a very high etime) >>>>> >>>>> If you do logconv.pl -V it will print out a lot of information, including the highest etimes. You can then use this report to look at the access logs to find out exactly which operations had these high etimes. >>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> Digital Reasoning Systems, Inc. >>>>>> John.Moyer at digitalreasoning.com >>>>>> Office: 703.678.2311 >>>>>> Mobile: 240.460.0023 >>>>>> Fax: 703.678.2312 >>>>>> www.digitalreasoning.com >>>>>> >>>>>> On Aug 30, 2013, at 3:49 PM, John Moyer wrote: >>>>>> >>>>>>> I'm sorry that was my top unique filter list not my unindexed list. Please disregard my last email. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> _____________________________________________________ >>>>>>> John Moyer >>>>>>> Director, IT Operations >>>>>>> Digital Reasoning Systems, Inc. >>>>>>> John.Moyer at digitalreasoning.com >>>>>>> Office: 703.678.2311 >>>>>>> Mobile: 240.460.0023 >>>>>>> Fax: 703.678.2312 >>>>>>> www.digitalreasoning.com >>>>>>> >>>>>>> On Aug 30, 2013, at 3:47 PM, John Moyer wrote: >>>>>>> >>>>>>>> If objectclass eq is already indexed how are these on my top unindexed list? Wouldn't objectclass eq cover this (objectclass=inetorgperson)? and the third and fourth entry? I apologize if I'm way off as I am new to the intricacies of LDAP indexing. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> _____________________________________________________ >>>>>>>> John Moyer >>>>>>>> Director, IT Operations >>>>>>>> >>>>>>>> On Aug 30, 2013, at 3:41 PM, Rich Megginson wrote: >>>>>>>> >>>>>>>>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>>>>>>>> Rob or anyone else, >>>>>>>>>> >>>>>>>>>> So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. >>>>>>>>>> >>>>>>>>>> So just to double check what I'm thinking. I need to create three indexes >>>>>>>>>> 1. objectclass pres >>>>>>>>> No, do not create this one >>>>>>>>>> 2. objectclass eq >>>>>>>>> This should already be indexed >>>>>>>>>> 3. uid pres >>>>>>>>> I suppose the UI might be doing this search? >>>>>>>>>> >>>>>>>>>> Please let me know if I'm reading this correctly or if I'm way off? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 7337 (objectclass=inetorgperson) >>>>>>>>>> 4597 (objectclass=*) >>>>>>>>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>>>>>>>> 307 (objectclass=krbticketpolicyaux) >>>>>>>>>> 292 (uid=*) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> _____________________________________________________ >>>>>>>>>> John Moyer >>>>>>>>>> Director, IT Operations >>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>> Office: 703.678.2311 >>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>> Fax: 703.678.2312 >>>>>>>>>> www.digitalreasoning.com >>>>>>>>>> >>>>>>>>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: >>>>>>>>>> >>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? >>>>>>>>>>> >>>>>>>>>>> I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>>>>>>>> >>>>>>>>>>> Note that you'll want to create the same index on all hosts. This configuration is not replicated. >>>>>>>>>>> >>>>>>>>>>> You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>> John Moyer >>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>>> Office: 703.678.2311 >>>>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>>>> Fax: >>>>>>>>>>>> 703.678.2312 >>>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>>> >>>>>>>>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>>>>>>>>>>> >>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>>>>>>>>>>>> >>>>>>>>>>>>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>>>>>>>> >>>>>>>>>>>>>> Restarts: 2 >>>>>>>>>>>>>> Total Connections: 45224 >>>>>>>>>>>>>> SSL Connections: 44735 >>>>>>>>>>>>>> Peak Concurrent Connections: 76 >>>>>>>>>>>>>> Total Operations: 132568 >>>>>>>>>>>>>> Total Results: 132737 >>>>>>>>>>>>>> Overall Performance: 100.0% >>>>>>>>>>>>>> >>>>>>>>>>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>>>>>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>>>>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>>>>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>>>>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>>>>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>>>>>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Proxied Auth Operations: 0 >>>>>>>>>>>>>> Persistent Searches: 3 >>>>>>>>>>>>>> Internal Operations: 0 >>>>>>>>>>>>>> Entry Operations: 0 >>>>>>>>>>>>>> Extended Operations: 8808 >>>>>>>>>>>>>> Abandoned Requests: 0 >>>>>>>>>>>>>> Smart Referrals Received: 0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> VLV Operations: 0 >>>>>>>>>>>>>> VLV Unindexed Searches: 0 >>>>>>>>>>>>>> SORT Operations: 353 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Entire Search Base Queries: 106 >>>>>>>>>>>>>> Unindexed Searches: 319 >>>>>>>>>>>>>> >>>>>>>>>>>>>> FDs Taken: 45262 >>>>>>>>>>>>>> FDs Returned: 45210 >>>>>>>>>>>>>> Highest FD Taken: 139 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Broken Pipes: 0 >>>>>>>>>>>>>> Connections Reset By Peer: 0 >>>>>>>>>>>>>> Resource Unavailable: 0 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Binds: 62143 >>>>>>>>>>>>>> Unbinds: 44539 >>>>>>>>>>>>>> >>>>>>>>>>>>>> LDAP v2 Binds: 2 >>>>>>>>>>>>>> LDAP v3 Binds: 62141 >>>>>>>>>>>>>> SSL Client Binds: 0 >>>>>>>>>>>>>> Failed SSL Client Binds: 0 >>>>>>>>>>>>>> SASL Binds: 1466 >>>>>>>>>>>>>> 1458 GSSAPI >>>>>>>>>>>>>> 8 EXTERNAL >>>>>>>>>>>>>> >>>>>>>>>>>>>> Directory Manager Binds: 10 >>>>>>>>>>>>>> Anonymous Binds: 1476 >>>>>>>>>>>>>> Other Binds: 60657 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>>>>>>> Office: 703.678.2311 >>>>>>>>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>>>>>>>> Fax: >>>>>>>>>>>>>>>> 703.678.2312 >>>>>>>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>>>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>>>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>>>>>> John Moyer >>>>>>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>>>>>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>>>>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>>>>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>>>>>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>>>>>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>>>>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>>>>>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> They should look like: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From john.moyer at digitalreasoning.com Fri Aug 30 20:34:39 2013 From: john.moyer at digitalreasoning.com (John Moyer) Date: Fri, 30 Aug 2013 16:34:39 -0400 Subject: [Freeipa-users] IPA Load Problems? In-Reply-To: <5220FFC2.90207@redhat.com> References: <70FAFDC6-AC64-4B42-AB75-ABECDE7B350E@digitalreasoning.com> <52010EE7.8090402@redhat.com> <7AC4F9FE-D4A8-44A6-89FB-3DDC27398D14@digitalreasoning.com> <521CB43A.40104@redhat.com> <521CDE4F.50307@redhat.com> <3D595AA0-9168-45C3-AB29-602567EAC5E4@digitalreasoning.com> <521D0FCE.3080800@redhat.com> <3B676396-41CB-40BA-A7D2-0E1CB8800C30@digitalreasoning.com> <521E1A03.1060904@redhat.com> <5220F587.6000808@redhat.com> <414556BA-3DDF-4259-AC40-D0778BF474DC@digitalreasoning.com> <5220FFC2.90207@redhat.com> Message-ID: <2C999C19-A8A2-435E-B54F-E6F9F7F964AF@digitalreasoning.com> I put the output in a log and did some command line magic. It doesn't look bad at all. Highest Etime was 6. cat access_output | grep Etime | sort -n | uniq -c 225 - Etime: 0 67 - Etime: 1 1 - Etime: 11 138 - Etime: 2 74 - Etime: 3 35 - Etime: 4 7 - Etime: 5 4 - Etime: 6 Thanks, _____________________________________________________ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. John.Moyer at digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Aug 30, 2013, at 4:25 PM, Rich Megginson wrote: > On 08/30/2013 02:10 PM, John Moyer wrote: >> So I'm trying to focus on just one use case having the performance issues. I figured out my JIRA server does a sync against this which brings it to it's knees. It takes it 900 seconds to complete the sync and seems to go through every group and user on the machine. I have the base for the user group set to cn=users,cn=accounts,dc=example,dc=com and the groups to cn=groups,cn=accounts,dc=example,dc=com. This sync took around 60 seconds off my openldap server I had before this. Any suggestions what to look at? > > The access log should have an operation which has etime=900 (or at least a very high etime) > > If you do logconv.pl -V it will print out a lot of information, including the highest etimes. You can then use this report to look at the access logs to find out exactly which operations had these high etimes. > >> >> >> >> Thanks, >> _____________________________________________________ >> John Moyer >> Director, IT Operations >> Digital Reasoning Systems, Inc. >> John.Moyer at digitalreasoning.com >> Office: 703.678.2311 >> Mobile: 240.460.0023 >> Fax: 703.678.2312 >> www.digitalreasoning.com >> >> On Aug 30, 2013, at 3:49 PM, John Moyer wrote: >> >>> I'm sorry that was my top unique filter list not my unindexed list. Please disregard my last email. >>> >>> >>> Thanks, >>> _____________________________________________________ >>> John Moyer >>> Director, IT Operations >>> Digital Reasoning Systems, Inc. >>> John.Moyer at digitalreasoning.com >>> Office: 703.678.2311 >>> Mobile: 240.460.0023 >>> Fax: 703.678.2312 >>> www.digitalreasoning.com >>> >>> On Aug 30, 2013, at 3:47 PM, John Moyer wrote: >>> >>>> If objectclass eq is already indexed how are these on my top unindexed list? Wouldn't objectclass eq cover this (objectclass=inetorgperson)? and the third and fourth entry? I apologize if I'm way off as I am new to the intricacies of LDAP indexing. >>>> >>>> >>>> >>>> Thanks, >>>> _____________________________________________________ >>>> John Moyer >>>> Director, IT Operations >>>> >>>> On Aug 30, 2013, at 3:41 PM, Rich Megginson wrote: >>>> >>>>> On 08/30/2013 01:31 PM, John Moyer wrote: >>>>>> Rob or anyone else, >>>>>> >>>>>> So while struggling along on this server I just grabbed the logs off it and ran that log program with the options you suggested. There are a lot of unindexed requests. These are the top issues I've removed the one username that showed up. >>>>>> >>>>>> So just to double check what I'm thinking. I need to create three indexes >>>>>> 1. objectclass pres >>>>> No, do not create this one >>>>>> 2. objectclass eq >>>>> This should already be indexed >>>>>> 3. uid pres >>>>> I suppose the UI might be doing this search? >>>>>> >>>>>> Please let me know if I'm reading this correctly or if I'm way off? >>>>>> >>>>>> >>>>>> 7337 (objectclass=inetorgperson) >>>>>> 4597 (objectclass=*) >>>>>> 4560 (&(objectclass=inetorgperson)(uid=senior.developer.login)) >>>>>> 307 (objectclass=krbticketpolicyaux) >>>>>> 292 (uid=*) >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> _____________________________________________________ >>>>>> John Moyer >>>>>> Director, IT Operations >>>>>> Digital Reasoning Systems, Inc. >>>>>> John.Moyer at digitalreasoning.com >>>>>> Office: 703.678.2311 >>>>>> Mobile: 240.460.0023 >>>>>> Fax: 703.678.2312 >>>>>> www.digitalreasoning.com >>>>>> >>>>>> On Aug 28, 2013, at 11:40 AM, Rob Crittenden wrote: >>>>>> >>>>>>> John Moyer wrote: >>>>>>>> So this method of search logs is great, and it shows some indexes that would likely highly increase efficiency with my usage. So, are there instructions how to do that? or do you know off hand how to do that? >>>>>>> >>>>>>> I'd start with https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html-single/Administration_Guide/index.html#Managing_Indexes-About_Indexes >>>>>>> >>>>>>> Note that you'll want to create the same index on all hosts. This configuration is not replicated. >>>>>>> >>>>>>> You can see the ones we create in /usr/share/ipa/indices.ldif and /usr/share/ipa/updates/20-indices.update >>>>>>> >>>>>>> rob >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> _____________________________________________________ >>>>>>>> John Moyer >>>>>>>> Director, IT Operations >>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>> Office: >>>>>>>> 703.678.2311 >>>>>>>> Mobile: >>>>>>>> 240.460.0023 >>>>>>>> Fax: >>>>>>>> >>>>>>>> 703.678.2312 >>>>>>>> www.digitalreasoning.com >>>>>>>> >>>>>>>> On Aug 27, 2013, at 4:45 PM, Rob Crittenden wrote: >>>>>>>> >>>>>>>>> John Moyer wrote: >>>>>>>>>> Wow, this is quite insightful, this is the output from that, it looks like there aren't many unindexed searches (319 doesn't seem like a lot to me at least). Do you have any suggestions from this output? >>>>>>>>> >>>>>>>>> There are a slew of options you can provide to logconv.pl. I typically use logconv.pl -ula /var/log/dirsrv/slapd-EXAMPLE-COM/access when doing search analysis. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Start of Log: 27/Aug/2013:02:36:08 >>>>>>>>>> End of Log: 27/Aug/2013:12:17:15 >>>>>>>>>> >>>>>>>>>> Processed Log Time: 9 Hours, 41 Minutes, 7 Seconds >>>>>>>>>> >>>>>>>>>> Restarts: 2 >>>>>>>>>> Total Connections: 45224 >>>>>>>>>> SSL Connections: 44735 >>>>>>>>>> Peak Concurrent Connections: 76 >>>>>>>>>> Total Operations: 132568 >>>>>>>>>> Total Results: 132737 >>>>>>>>>> Overall Performance: 100.0% >>>>>>>>>> >>>>>>>>>> Searches: 61318 (1.76/sec) (105.52/min) >>>>>>>>>> Modifications: 277 (0.01/sec) (0.48/min) >>>>>>>>>> Adds: 10 (0.00/sec) (0.02/min) >>>>>>>>>> Deletes: 12 (0.00/sec) (0.02/min) >>>>>>>>>> Mod RDNs: 0 (0.00/sec) (0.00/min) >>>>>>>>>> Compares: 0 (0.00/sec) (0.00/min) >>>>>>>>>> Binds: 62143 (1.78/sec) (106.94/min) >>>>>>>>>> >>>>>>>>>> Proxied Auth Operations: 0 >>>>>>>>>> Persistent Searches: 3 >>>>>>>>>> Internal Operations: 0 >>>>>>>>>> Entry Operations: 0 >>>>>>>>>> Extended Operations: 8808 >>>>>>>>>> Abandoned Requests: 0 >>>>>>>>>> Smart Referrals Received: 0 >>>>>>>>>> >>>>>>>>>> VLV Operations: 0 >>>>>>>>>> VLV Unindexed Searches: 0 >>>>>>>>>> SORT Operations: 353 >>>>>>>>>> >>>>>>>>>> Entire Search Base Queries: 106 >>>>>>>>>> Unindexed Searches: 319 >>>>>>>>>> >>>>>>>>>> FDs Taken: 45262 >>>>>>>>>> FDs Returned: 45210 >>>>>>>>>> Highest FD Taken: 139 >>>>>>>>>> >>>>>>>>>> Broken Pipes: 0 >>>>>>>>>> Connections Reset By Peer: 0 >>>>>>>>>> Resource Unavailable: 0 >>>>>>>>>> >>>>>>>>>> Binds: 62143 >>>>>>>>>> Unbinds: 44539 >>>>>>>>>> >>>>>>>>>> LDAP v2 Binds: 2 >>>>>>>>>> LDAP v3 Binds: 62141 >>>>>>>>>> SSL Client Binds: 0 >>>>>>>>>> Failed SSL Client Binds: 0 >>>>>>>>>> SASL Binds: 1466 >>>>>>>>>> 1458 GSSAPI >>>>>>>>>> 8 EXTERNAL >>>>>>>>>> >>>>>>>>>> Directory Manager Binds: 10 >>>>>>>>>> Anonymous Binds: 1476 >>>>>>>>>> Other Binds: 60657 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> _____________________________________________________ >>>>>>>>>> John Moyer >>>>>>>>>> Director, IT Operations >>>>>>>>>> On Aug 27, 2013, at 1:13 PM, Rob Crittenden wrote: >>>>>>>>>> >>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>> Is there any way to see what fields are index'ed? >>>>>>>>>>> >>>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -x -b 'cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config' >>>>>>>>>>> >>>>>>>>>>> Your best bet is to use the logconv.pl tool to examine your logs. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>> John Moyer >>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>> Digital Reasoning Systems, Inc. >>>>>>>>>>>> John.Moyer at digitalreasoning.com >>>>>>>>>>>> Office: 703.678.2311 >>>>>>>>>>>> Mobile: 240.460.0023 >>>>>>>>>>>> Fax: >>>>>>>>>>>> 703.678.2312 >>>>>>>>>>>> www.digitalreasoning.com >>>>>>>>>>>> >>>>>>>>>>>> On Aug 27, 2013, at 10:36 AM, John Moyer wrote: >>>>>>>>>>>> >>>>>>>>>>>>> That looks like the output I just got shown below: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> dn: cn=mapping tree,cn=config >>>>>>>>>>>>> >>>>>>>>>>>>> dn: cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>> >>>>>>>>>>>>> dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>> >>>>>>>>>>>>> dn: cn=meToipa2.example.com,cn=replica,cn=dc\3Dexample\ >>>>>>>>>>>>> 2Cdc\3Dcom,cn=mapping tree,cn=config >>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial >>>>>>>>>>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts >>>>>>>>>>>>> uccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> _____________________________________________________ >>>>>>>>>>>>> John Moyer >>>>>>>>>>>>> Director, IT Operations >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Aug 27, 2013, at 10:14 AM, Rob Crittenden wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> John Moyer wrote: >>>>>>>>>>>>>>> Ok, so we tried to implement this again, and as soon as we put on a >>>>>>>>>>>>>>> server that authenticates heavily the IPA came to it's knees again. >>>>>>>>>>>>>>> This time I was able to watch it closely and try to troubleshoot a lot >>>>>>>>>>>>>>> more, and also know exactly what server caused it (Mercurial with help >>>>>>>>>>>>>>> of bamboo). This runs fine on a normal old openldap servers. The >>>>>>>>>>>>>>> user is logging in very quickly and each time it logs in I can see in >>>>>>>>>>>>>>> the logs that the krbLastsuccessfullogin parameter (or whatever it is >>>>>>>>>>>>>>> called) is updated over and over and over in the changelog >>>>>>>>>>>>>>> (/var/lib/dirsrv/slapd-$instanceid/db) those logs are filling VERY >>>>>>>>>>>>>>> quickly and then disappear fairly quickly as well. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issue 1: This is causing severe disk latency which obviously slows >>>>>>>>>>>>>>> everything down wait times were around 25%+ >>>>>>>>>>>>>>> Issue 2: These changes need to be replicated to my slave server thus >>>>>>>>>>>>>>> adding to the mess >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My question is, why does the IPA server fail to keep up with the load >>>>>>>>>>>>>>> when the openLDAP server didn't have an issue. Indexes? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm running the following: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> CentOS release 6.4 (Final) >>>>>>>>>>>>>>> 389-ds-base-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>> 389-ds-base-libs-1.2.11.15-20.el6_4.x86_64 >>>>>>>>>>>>>>> ipa-python-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>> ipa-admintools-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>> ipa-pki-common-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>> python-iniparse-0.3.1-2.1.el6.noarch >>>>>>>>>>>>>>> ipa-server-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>> ipa-pki-ca-theme-9.0.3-7.el6.noarch >>>>>>>>>>>>>>> ipa-server-selinux-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>> libipa_hbac-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>> ipa-client-3.0.0-26.el6_4.4.x86_64 >>>>>>>>>>>>>>> libipa_hbac-python-1.9.2-82.7.el6_4.x86_64 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So I've implemented this server anyway (against my better judgement with >>>>>>>>>>>>>>> these issues and just made the user that logs into mercurial a local >>>>>>>>>>>>>>> user instead of IPA). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also note before I did that for fun I implemented a RAM disk to put the >>>>>>>>>>>>>>> change logs on, and that dropped the wait time to 0 (except bursts where >>>>>>>>>>>>>>> it would raise to 30 to write the access log) but the CPU drove to 100% >>>>>>>>>>>>>>> trying to keep up with the load. I have also killed the replication as >>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> krblastsuccessfulauth should be excluded from replication, though I guess that doesn't prevent it from ending up in the changelog. >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can confirm that they are excluded by searching the agreements: >>>>>>>>>>>>>> >>>>>>>>>>>>>> $ ldapsearch -LLL -x -b 'cn=mapping tree,cn=config' -D 'cn=directory manager' -W nsDS5ReplicatedAttributeList nsDS5ReplicatedAttributeListTotal >>>>>>>>>>>>>> >>>>>>>>>>>>>> They should look like: >>>>>>>>>>>>>> >>>>>>>>>>>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>> >>>>>>>>>>>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail URL: From michal.dwuznik at gmail.com Sat Aug 31 19:50:57 2013 From: michal.dwuznik at gmail.com (=?UTF-8?B?TWljaGHFgiBEd3XFvG5paw==?=) Date: Sat, 31 Aug 2013 21:50:57 +0200 Subject: [Freeipa-users] FreeIPA on Debian In-Reply-To: <522108CF.8040707@redhat.com> References: <522108CF.8040707@redhat.com> Message-ID: Hi guys, I do not know whether it will reach ALL the lists Dmitri put in, but anyway: I do am interested heavily in getting a nice inter distro product (and if sth works both on RH-like and Deb-like distros that's quite some bases covered...) I'm afraid I'm not able to take the responsibility of building the deb support myself (no skills, no time), but feel like I do need it and I can spent some considerable time testing (I'm still having a production NIS around and I would like to test the interoperability when it stops being 'production'...) builds if they appear... I feel like IPA is getting the well established components and builds an added value ON them and not AGAINST them, making life easier (and hiding the not so beatiful guts under a nice interface, too...): Integrating KRB5 and LDAP is something people do every now and then, but it comes with cnsiderable pain of reading contradictory guides not updated for 10 years, dealing with examples using crypto mechanism that should be long forgotten... ('first, before configuring LDAP set up KRB5, having a test principal get back to this LDAP guide' and some two links away: 'first, get the your LDAP feet wet, when you're able to do ldapsearch get back and construct those ldifs to build krb5 database in ldap' followed by 'make a new realm, but don't use krb5_newrealm'...). Freeipa gives hope of NOT having to deal with cn=config manually, (it's a really nice thing, but ldifs are sth that should be hidden from view, and most guides for ldap/krb5 integration require creating LOTS of those 'by hand', which makes quite a steep learning curve...). The abundance of PAM modules for ldap/krb5 does not make it any easier (shishi? heimdall? MIT?; libpam-ldap or libpam-ldapd?), nor the multitude of different caching tools. (to mention only nslcd, nsscache, libpam-ccreds, nss_updatedb...). Having something solid to start with todays hordes of products requiring some auth integration thingie would be really nice OTOH that would be nice to have some documentation without EXAMPLE.COM inside :> I think getting freeipa working on Debian would be a great 'social' move, sure to be valued among the Linux community (ok, at least the part of community not centered on their own personal computers...), but the transition to 'Freeipa is wideely adopted product for ...' would surely need more people than a couple of guys in RH raising the Debian cause and a few Debian users like me. Thanks to work by Alexandre Ellert it's possible to get freeipa working with wheezy with relatively no hassle, but I'm afraid the world needs more than him :> Trying that I haven't seen any obvious 'fedorisms' inside... As for 'let's have a dream' part -> I would like to see sth similar to nsscache included with the freeipa suite for some really lightweight clients, for more than one reason... Dmitri, thanks for raising the flag! Micha? PS:Any idea for some advertisement on Debian side? On Fri, Aug 30, 2013 at 11:04 PM, Dmitri Pal wrote: > Hello, > > Sorry for cross posting to 4 different lists but it seems that this is > the best way to include most of people who might be interested in this > discussion. > > The question of "When FreeIPA will be available on Debian?" has been > coming up periodically on the list(s) without any resolution. However it > is clear that it would be beneficial for the community and the project. > > May be it is time to try again? > Let us see why it yet has not happened? > > 1) Some components need to be ported to Debian especially Dogtag and a > slew of its new RESTEasy dependencies. This requires time and quite an > effort from someone familiar with the domain. > 2) The code needs to be changed in installer and potentially in other > places as it might have had some Fedorizms blended in > 3) Someone needs to own packages in Debian and maintain them, someone > with good knowledge of the distro and time to take ownership of about 50 > packages. > > Can we pull it off together this time? > Say we plan for some Dogtag and IPA domain experts to work on the port > during Nov 13 - Feb 14 and address 1) and 2). Would there be any > interest to join forces with them? Would there be anyone to take on item > 3) from the list above? > > > -- > Thank you, > Dmitri Pal > > Sr. Engineering Manager for IdM portfolio > Red Hat Inc. > > > ------------------------------- > Looking to carve out IT costs? > www.redhat.com/carveoutcosts/ > > > > _______________________________________________ > Freeipa-users mailing list > Freeipa-users at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Michal Dwuznik From aborrero at cica.es Sat Aug 31 21:03:37 2013 From: aborrero at cica.es (Arturo Borrero Gonzalez) Date: Sat, 31 Aug 2013 23:03:37 +0200 Subject: [Freeipa-users] FreeIPA on Debian In-Reply-To: References: <522108CF.8040707@redhat.com> Message-ID: It's a nice idea to get FreeIPA on Debian. Let me point to some Debian resources related to FreeIPA: http://lists.alioth.debian.org/mailman/listinfo/pkg-freeipa-devel http://qa.debian.org/developer.php?login=pkg-freeipa-devel%40lists.alioth.debian.org I don't know who is behind pkg-freeipa-devel at lists.alioth.debian.org. I would recommend sending there an email, CC'ing debian-devel. I can maintain one or two Debian packages (but not 50) however i'm not an official Debian Developer. Best regards. -- Arturo Borrero Gonz?lez Departamento de Seguridad Inform?tica (nis at cica.es) Centro Informatico Cientifico de Andalucia (CICA) Avda. Reina Mercedes s/n - 41012 - Sevilla (Spain) Tfno.: +34 955 056 600 / FAX: +34 955 056 650 Consejer?a de Econom?a, Innovaci?n, Ciencia y Empleo Junta de Andaluc?a