[Freeipa-users] FreeIPA v4.2 stopped working, wants me to run ipa-server-upgrade, but has errors
Martin Basti
mbasti at redhat.com
Thu Oct 13 07:12:28 UTC 2016
Oh you are lucky to have ~150 replication conflicts :)
How did you get those? Did you run upgrade in parallel or did you have
some network issues?
You have to manually fix all replication conflicts and the re-run
ipa-server-upgrade
Please follow guide I posted previously, sorry :(
Martin
On 12.10.2016 21:30, John Popowitch wrote:
> I ran the following on each of my three servers:
> kinit admin
> ldapsearch -Y GSSAPI -b 'dc=aws,dc=cappex,dc=com' "nsds5ReplConflict=*" \* nsds5ReplConflict
> There are 49, 57, 49 entries returned by that query on the respective server.
> Here is the one related to 'System: Modify Certificate Profile' from the first server:
>
> # CA Administrator + c93bf230-a32311e5-b492895f-f9294e47, privileges, pbac, aws
> .cappex.com
> dn: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=priv
> ileges,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Add CA ACL+nsuniqueid=c93bf269-a32311e5-b492895f-f9294e47
> ,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Delete CA ACL+nsuniqueid=c93bf26d-a32311e5-b492895f-f9294
> e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Manage CA ACL Membership+nsuniqueid=c93bf271-a32311e5-b49
> 2895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Modify CA ACL+nsuniqueid=c93bf275-a32311e5-b492895f-f9294
> e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Delete Certificate Profile+nsuniqueid=c93bf27c-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Import Certificate Profile+nsuniqueid=c93bf280-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Modify Certificate Profile+nsuniqueid=c93bf284-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> objectClass: groupofnames
> objectClass: top
> objectClass: nestedgroup
> cn: CA Administrator
> description: CA Administrator
> nsds5ReplConflict: namingConflict cn=CA Administrator,cn=privileges,cn=pbac,dc
> =aws,dc=cappex,dc=com
>
>
> Here are the related entries from the second server:
>
> # CA Administrator + c93bf230-a32311e5-b492895f-f9294e47, privileges, pbac, aws
> .cappex.com
> dn: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=priv
> ileges,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Add CA ACL+nsuniqueid=c93bf269-a32311e5-b492895f-f9294e47
> ,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Delete CA ACL+nsuniqueid=c93bf26d-a32311e5-b492895f-f9294
> e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Manage CA ACL Membership+nsuniqueid=c93bf271-a32311e5-b49
> 2895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Modify CA ACL+nsuniqueid=c93bf275-a32311e5-b492895f-f9294
> e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Delete Certificate Profile+nsuniqueid=c93bf27c-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Import Certificate Profile+nsuniqueid=c93bf280-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Modify Certificate Profile+nsuniqueid=c93bf284-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> objectClass: groupofnames
> objectClass: top
> objectClass: nestedgroup
> cn: CA Administrator
> description: CA Administrator
> nsds5ReplConflict: namingConflict cn=ca administrator,cn=privileges,cn=pbac,dc
> =aws,dc=cappex,dc=com
>
> # System: Modify Certificate Profile + c93bf284-a32311e5-b492895f-f9294e47, per
> missions, pbac, aws.cappex.com
> dn: cn=System: Modify Certificate Profile+nsuniqueid=c93bf284-a32311e5-b492895
> f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> member: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=
> privileges,cn=pbac,dc=aws,dc=cappex,dc=com
> ipaPermTargetFilter: (objectclass=ipacertprofile)
> ipaPermRight: write
> ipaPermBindRuleType: permission
> ipaPermissionType: V2
> ipaPermissionType: MANAGED
> ipaPermissionType: SYSTEM
> cn: System: Modify Certificate Profile
> objectClass: ipapermission
> objectClass: top
> objectClass: groupofnames
> objectClass: ipapermissionv2
> ipaPermDefaultAttr: description
> ipaPermDefaultAttr: ipacertprofilestoreissued
> ipaPermDefaultAttr: cn
> ipaPermLocation: cn=certprofiles,cn=ca,dc=aws,dc=cappex,dc=com
> nsds5ReplConflict: namingConflict cn=system: modify certificate profile,cn=per
> missions,cn=pbac,dc=aws,dc=cappex,dc=com
>
>
> And from the third server:
>
> # CA Administrator + c93bf230-a32311e5-b492895f-f9294e47, privileges, pbac, aws
> .cappex.com
> dn: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=priv
> ileges,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Add CA ACL+nsuniqueid=c93bf269-a32311e5-b492895f-f9294e47
> ,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Delete CA ACL+nsuniqueid=c93bf26d-a32311e5-b492895f-f9294
> e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Manage CA ACL Membership+nsuniqueid=c93bf271-a32311e5-b49
> 2895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Modify CA ACL+nsuniqueid=c93bf275-a32311e5-b492895f-f9294
> e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Delete Certificate Profile+nsuniqueid=c93bf27c-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Import Certificate Profile+nsuniqueid=c93bf280-a32311e5-b
> 492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
> memberOf: cn=System: Modify Certificate Profile,cn=permissions,cn=pbac,dc=aws,
> dc=cappex,dc=com
> objectClass: groupofnames
> objectClass: top
> objectClass: nestedgroup
> cn: CA Administrator
> description: CA Administrator
> nsds5ReplConflict: namingConflict cn=CA Administrator,cn=privileges,cn=pbac,dc
> =aws,dc=cappex,dc=com
>
>
> Thank you for sending a link with more info on replication conflicts.
> I'm reading it now.
> -John
>
>
>
> -----Original Message-----
> From: Martin Basti [mailto:mbasti at redhat.com]
> Sent: Wednesday, October 12, 2016 5:46 AM
> To: John Popowitch; Alexander Bokovoy
> Cc: freeipa-users at redhat.com
> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to run ipa-server-upgrade, but has errors
>
>
>
> On 11.10.2016 22:01, John Popowitch wrote:
>> Ah, yes, thank you, Alexander.
>> I agree it would help if I followed the example better.
>> It would also help if I understood the example so a little description of what each command does would be very helpful.
> Sorry, we don't have time to explain everything here. `man ldapsearch` is your friend
>
>
>> It looks like that ACI record does exist.
>> Now how would I remove these LDAP records?
> I dig deeper into code, and actually this error is not caused by ACIs,
> because it even does not get there. I think that this may be caused by
> replication conflict on permission entry that caused the IPA doesn't see
> it but DS refuses to add it there.
>
> Can you please check as Directory Manager if there are any replication
> conflicts using this command?
> ldapsearch -D 'cn=directory manager' -W -b 'dc=aws,dc=cappex,dc=com'
> "nsds5ReplConflict=*" \* nsds5ReplConflict
>
> Please check if there is replication conflict on entry 'System: Modify
> Certificate Profile'
>
> More info about replication conflicts:
> https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
>>
>> -----Original Message-----
>> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
>> Sent: Tuesday, October 11, 2016 2:44 PM
>> To: John Popowitch
>> Cc: Martin Basti; freeipa-users at redhat.com
>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to run ipa-server-upgrade, but has errors
>>
>> On ti, 11 loka 2016, John Popowitch wrote:
>>> It doesn't look like there are any entries.
>>>
>>> # ldapsearch -x -b 'cn=certprofiles,cn=ca,dc=aws,dc=cappex,dc=com' -s
>>> base aci
>> 'ldapsearch -x' is 'use simple authentication instead of SASL' -- given that you didn't specify any identity for simple authentication, you are running an anonymous search. Martin asked you to 'kinit' as administrator and then use SASL GSSAPI.
>>
>> ACIs only available for retrieval to administrators. It is not a surprise that anonymous access does not see them.
>>
>> It would be good if you would have followed the example:
>>> Here you have example
>>>
>>> kinit admin
>>>
>>> ldapsearch -Y GSSAPI -b 'cn=certprofiles,cn=ca,dc=<your>,dc=<suffix>'
>>> -s base aci
>>>
>>> On 11.10.2016 17:48, John Popowitch wrote:
>>> Thanks, Martin.
>>> But I'm afraid you've gone beyond my level of LDAP knowledge.
>>> How would I check for that ACI?
>>> -John
>>>
>>> From: Martin Basti [mailto:mbasti at redhat.com]
>>> Sent: Tuesday, October 11, 2016 10:38 AM
>>> To: John Popowitch;
>>> freeipa-users at redhat.com<mailto:freeipa-users at redhat.com>
>>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to
>>> run ipa-server-upgrade, but has errors
>>>
>>>
>>>
>>>
>>> On 11.10.2016 17:21, John Popowitch wrote:
>>> I agree that is weird.
>>> Several of the other managed permissions are updated successfully and they are very similar.
>>> Yes, I can try to remove the permission manually.
>>> Is there any risk in corrupting or breaking the system?
>>> This is, I believe, one of three IPA servers in a multi-master replication.
>>> And we run our production website (basically our company) off of these servers.
>>> Assuming it's safe enough to do, could I delete that permission via the UI or does it need to be directly via LDAP?
>>>
>>> Upgrade will re-create permission.
>>>
>>> You have to directly using LDAP as Directory Manager
>>>
>>> Also please check in: cn=certprofiles,cn=ca,$SUFFIX
>>>
>>> if you have this ACI there
>>>
>>> aci: (targetattr = "cn || description ||
>>> ipacertprofilestoreissued")(targetfil
>>> ter = "(objectclass=ipacertprofile)")(version 3.0;acl
>>> "permission:System: Mod ify Certificate Profile";allow (write) groupdn
>>> = "ldap:///cn=System<ldap://cn=System>: Modify C ertificate
>>> Profile,cn=permissions,cn=pbac,dc=dom-058-017,dc=abc,dc=idm,dc=lab
>>> ,dc=eng,dc=brq,dc=redhat,dc=com";)
>>>
>>> This may also cause an issue, so if removing of permission itself did
>>> not help (or permission does not exist) you may need to remove this ACI
>>>
>>> Martin
>>>
>>>
>>>
>>>
>>> From: Martin Basti [mailto:mbasti at redhat.com]
>>> Sent: Tuesday, October 11, 2016 9:47 AM
>>> To: John Popowitch;
>>> freeipa-users at redhat.com<mailto:freeipa-users at redhat.com>
>>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to
>>> run ipa-server-upgrade, but has errors
>>>
>>>
>>> That's weird because the code is checking if a permission exists before
>>> it tries to add a new one
>>>
>>> Can you try to remove 'System: Modify Certificate Profile' manually from LDAP and re-run ipa-server-upgrade?
>>>
>>>
>>>
>>> On 11.10.2016 15:53, John Popowitch wrote:
>>> 2016-10-10T19:51:38Z DEBUG Updating managed permission: System: Modify
>>> Certificate Profile 2016-10-10T19:51:38Z DEBUG Destroyed connection
>>> context.ldap2_82077392 2016-10-10T19:51:38Z ERROR Upgrade failed with
>>> This entry already exists 2016-10-10T19:51:38Z DEBUG Traceback (most recent call last):
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 306, in __upgrade
>>> self.modified = (ld.update(self.files) or self.modified)
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 905, in update
>>> self._run_updates(all_updates)
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 877, in _run_updates
>>> self._run_update_plugin(update['plugin'])
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 852, in _run_update_plugin
>>> restart_ds, updates = self.api.Updater[plugin_name]()
>>> File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1400, in __call__
>>> return self.execute(**options)
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_managed_permissions.py", line 433, in execute
>>> anonymous_read_aci)
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_managed_permissions.py", line 529, in update_permission
>>> ldap.add_entry(entry)
>>> File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1428, in add_entry
>>> self.conn.add_s(str(entry.dn), attrs.items())
>>> File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
>>> self.gen.throw(type, value, traceback)
>>> File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 938, in error_handler
>>> raise errors.DuplicateEntry()
>>> DuplicateEntry: This entry already exists
>>>
>>> 2016-10-10T19:51:38Z DEBUG Traceback (most recent call last):
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 418, in start_creation
>>> run_step(full_msg, method)
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 408, in run_step
>>> method()
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 314, in __upgrade
>>> raise RuntimeError(e)
>>> RuntimeError: This entry already exists
>>>
>>> 2016-10-10T19:51:38Z DEBUG [error] RuntimeError: This entry already exists
>>> 2016-10-10T19:51:38Z DEBUG [cleanup]: stopping directory server
>>> 2016-10-10T19:51:38Z DEBUG Starting external process
>>> 2016-10-10T19:51:38Z DEBUG args='/bin/systemctl' 'stop' 'dirsrv at AWS-CAPPEX-COM.service<mailto:dirsrv at AWS-CAPPEX-COM.service>'
>>> 2016-10-10T19:51:40Z DEBUG Process finished, return code=0
>>> 2016-10-10T19:51:40Z DEBUG stdout= 2016-10-10T19:51:40Z DEBUG stderr=
>>> 2016-10-10T19:51:40Z DEBUG duration: 1 seconds
>>> 2016-10-10T19:51:40Z DEBUG [cleanup]: restoring configuration
>>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>>> 2016-10-10T19:51:40Z DEBUG duration: 0 seconds
>>> 2016-10-10T19:51:40Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
>>> 2016-10-10T19:51:40Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
>>> return_value = self.run()
>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
>>> raise admintool.ScriptError(str(e))
>>>
>>> 2016-10-10T19:51:40Z DEBUG The ipa-server-upgrade command failed,
>>> exception: ScriptError: ('IPA upgrade failed.', 1) 2016-10-10T19:51:40Z
>>> ERROR ('IPA upgrade failed.', 1)
>>>
>>>
>>>
>>> From: Martin Basti [mailto:mbasti at redhat.com]
>>> Sent: Tuesday, October 11, 2016 1:53 AM
>>> To: John Popowitch;
>>> freeipa-users at redhat.com<mailto:freeipa-users at redhat.com>
>>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to
>>> run ipa-server-upgrade, but has errors
>>>
>>>
>>>
>>>
>>> On 10.10.2016 23:30, John Popowitch wrote:
>>> Hello FreeIPA community.
>>> I've inherited a group of three FreeIPA v4.2 servers on CentOS 7.2.
>>> I had to reboot one of the servers and now IPA won't run saying, "Upgrade required: please run ipa-server-upgrade command."
>>> But when I run ipa-server-upgrade I get an error:
>>> ipa: ERROR: Upgrade failed with This entry already exists When I run it
>>> in debug mode the last action before the error is:
>>> ipa.ipaserver.install.plugins.update_managed_permissions.update_managed
>>> _permissions: DEBUG: Updating managed permission: System: Modify Certificate Profile It appears that several of the other managed permissions are processed successfully.
>>> When I look in the UI on one of the other servers it appears that this permission exists under IPA Server -> Role Based Access Control -> Permissions.
>>> I'm not familiar with FreeIPA so any help would be greatly appreciated.
>>> Thanks in advance.
>>> -John
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hello,
>>>
>>> can you post the related part of ipaupgrade.log here?
>>>
>>> Martin
>>>
>>>
>>>
>>> --
>>> Manage your subscription for the Freeipa-users mailing list:
>>> https://www.redhat.com/mailman/listinfo/freeipa-users
>>> Go to http://freeipa.org for more info on the project
More information about the Freeipa-users
mailing list