[Freeipa-users] FreeIPA v4.2 stopped working, wants me to run ipa-server-upgrade, but has errors
John Popowitch
jpopowitch at cappex.com
Wed Oct 12 19:30:11 UTC 2016
I ran the following on each of my three servers:
kinit admin
ldapsearch -Y GSSAPI -b 'dc=aws,dc=cappex,dc=com' "nsds5ReplConflict=*" \* nsds5ReplConflict
There are 49, 57, 49 entries returned by that query on the respective server.
Here is the one related to 'System: Modify Certificate Profile' from the first server:
# CA Administrator + c93bf230-a32311e5-b492895f-f9294e47, privileges, pbac, aws
.cappex.com
dn: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=priv
ileges,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Add CA ACL+nsuniqueid=c93bf269-a32311e5-b492895f-f9294e47
,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Delete CA ACL+nsuniqueid=c93bf26d-a32311e5-b492895f-f9294
e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Manage CA ACL Membership+nsuniqueid=c93bf271-a32311e5-b49
2895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Modify CA ACL+nsuniqueid=c93bf275-a32311e5-b492895f-f9294
e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Delete Certificate Profile+nsuniqueid=c93bf27c-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Import Certificate Profile+nsuniqueid=c93bf280-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Modify Certificate Profile+nsuniqueid=c93bf284-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
objectClass: groupofnames
objectClass: top
objectClass: nestedgroup
cn: CA Administrator
description: CA Administrator
nsds5ReplConflict: namingConflict cn=CA Administrator,cn=privileges,cn=pbac,dc
=aws,dc=cappex,dc=com
Here are the related entries from the second server:
# CA Administrator + c93bf230-a32311e5-b492895f-f9294e47, privileges, pbac, aws
.cappex.com
dn: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=priv
ileges,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Add CA ACL+nsuniqueid=c93bf269-a32311e5-b492895f-f9294e47
,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Delete CA ACL+nsuniqueid=c93bf26d-a32311e5-b492895f-f9294
e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Manage CA ACL Membership+nsuniqueid=c93bf271-a32311e5-b49
2895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Modify CA ACL+nsuniqueid=c93bf275-a32311e5-b492895f-f9294
e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Delete Certificate Profile+nsuniqueid=c93bf27c-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Import Certificate Profile+nsuniqueid=c93bf280-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Modify Certificate Profile+nsuniqueid=c93bf284-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
objectClass: groupofnames
objectClass: top
objectClass: nestedgroup
cn: CA Administrator
description: CA Administrator
nsds5ReplConflict: namingConflict cn=ca administrator,cn=privileges,cn=pbac,dc
=aws,dc=cappex,dc=com
# System: Modify Certificate Profile + c93bf284-a32311e5-b492895f-f9294e47, per
missions, pbac, aws.cappex.com
dn: cn=System: Modify Certificate Profile+nsuniqueid=c93bf284-a32311e5-b492895
f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
member: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=
privileges,cn=pbac,dc=aws,dc=cappex,dc=com
ipaPermTargetFilter: (objectclass=ipacertprofile)
ipaPermRight: write
ipaPermBindRuleType: permission
ipaPermissionType: V2
ipaPermissionType: MANAGED
ipaPermissionType: SYSTEM
cn: System: Modify Certificate Profile
objectClass: ipapermission
objectClass: top
objectClass: groupofnames
objectClass: ipapermissionv2
ipaPermDefaultAttr: description
ipaPermDefaultAttr: ipacertprofilestoreissued
ipaPermDefaultAttr: cn
ipaPermLocation: cn=certprofiles,cn=ca,dc=aws,dc=cappex,dc=com
nsds5ReplConflict: namingConflict cn=system: modify certificate profile,cn=per
missions,cn=pbac,dc=aws,dc=cappex,dc=com
And from the third server:
# CA Administrator + c93bf230-a32311e5-b492895f-f9294e47, privileges, pbac, aws
.cappex.com
dn: cn=CA Administrator+nsuniqueid=c93bf230-a32311e5-b492895f-f9294e47,cn=priv
ileges,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Add CA ACL+nsuniqueid=c93bf269-a32311e5-b492895f-f9294e47
,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Delete CA ACL+nsuniqueid=c93bf26d-a32311e5-b492895f-f9294
e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Manage CA ACL Membership+nsuniqueid=c93bf271-a32311e5-b49
2895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Modify CA ACL+nsuniqueid=c93bf275-a32311e5-b492895f-f9294
e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Delete Certificate Profile+nsuniqueid=c93bf27c-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Import Certificate Profile+nsuniqueid=c93bf280-a32311e5-b
492895f-f9294e47,cn=permissions,cn=pbac,dc=aws,dc=cappex,dc=com
memberOf: cn=System: Modify Certificate Profile,cn=permissions,cn=pbac,dc=aws,
dc=cappex,dc=com
objectClass: groupofnames
objectClass: top
objectClass: nestedgroup
cn: CA Administrator
description: CA Administrator
nsds5ReplConflict: namingConflict cn=CA Administrator,cn=privileges,cn=pbac,dc
=aws,dc=cappex,dc=com
Thank you for sending a link with more info on replication conflicts.
I'm reading it now.
-John
-----Original Message-----
From: Martin Basti [mailto:mbasti at redhat.com]
Sent: Wednesday, October 12, 2016 5:46 AM
To: John Popowitch; Alexander Bokovoy
Cc: freeipa-users at redhat.com
Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to run ipa-server-upgrade, but has errors
On 11.10.2016 22:01, John Popowitch wrote:
> Ah, yes, thank you, Alexander.
> I agree it would help if I followed the example better.
> It would also help if I understood the example so a little description of what each command does would be very helpful.
Sorry, we don't have time to explain everything here. `man ldapsearch` is your friend
> It looks like that ACI record does exist.
> Now how would I remove these LDAP records?
I dig deeper into code, and actually this error is not caused by ACIs,
because it even does not get there. I think that this may be caused by
replication conflict on permission entry that caused the IPA doesn't see
it but DS refuses to add it there.
Can you please check as Directory Manager if there are any replication
conflicts using this command?
ldapsearch -D 'cn=directory manager' -W -b 'dc=aws,dc=cappex,dc=com'
"nsds5ReplConflict=*" \* nsds5ReplConflict
Please check if there is replication conflict on entry 'System: Modify
Certificate Profile'
More info about replication conflicts:
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Managing_Replication-Solving_Common_Replication_Conflicts.html
>
>
> -----Original Message-----
> From: Alexander Bokovoy [mailto:abokovoy at redhat.com]
> Sent: Tuesday, October 11, 2016 2:44 PM
> To: John Popowitch
> Cc: Martin Basti; freeipa-users at redhat.com
> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to run ipa-server-upgrade, but has errors
>
> On ti, 11 loka 2016, John Popowitch wrote:
>> It doesn't look like there are any entries.
>>
>> # ldapsearch -x -b 'cn=certprofiles,cn=ca,dc=aws,dc=cappex,dc=com' -s
>> base aci
> 'ldapsearch -x' is 'use simple authentication instead of SASL' -- given that you didn't specify any identity for simple authentication, you are running an anonymous search. Martin asked you to 'kinit' as administrator and then use SASL GSSAPI.
>
> ACIs only available for retrieval to administrators. It is not a surprise that anonymous access does not see them.
>
> It would be good if you would have followed the example:
>> Here you have example
>>
>> kinit admin
>>
>> ldapsearch -Y GSSAPI -b 'cn=certprofiles,cn=ca,dc=<your>,dc=<suffix>'
>> -s base aci
>>
>> On 11.10.2016 17:48, John Popowitch wrote:
>> Thanks, Martin.
>> But I'm afraid you've gone beyond my level of LDAP knowledge.
>> How would I check for that ACI?
>> -John
>>
>> From: Martin Basti [mailto:mbasti at redhat.com]
>> Sent: Tuesday, October 11, 2016 10:38 AM
>> To: John Popowitch;
>> freeipa-users at redhat.com<mailto:freeipa-users at redhat.com>
>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to
>> run ipa-server-upgrade, but has errors
>>
>>
>>
>>
>> On 11.10.2016 17:21, John Popowitch wrote:
>> I agree that is weird.
>> Several of the other managed permissions are updated successfully and they are very similar.
>> Yes, I can try to remove the permission manually.
>> Is there any risk in corrupting or breaking the system?
>> This is, I believe, one of three IPA servers in a multi-master replication.
>> And we run our production website (basically our company) off of these servers.
>> Assuming it's safe enough to do, could I delete that permission via the UI or does it need to be directly via LDAP?
>>
>> Upgrade will re-create permission.
>>
>> You have to directly using LDAP as Directory Manager
>>
>> Also please check in: cn=certprofiles,cn=ca,$SUFFIX
>>
>> if you have this ACI there
>>
>> aci: (targetattr = "cn || description ||
>> ipacertprofilestoreissued")(targetfil
>> ter = "(objectclass=ipacertprofile)")(version 3.0;acl
>> "permission:System: Mod ify Certificate Profile";allow (write) groupdn
>> = "ldap:///cn=System<ldap://cn=System>: Modify C ertificate
>> Profile,cn=permissions,cn=pbac,dc=dom-058-017,dc=abc,dc=idm,dc=lab
>> ,dc=eng,dc=brq,dc=redhat,dc=com";)
>>
>> This may also cause an issue, so if removing of permission itself did
>> not help (or permission does not exist) you may need to remove this ACI
>>
>> Martin
>>
>>
>>
>>
>> From: Martin Basti [mailto:mbasti at redhat.com]
>> Sent: Tuesday, October 11, 2016 9:47 AM
>> To: John Popowitch;
>> freeipa-users at redhat.com<mailto:freeipa-users at redhat.com>
>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to
>> run ipa-server-upgrade, but has errors
>>
>>
>> That's weird because the code is checking if a permission exists before
>> it tries to add a new one
>>
>> Can you try to remove 'System: Modify Certificate Profile' manually from LDAP and re-run ipa-server-upgrade?
>>
>>
>>
>> On 11.10.2016 15:53, John Popowitch wrote:
>> 2016-10-10T19:51:38Z DEBUG Updating managed permission: System: Modify
>> Certificate Profile 2016-10-10T19:51:38Z DEBUG Destroyed connection
>> context.ldap2_82077392 2016-10-10T19:51:38Z ERROR Upgrade failed with
>> This entry already exists 2016-10-10T19:51:38Z DEBUG Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 306, in __upgrade
>> self.modified = (ld.update(self.files) or self.modified)
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 905, in update
>> self._run_updates(all_updates)
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 877, in _run_updates
>> self._run_update_plugin(update['plugin'])
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 852, in _run_update_plugin
>> restart_ds, updates = self.api.Updater[plugin_name]()
>> File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 1400, in __call__
>> return self.execute(**options)
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_managed_permissions.py", line 433, in execute
>> anonymous_read_aci)
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/plugins/update_managed_permissions.py", line 529, in update_permission
>> ldap.add_entry(entry)
>> File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1428, in add_entry
>> self.conn.add_s(str(entry.dn), attrs.items())
>> File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
>> self.gen.throw(type, value, traceback)
>> File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 938, in error_handler
>> raise errors.DuplicateEntry()
>> DuplicateEntry: This entry already exists
>>
>> 2016-10-10T19:51:38Z DEBUG Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 418, in start_creation
>> run_step(full_msg, method)
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 408, in run_step
>> method()
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 314, in __upgrade
>> raise RuntimeError(e)
>> RuntimeError: This entry already exists
>>
>> 2016-10-10T19:51:38Z DEBUG [error] RuntimeError: This entry already exists
>> 2016-10-10T19:51:38Z DEBUG [cleanup]: stopping directory server
>> 2016-10-10T19:51:38Z DEBUG Starting external process
>> 2016-10-10T19:51:38Z DEBUG args='/bin/systemctl' 'stop' 'dirsrv at AWS-CAPPEX-COM.service<mailto:dirsrv at AWS-CAPPEX-COM.service>'
>> 2016-10-10T19:51:40Z DEBUG Process finished, return code=0
>> 2016-10-10T19:51:40Z DEBUG stdout= 2016-10-10T19:51:40Z DEBUG stderr=
>> 2016-10-10T19:51:40Z DEBUG duration: 1 seconds
>> 2016-10-10T19:51:40Z DEBUG [cleanup]: restoring configuration
>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2016-10-10T19:51:40Z DEBUG duration: 0 seconds
>> 2016-10-10T19:51:40Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
>> 2016-10-10T19:51:40Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
>> return_value = self.run()
>> File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
>> raise admintool.ScriptError(str(e))
>>
>> 2016-10-10T19:51:40Z DEBUG The ipa-server-upgrade command failed,
>> exception: ScriptError: ('IPA upgrade failed.', 1) 2016-10-10T19:51:40Z
>> ERROR ('IPA upgrade failed.', 1)
>>
>>
>>
>> From: Martin Basti [mailto:mbasti at redhat.com]
>> Sent: Tuesday, October 11, 2016 1:53 AM
>> To: John Popowitch;
>> freeipa-users at redhat.com<mailto:freeipa-users at redhat.com>
>> Subject: Re: [Freeipa-users] FreeIPA v4.2 stopped working, wants me to
>> run ipa-server-upgrade, but has errors
>>
>>
>>
>>
>> On 10.10.2016 23:30, John Popowitch wrote:
>> Hello FreeIPA community.
>> I've inherited a group of three FreeIPA v4.2 servers on CentOS 7.2.
>> I had to reboot one of the servers and now IPA won't run saying, "Upgrade required: please run ipa-server-upgrade command."
>> But when I run ipa-server-upgrade I get an error:
>> ipa: ERROR: Upgrade failed with This entry already exists When I run it
>> in debug mode the last action before the error is:
>> ipa.ipaserver.install.plugins.update_managed_permissions.update_managed
>> _permissions: DEBUG: Updating managed permission: System: Modify Certificate Profile It appears that several of the other managed permissions are processed successfully.
>> When I look in the UI on one of the other servers it appears that this permission exists under IPA Server -> Role Based Access Control -> Permissions.
>> I'm not familiar with FreeIPA so any help would be greatly appreciated.
>> Thanks in advance.
>> -John
>>
>>
>>
>>
>>
>>
>>
>> Hello,
>>
>> can you post the related part of ipaupgrade.log here?
>>
>> Martin
>>
>>
>>
>> --
>> Manage your subscription for the Freeipa-users mailing list:
>> https://www.redhat.com/mailman/listinfo/freeipa-users
>> Go to http://freeipa.org for more info on the project
>
More information about the Freeipa-users
mailing list