[Freeipa-devel] disabling topology segment has no effect
Oleg Fayans
ofayans at redhat.com
Thu Jun 18 08:39:39 UTC 2015
Hi Ludwig,
I've saved and analyzed all the console outputs from my activity on
master and replica1 (from consoles that I by chance did not close). I
was not able to detect the moment when the things went wrong. My guess
is that the changes in topology get replicated slowly enough to be able
to introduce 2 contradictory changes on different nodes, that leads to
infrastructure inconsistency. It could also be that at some point I made
topology changes having one of the nodes turned off, which could have
lead to temporary loss in infrastructure integrity.
Right now I have 4 nodes, one of which has incorrect topology data. As a
result any changes that are made on this node do not get replicated on
others, while changes made on other nodes do get replicated to this
first one.
Today I'll try to reproduce this on fresh VMs with the fresh code.
On 06/17/2015 05:53 PM, Ludwig Krispenz wrote:
>
> On 06/17/2015 05:43 PM, Oleg Fayans wrote:
>>
>>
>> On 06/17/2015 05:34 PM, Ludwig Krispenz wrote:
>>>
>>> On 06/17/2015 05:26 PM, Oleg Fayans wrote:
>>>> Hi Ludwig,
>>>>
>>>> On 06/17/2015 05:13 PM, Ludwig Krispenz wrote:
>>>>> Hi,
>>>>> On 06/17/2015 05:07 PM, Oleg Fayans wrote:
>>>>>>
>>>>>>
>>>>>> On 06/17/2015 04:59 PM, Ludwig Krispenz wrote:
>>>>>>>
>>>>>>> On 06/17/2015 04:46 PM, Oleg Fayans wrote:
>>>>>>>> Hi Ludwig,
>>>>>>>>
>>>>>>>> On 06/17/2015 04:15 PM, Ludwig Krispenz wrote:
>>>>>>>>>
>>>>>>>>> On 06/17/2015 03:37 PM, Oleg Fayans wrote:
>>>>>>>>>> Hi Ludwig, Petr,
>>>>>>>>>>
>>>>>>>>>> Presently I have noticed that disabling a segment, using `ipa
>>>>>>>>>> topologysegment-mod realm replica1-to-replica2
>>>>>>>>>> --enabled=off` does not have effect on the way the data is
>>>>>>>>>> replicated.
>>>>>>>>>>
>>>>>>>>>> I mean that if we have the following tolopogy:
>>>>>>>>>> master <-> replica1 <-> replica2
>>>>>>>>> on which server did you apply the mod ?
>>>>>>>> On master.
>>>>>>> just to be clear, you have master <-> replica1 <-> replica2
>>>>>>> on master you disable replica1-replica2
>>>>>>> why would you expect mods on master not to be replicated ? at
>>>>>>> least to replica1 ?
>>>>>>> the disable should only effect the connection between r1 and r2.
>>>>>>> There is one problem in this linear topology, the disable
>>>>>>> reaches r1, it disables the agmt to r2 and so fails to
>>>>>>> replicate the disable to r2.
>>>>>>
>>>>>> To be precise, my topology is as follows
>>>>>>
>>>>>> master <-> replica3 <-> replica2 <-> replica1
>>>>>> And I disabled the replica3 <-> replica2. So I expected the
>>>>>> changes on master to be only visible on master and replica3, but
>>>>>> actually it kept replicating to all nodes.
>>>>>>
>>>>>> root at f22replica1:/home/ofayans]$ ipa topologysegment-find realm
>>>>>> ------------------
>>>>>> 3 segments matched
>>>>>> ------------------
>>>>>> Segment name: f22master.bagam.net-to-f22replica3.bagam.net
>>>>>> Left node: f22master.bagam.net
>>>>>> Right node: f22replica3.bagam.net
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name: replica1-to-replica2
>>>>>> Left node: f22replica1.bagam.net
>>>>>> Right node: f22replica2.bagam.net
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name: replica3-to-replica2
>>>>>> Left node: f22replica3.bagam.net
>>>>>> Right node: f22replica2.bagam.net
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 3
>>>>>> ----------------------------
>>>>>> root at f22replica1:/home/ofayans]$ ipa topologysegment-show realm
>>>>>> replica3-to-replica2
>>>>>> Segment name: replica3-to-replica2
>>>>>> Left node: f22replica3.bagam.net
>>>>>> Right node: f22replica2.bagam.net
>>>>>> Connectivity: both
>>>>>> Replication agreement enabled: off
>>>>> can you do a ldapsearch on cn=realm,cn=topology, ......
>>>> $ ldapsearch -LLL -b
>>>> "cn=realm,cn=topology,cn=ipa,cn=etc,dc=bagam,dc=net" -D
>>>> "cn=Directory Manager" -w '<password>'
>>>> dn: cn=realm,cn=topology,cn=ipa,cn=etc,dc=bagam,dc=net
>>>> cn: realm
>>>> ipaReplTopoConfRoot: dc=bagam,dc=net
>>>> objectClass: top
>>>> objectClass: iparepltopoconf
>>>>
>>>> dn:
>>>> cn=replica1-to-replica2,cn=realm,cn=topology,cn=ipa,cn=etc,dc=bagam,dc=net
>>>> ipaReplTopoSegmentRightNode: f22replica2.bagam.net
>>>> ipaReplTopoSegmentDirection: both
>>>> cn: replica1-to-replica2
>>>> ipaReplTopoSegmentLeftNode: f22replica1.bagam.net
>>>> objectClass: iparepltoposegment
>>>> objectClass: top
>>> replica1 - replica2
>>>>
>>>> dn:
>>>> cn=f22master.bagam.net-to-f22replica3.bagam.net,cn=realm,cn=topology,cn=ip
>>>> a,cn=etc,dc=bagam,dc=net
>>>> ipaReplTopoSegmentDirection: both
>>>> objectClass: iparepltoposegment
>>>> objectClass: top
>>>> cn: f22master.bagam.net-to-f22replica3.bagam.net
>>>> ipaReplTopoSegmentLeftNode: f22master.bagam.net
>>>> ipaReplTopoSegmentRightNode: f22replica3.bagam.net
>>>> ipaReplTopoSegmentStatus: autogen
>>> master - replica3
>>>>
>>>> dn:
>>>> cn=f22replica3.bagam.net-f22replica1.bagam.net,cn=realm,cn=topology,cn=ipa
>>>> ,cn=etc,dc=bagam,dc=net
>>>> objectClass: iparepltoposegment
>>>> objectClass: top
>>>> ipaReplTopoSegmentLeftNode: f22replica3.bagam.net
>>>> cn: f22replica3.bagam.net-f22replica1.bagam.net
>>>> ipaReplTopoSegmentDirection: both
>>>> ipaReplTopoSegmentRightNode: f22replica1.bagam.net
>>> replica3 - replica1
>>> but this does not match your segment-find output, there is no
>>> segment replica2 - replica3
>> You know what, this is because I did ldapsearch on replica3, while I
>> posted the results of topologysegment-find run on replica1.
>> But this means that there is a breakage in the replication between
>> replica1 and the rest of topology (the result of topologysegment-find
>> is the same across master-replica2-replica3 and different on replica1)
> the replication agreements on r3 match the output of the cn=realm
> search, saying you have a topology
> master <--> r3 <--> r1 <--> r2.
>
> could it be that you made changes while the segment was (partially)
> disabled. We would need the full history of topology changes
>>
>>
>>>>
>>>>>
>>>>> and on replica3 do a search -b "cn=config"
>>>>> "objectclass=nsds5replicationagreement"
>>>> $ ldapsearch -LLL -b "cn=config"
>>>> "objectclass=nsds5replicationagreement" -D "cn=Directory Manager"
>>>> -w '<password>'
>>>> dn:
>>>> cn=f22replica3.bagam.net-to-f22replica1.bagam.net,cn=replica,cn=dc\3Dbagam
>>>> \2Cdc\3Dnet,cn=mapping tree,cn=config
>>>> objectClass: nsds5replicationagreement
>>>> objectClass: ipaReplTopoManagedAgreement
>>>> objectClass: top
>>>> cn: f22replica3.bagam.net-to-f22replica1.bagam.net
>>>> nsDS5ReplicaHost: f22replica1.bagam.net
>>>> nsDS5ReplicaPort: 389
>>>> nsds5replicaTimeout: 300
>>>> nsDS5ReplicaRoot: dc=bagam,dc=net
>>>> description: f22replica3.bagam.net to f22replica1.bagam.net
>>>> ipaReplTopoManagedAgreementState: managed agreement - generated by
>>>> topology pl
>>>> ugin
>>>> nsDS5ReplicaTransportInfo: LDAP
>>>> nsDS5ReplicaBindMethod: SASL/GSSAPI
>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
>>>> idnssoaserial
>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
>>>> nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
>>>> internalModifiersName in
>>>> ternalModifyTimestamp
>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE
>>>> entryusn krblasts
>>>> uccessfulauth krblastfailedauth krbloginfailedcount
>>>> nsds5replicareapactive: 0
>>>> nsds5replicaLastUpdateStart: 20150617151930Z
>>>> nsds5replicaLastUpdateEnd: 20150617151930Z
>>>> nsds5replicaChangesSentSinceStartup:: Njo1LzMyOSA0OjcvMCA=
>>>> nsds5replicaLastUpdateStatus: 0 Replica acquired successfully:
>>>> Incremental upd
>>>> ate succeeded
>>>> nsds5replicaUpdateInProgress: FALSE
>>>> nsds5replicaLastInitStart: 19700101000000Z
>>>> nsds5replicaLastInitEnd: 19700101000000Z
>>>>
>>>> dn:
>>>> cn=meTof22master.bagam.net,cn=replica,cn=dc\3Dbagam\2Cdc\3Dnet,cn=mapping
>>>> tree,cn=config
>>>> cn: meTof22master.bagam.net
>>>> description: me to f22master.bagam.net
>>>> ipaReplTopoManagedAgreementState: managed agreement - controlled by
>>>> topology p
>>>> lugin
>>>> nsDS5ReplicaBindMethod: SASL/GSSAPI
>>>> nsDS5ReplicaHost: f22master.bagam.net
>>>> nsDS5ReplicaPort: 389
>>>> nsDS5ReplicaRoot: dc=bagam,dc=net
>>>> nsDS5ReplicaTransportInfo: LDAP
>>>> nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
>>>> idnssoaserial
>>>> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
>>>> nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE
>>>> entryusn krblasts
>>>> uccessfulauth krblastfailedauth krbloginfailedcount
>>>> nsds50ruv: {replicageneration} 557fdff1000000040000
>>>> nsds50ruv: {replica 4 ldap://f22master.bagam.net:389}
>>>> 557fdffc000100040000 558
>>>> 00f44000300040000
>>>> nsds50ruv: {replica 6 ldap://f22replica3.bagam.net:389}
>>>> 55800e1b000000060000 5
>>>> 5800f44000400060000
>>>> nsds50ruv: {replica 5 ldap://f22replica2.bagam.net:389}
>>>> 557fed70000000050000 5
>>>> 5800553000300050000
>>>> nsds50ruv: {replica 3 ldap://f22replica1.bagam.net:389}
>>>> 557fdffa000000030000 5
>>>> 58009b4000200030000
>>>> nsds5ReplicaStripAttrs: modifiersName modifyTimestamp
>>>> internalModifiersName in
>>>> ternalModifyTimestamp
>>>> nsds5replicaTimeout: 120
>>>> nsruvReplicaLastModified: {replica 4
>>>> ldap://f22master.bagam.net:389} 00000000
>>>> nsruvReplicaLastModified: {replica 6
>>>> ldap://f22replica3.bagam.net:389} 0000000
>>>> 0
>>>> nsruvReplicaLastModified: {replica 5
>>>> ldap://f22replica2.bagam.net:389} 0000000
>>>> 0
>>>> nsruvReplicaLastModified: {replica 3
>>>> ldap://f22replica1.bagam.net:389} 0000000
>>>> 0
>>>> objectClass: nsds5replicationagreement
>>>> objectClass: top
>>>> objectClass: ipaReplTopoManagedAgreement
>>>> nsds5replicareapactive: 0
>>>> nsds5replicaLastUpdateStart: 20150617151930Z
>>>> nsds5replicaLastUpdateEnd: 20150617151930Z
>>>> nsds5replicaChangesSentSinceStartup:: Njo1LzMzNCA=
>>>> nsds5replicaLastUpdateStatus: 0 Replica acquired successfully:
>>>> Incremental upd
>>>> ate succeeded
>>>> nsds5replicaUpdateInProgress: FALSE
>>>> nsds5replicaLastInitStart: 19700101000000Z
>>>> nsds5replicaLastInitEnd: 19700101000000Z
>>>>
>>>> dn:
>>>> cn=cloneAgreement1-f22replica3.bagam.net-pki-tomcat,cn=replica,cn=o\3Dipac
>>>> a,cn=mapping tree,cn=config
>>>> cn: cloneAgreement1-f22replica3.bagam.net-pki-tomcat
>>>> description: cloneAgreement1-f22replica3.bagam.net-pki-tomcat
>>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>>> masterAgreement1-f22replica3.bagam.
>>>> net-pki-tomcat,ou=csusers,cn=config
>>>> nsDS5ReplicaBindMethod: Simple
>>>> nsDS5ReplicaCredentials:
>>>> {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG
>>>> RERBNEJDUTRZbVk0TUdFM1l5MHpZV1F4TTJFeg0KTnkwNE5HVXhNamczTmkxak1qSmtNalkwTndBQ
>>>>
>>>> 0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQmxGYWZ1U3ROY2pNbV
>>>>
>>>> J4NFNUc2pBcQ==}j+d3WWGnksSdSnVQ2S0irQ==
>>>> nsDS5ReplicaHost: f22master.bagam.net
>>>> nsDS5ReplicaPort: 389
>>>> nsDS5ReplicaRoot: o=ipaca
>>>> nsDS5ReplicaTransportInfo: TLS
>>>> nsds50ruv: {replicageneration} 557fe04c000000600000
>>>> nsds50ruv: {replica 96 ldap://f22master.bagam.net:389}
>>>> 557fe05b000000600000 55
>>>> 800ea7000000600000
>>>> nsds50ruv: {replica 86 ldap://f22replica3.bagam.net:389}
>>>> 55800eb4000000560000
>>>> 55800eb6000200560000
>>>> nsds50ruv: {replica 91 ldap://f22replica2.bagam.net:389}
>>>> 557fede80000005b0000
>>>> 557fedea0002005b0000
>>>> nsds50ruv: {replica 97 ldap://f22replica1.bagam.net:389}
>>>> 557fe06c000000610000
>>>> 557fe326000000610000
>>>> nsruvReplicaLastModified: {replica 96
>>>> ldap://f22master.bagam.net:389} 00000000
>>>> nsruvReplicaLastModified: {replica 86
>>>> ldap://f22replica3.bagam.net:389} 000000
>>>> 00
>>>> nsruvReplicaLastModified: {replica 91
>>>> ldap://f22replica2.bagam.net:389} 000000
>>>> 00
>>>> nsruvReplicaLastModified: {replica 97
>>>> ldap://f22replica1.bagam.net:389} 000000
>>>> 00
>>>> objectClass: top
>>>> objectClass: nsds5replicationagreement
>>>> nsds5replicareapactive: 0
>>>> nsds5replicaLastUpdateStart: 20150617150850Z
>>>> nsds5replicaLastUpdateEnd: 20150617150850Z
>>>> nsds5replicaChangesSentSinceStartup:
>>>> nsds5replicaLastUpdateStatus: 0 Replica acquired successfully:
>>>> Incremental upd
>>>> ate succeeded
>>>> nsds5replicaUpdateInProgress: FALSE
>>>> nsds5replicaLastInitStart: 19700101000000Z
>>>> nsds5replicaLastInitEnd: 19700101000000Z
>>>>
>>>>>
>>>>> would like to see the raw data.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> It reproduces though even in a situation with the topology
>>>>>>>> replica3 <-> master <-> replica1 <-> replica2 and you disable
>>>>>>>> the replica1-replica2 segment on replica3 (quite expectedly)
>>>>>>>>>> and disable one of the segments, one would expect the changes
>>>>>>>>>> implemented on master would not be replicated to other nodes
>>>>>>>>>> (or do I misunderstand the concept of disabling a segment?).
>>>>>>>>>> However, in reality any changes in master do get replicated
>>>>>>>>>> despite the segment is disabled.
>>>>>>>>>>
>>>>>>>>>> Is it a correct behavior?
>>>>>>>>>>
>>>>>>>>>> The second question is: if disabled segments should not let
>>>>>>>>>> the changes through, then we probably should implement a
>>>>>>>>>> check for topology disconnection in similar way as `ipa
>>>>>>>>>> topologysegment-del` does. I mean, whenever a user tries to
>>>>>>>>>> disable a segment, the plugin should probably check whether
>>>>>>>>>> it disconnects any of the nodes.
>>>>>>>>> well, I think disabling should be temporary, you want to
>>>>>>>>> disconnect for some time. eg for debugging, not deleting the
>>>>>>>>> agreement completely, I would allow this.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.
More information about the Freeipa-devel
mailing list