<div dir="ltr"><div class="gmail_default" style="color:#666666">on ld4ipa01, I removed it with ipa-server-install --uninstall</div><div class="gmail_default" style="color:#666666"><br></div><div class="gmail_default" style="color:#666666">this was an attempt to recreate the replica from nyc02ipa02</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 23, 2017 at 3:17 AM, Martin Basti <span dir="ltr"><<a href="mailto:mbasti@redhat.com" target="_blank">mbasti@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<br>
<div class="m_-6106124880320625902moz-cite-prefix">On 22.02.2017 23:26, Aaron Young wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" style="color:rgb(102,102,102)">Hello
Everyone</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">I
recently lost the master master IPA server setup by the
previous administrator. </div>
<div class="gmail_default" style="color:rgb(102,102,102)">As it
stands now, if I try to add a new client, in order to standup
a new replica, I get errors while trying to setup DNS. This
led me to look at how authentication worked (I'm new to IPA)
and I learned about the kerberos tools</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">I
don't know if I'm familiar enough with the terminology to
adequately describe what I'm experiencing, so I'll give you
some of the commands and their results</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">but
first, a bit on the design</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">before
I got to this, we had</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">a
<-> b <-> c <-> d</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">b was
the master master</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">a,
happened to point to two test servers nyc02ipa01 and
nyc02ipa02 (not pictured, I discovered them later when c and d
started having problems)</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">a -
nyc01ipa02</div>
<div class="gmail_default" style="color:rgb(102,102,102)">b -
nyc01ipa01</div>
<div class="gmail_default" style="color:rgb(102,102,102)">c -
ld4ipa01</div>
<div class="gmail_default" style="color:rgb(102,102,102)">d -
ld4ipa02</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">currently,
I have nyc02ipa02 <-> nyc01ipa02<br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)"> </div>
<div class="gmail_default" style="color:rgb(102,102,102)">the
reason I have it limited like this is because all the other
servers stopped replicating for one reason or another (mainly
that they can't authenticate or in one case, there was a
database record corruption)</div>
<div class="gmail_default" style="color:rgb(102,102,102)"> </div>
<div class="gmail_default" style="color:rgb(102,102,102)">Anyway,
here are some activities and logs from the latest round of
fixes and information activities I've been engaging in</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">22:54:32
root@nyc01ipa02:~# kinit admin<br>
kinit: Clients credentials have been revoked while getting
initial credentials</p>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">Reading
through <a href="http://web.mit.edu/Kerberos/krb5-1.13/doc/admin/lockout.html" class="m_-6106124880320625902external-link" rel="nofollow" style="color:rgb(53,114,176);text-decoration:none" target="_blank">this</a> tells
me that</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px;font-family:arial,sans-serif;font-size:14px">
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace"># kadmin: modprinc -unlock PRINCNAME</pre>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">will
unlock an account...but if I can't get in....</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px;font-family:arial,sans-serif;font-size:14px">
<p style="margin:0px;padding:0px">22:54:37
root@nyc01ipa02:~# kadmin<br>
Authenticating as principal root/admin@MF with password.<br>
kadmin: Client 'root/admin@MF' not found in Kerberos
database while initializing kadmin interface</p>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">on
ld4ipa02, did a</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px;font-family:arial,sans-serif;font-size:14px">
<p style="margin:0px;padding:0px"># ipa-client-install
--uninstall </p>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">then </p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px;font-family:arial,sans-serif;font-size:14px">
<p style="margin:0px;padding:0px"># ipa-client-install
--force-join --enable-dns-updates --permit -f
--ssh-trust-dns --request-cert --automount-location=LD4
--enable-dns-updates</p>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">DNS
did not update, here is the relevant portion from
/var/log/ipaclient-install.log</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px;font-family:arial,sans-serif;font-size:14px">
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">2017-02-20T18:46:49Z DEBUG Writing nsupdate commands to /etc/ipa/.dns_update.txt:
2017-02-20T18:46:49Z DEBUG debug</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">update delete ld4ipa02.mf. IN A
show
send</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">update delete ld4ipa02.mf. IN AAAA
show
send</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">update add ld4ipa02.mf. 1200 IN A 10.102.100.140
show
send</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">2017-02-20T18:46:49Z DEBUG Starting external process
2017-02-20T18:46:49Z DEBUG args=/usr/bin/nsupdate -g /etc/ipa/.dns_update.txt
2017-02-20T18:46:49Z DEBUG Process finished, return code=1
2017-02-20T18:46:49Z DEBUG stdout=Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id: 0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
ld4ipa02.mf. 0 ANY A</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">2017-02-20T18:46:49Z DEBUG stderr=Reply from SOA query:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34702
;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;ld4ipa02.mf. IN SOA</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">;; AUTHORITY SECTION:
mf. 1800 IN SOA ld4ipa01.mf. hostmaster.mf. 1487615509 3600 900 1209600 3600</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">Found zone name: mf
The master is: ld4ipa01.mf
start_gssrequest
tkey query failed: GSSAPI error: Major = Unspecified GSS failure. Minor code may provide more information, Minor = Server DNS/ld4ipa01.mf@MF not found in Kerberos database.</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace">2017-02-20T18:46:49Z DEBUG nsupdate failed: Command '/usr/bin/nsupdate -g /etc/ipa/.dns_update.txt' returned non-zero exit status 1
2017-02-20T18:46:49Z ERROR Failed to update DNS records.
2017-02-20T18:46:49Z DEBUG DNS resolver: Query: ld4ipa02.mf IN A
2017-02-20T18:46:49Z DEBUG DNS resolver: No record.
2017-02-20T18:46:49Z DEBUG DNS resolver: Query: ld4ipa02.mf IN AAAA
2017-02-20T18:46:49Z DEBUG DNS resolver: No record.
2017-02-20T18:46:49Z DEBUG DNS resolver: Query: <a href="http://140.100.102.10.in-addr.arpa/" class="m_-6106124880320625902external-link" rel="nofollow" style="color:rgb(53,114,176);text-decoration:none" target="_blank">140.100.102.10.in-addr.arpa</a>. IN PTR
2017-02-20T18:46:49Z DEBUG DNS resolver: No record.
2017-02-20T18:46:49Z WARNING Missing A/AAAA record(s) for host ld4ipa02.mf: 10.102.100.140.
2017-02-20T18:46:49Z WARNING Missing reverse record(s) for address(es): 10.102.100.140.</pre>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">Why
isn't there an entry for "DNS/ld4ipa01.mf@MF" in the
Kerberos database?</p>
<p style="margin:10px 0px 0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">klist
-ktK /etc/dirsrv/ds.keytab on ld4ipa01 returns</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px;font-family:arial,sans-serif;font-size:14px">
<p style="margin:0px;padding:0px">Keytab name: <a href="http://file/etc/dirsrv/ds.keytab" class="m_-6106124880320625902external-link" rel="nofollow" style="color:rgb(53,114,176);text-decoration:none" target="_blank">FILE:/etc/dirsrv/ds.<wbr>keytab</a><br>
KVNO Timestamp Principal<br>
---- -------------------
------------------------------<wbr>------------------------<br>
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(<wbr>0x696a502bc73d209acdd36c42242f<wbr>7f8aff9dbba1073b34ea018ed3bd9c<wbr>dfd970)<br>
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(<wbr>0xe031464b6948ea34f4291d40fca7<wbr>a21e)<br>
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(<wbr>0xe94a1c98fe79b6317901435d9e9e<wbr>0257cefe438ff2ec527f)<br>
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(<wbr>0x6aaf4c7fa6b51b9de032b7c64283<wbr>07b5)<br>
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(<wbr>0x5e0702f44aef9e0633e09eede7ca<wbr>8041)<br>
2 11/17/2016 20:38:39 ldap/ld4ipa01.mf@MF
(<wbr>0x6e3a9d29ee3f129a156ae6228ab7<wbr>728df8ce5de923a61eba6a2e7802b8<wbr>d230b6)</p>
</blockquote>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)">
<div id="m_-6106124880320625902gmail-main-content" class="m_-6106124880320625902gmail-wiki-content" style="margin:0px;padding:0px;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">
<p style="margin:10px 0px 0px;padding:0px">Tried to test
connectivity using ldapsearch found that I could connect
to other hosts on 389 but not 636</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px">
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace"># ldapsearch -H <a rel="nofollow" style="color:rgb(53,114,176);text-decoration:none">ldap://nyc02ipa02:389</a> -D "cn=directory manager" -W -b "" -s base</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace"># ldapsearch -H <a rel="nofollow" style="color:rgb(53,114,176);text-decoration:none">ldaps://nyc02ipa02:</a>686 -D "cn=directory manager" -W -b "" -s base</pre>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px">Testing the kvno</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px">
<p style="margin:0px;padding:0px">02:10:00
root@ld4ipa01:~# kvno DNS/ld4ipa01.mf@MF<br>
DNS/ld4ipa01.mf@MF: kvno = 2</p>
<p style="margin:10px 0px 0px;padding:0px">02:10:52
root@ld4ipa02:~# kvno DNS/ld4ipa01.mf@MF<br>
kvno: Server DNS/ld4ipa01.mf@MF not found in Kerberos
database while getting credentials for
DNS/ld4ipa01.mf@MF</p>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px">Add this to any
command line to get debug on kerberos commands</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px">
<p style="margin:0px;padding:0px">KRB5_TRACE=/dev/stdout
kvno DNS/ld4ipa01.mf@MF</p>
</blockquote>
<p style="margin:10px 0px 0px;padding:0px">So, looking at
the debug<br>
kvno from ld4ipa02, does not return tickets. It does this
because it contacts the KDC which is nyc02ipa02, and
nyc02ipa02 does not recognize ldipa02 as an IPA server. It
doesn't recognize ld4ipa01 either.</p>
<p style="margin:10px 0px 0px;padding:0px"> </p>
<p style="margin:10px 0px 0px;padding:0px">right now, if I
try to connect nyc02ipa02 to ld4ipa01 I get</p>
<blockquote style="margin:10px 0px 0px 19px;border-left:1px solid rgb(57,71,44);color:rgb(112,112,112);padding:10px 20px">
<p style="margin:0px;padding:0px">21:56:27
root@nyc02ipa02:~# ipa topologysegment-add domain
ld4ipa01-to-nyc02ipa02 --leftnode ld4ipa01.mf
--rightnode nyc02ipa02.mf<br>
ipa: ERROR: invalid 'leftnode': left node is not a
topology node: ld4ipa01.mf</p>
</blockquote>
</div>
<div id="m_-6106124880320625902gmail-likes-and-labels-container" style="margin:10px 0px;padding:10px 0px;overflow:hidden;clear:both;color:rgb(51,51,51);font-family:arial,sans-serif;font-size:14px">
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace;color:rgb(112,112,112)">ipa privilege-show 'DNS Servers' --all --raw</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace;color:rgb(112,112,112)"> dn: cn=DNS Servers,cn=privileges,cn=pbac,<wbr>dc=mf
cn: DNS Servers
description: DNS Servers
member: krbprincipalname=DNS/<wbr>nyc01ipa02.mf@MF,cn=services,<wbr>cn=accounts,dc=mf
member: krbprincipalname=ipa-<wbr>dnskeysyncd/nyc01ipa02.mf@MF,<wbr>cn=services,cn=accounts,dc=mf
member: krbprincipalname=DNS/<wbr>nyc02ipa02.mf@MF,cn=services,<wbr>cn=accounts,dc=mf
member: krbprincipalname=ipa-<wbr>dnskeysyncd/nyc02ipa02.mf@MF,<wbr>cn=services,cn=accounts,dc=mf
member: krbprincipalname=ipa-ods-<wbr>exporter/nyc01ipa02.mf@MF,cn=<wbr>services,cn=accounts,dc=mf
memberof: cn=System: Read DNS Configuration,cn=permissions,<wbr>cn=pbac,dc=mf
memberof: cn=System: Write DNS Configuration,cn=permissions,<wbr>cn=pbac,dc=mf
memberof: cn=System: Add DNS Entries,cn=permissions,cn=<wbr>pbac,dc=mf
memberof: cn=System: Manage DNSSEC keys,cn=permissions,cn=pbac,<wbr>dc=mf
memberof: cn=System: Manage DNSSEC metadata,cn=permissions,cn=<wbr>pbac,dc=mf
memberof: cn=System: Read DNS Entries,cn=permissions,cn=<wbr>pbac,dc=mf
memberof: cn=System: Remove DNS Entries,cn=permissions,cn=<wbr>pbac,dc=mf
memberof: cn=System: Update DNS Entries,cn=permissions,cn=<wbr>pbac,dc=mf
memberof: cn=System: Read DNS Servers Configuration,cn=permissions,<wbr>cn=pbac,dc=mf
objectClass: top
objectClass: groupofnames
objectClass: nestedgroup</pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace;color:rgb(112,112,112)"></pre>
<pre style="margin-top:10px;margin-bottom:0px;padding:0px;font-family:confluenceinstalledfont,monospace;color:rgb(112,112,112)"></pre>
</div>
</div>
<div class="gmail_default" style="color:rgb(102,102,102)"><br>
</div>
-- <br>
<div class="m_-6106124880320625902gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">Aaron Young<br>
MarketFactory, Manager of Site
Reliability Engineering
<div>425 Broadway, 3FL</div>
<div>New York, NY 10013<br>
Office: <a href="tel:(212)%20625-9988" value="+12126259988" target="_blank">+1 212 625 9988</a></div>
<div>Direct <a href="tel:(646)%20779-3710" value="+16467793710" target="_blank">+1 646 779 3710</a></div>
<div><span style="font-size:12.8px">US
Support: </span><a href="tel:%2B1%20%28212%29%20625-0688" value="+12126250688" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">+1 (212) 625-0688</a><span style="font-size:12.8px"> | UK
Support: </span><a href="tel:%2B44%20%280%29%20203%20695-7997" value="+442036957997" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">+44 (0) 203 695-7997</a><span style="font-size:12.8px"> </span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_-6106124880320625902mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
Hello,<br>
<br>
please don't use kadmin utility, it is not integrated very well with
IPA (or unsupported is a better word) and may break it even more. It
looks that you have corrupted kerberos credentials for id4ipa01 <br>
<br>
I see you are installing client on id4ipa01, was IPA server removed
properly previosly?<span class="HOEnZb"><font color="#888888"><br>
<br>
Martin<br>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Aaron Young<br>MarketFactory, Manager of Site Reliability Engineering<div>425 Broadway, 3FL</div><div>New York, NY 10013<br>Office: +1 212 625 9988</div><div>Direct +1 646 779 3710</div><div><span style="font-size:12.8px">US Support: </span><a href="tel:%2B1%20%28212%29%20625-0688" value="+12126250688" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">+1 (212) 625-0688</a><span style="font-size:12.8px"> | UK Support: </span><a href="tel:%2B44%20%280%29%20203%20695-7997" value="+442036957997" style="color:rgb(17,85,204);font-size:12.8px" target="_blank">+44 (0) 203 695-7997</a><span style="font-size:12.8px"> </span><br></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>