[Freeipa-users] Fwd: Fwd: Fwd: Scorched earth
Bret Wortman
bret.wortman at damascusgrp.com
Fri Aug 30 10:08:17 UTC 2013
Now, look. Maybe I'm being thick, or maybe my master is really messed up.
But when I take a replica file to a freshly-created replica with a virgin
OS and install that replica file as the first act on that system and it
fails with the same error, basically that "a replication agreement for this
host already exists," I have to say, "of *course* a replication agreement
already exists, because I *just bleeding set it up!*"
Obviously, I am not understanding something. I asked the master to create a
file for a new replica. I take that file to a new replica. Installation
fails because the agreement exists. So I have to go back to the master to
delete the replication agreement before I can add the replica? Now that I
think of it, this has been what I've had to do for every replica along the
way, but until the master got hosed, it hasn't been a show-stopper.
Given that I can't remove agreements from my server (which is the root
problem I'm trying to solve by creating one replica that can hold my actual
data long enough for me to nuke that master and restore it from the
temporary replica), is there any way to make this work?
Sorry. It's been a long ten or eleven days. I'm not upset, just punchy. I
woke up at 1AM with an idea that hasn't panned out, so it's also been a
long morning already.
*
*
*Bret Wortman*
http://damascusgrp.com/
http://about.me/wortmanbret
On Fri, Aug 30, 2013 at 5:17 AM, Bret Wortman
<bret.wortman at damascusgrp.com>wrote:
> On Fri, Aug 30, 2013 at 5:03 AM, Petr Viktorin <pviktori at redhat.com>wrote:
>
>> On 08/30/2013 10:23 AM, Bret Wortman wrote:
>>
>>> Morning update. I made the change Rob suggested to
>>> /etc/ipa/default.conf, which appeared to work, but didn't quite. It
>>> asked me to back out the whole server installation and start over:
>>>
>>> [ipamaster2]# ipa-ca-install --skip-conncheck
>>> replica-info-ipamaster2.foo.**net.gpg
>>> Directory Manager (existing master) password:
>>>
>>> COnfiguring certificate server (pki-tomcatd): Estimated time 3 minutes
>>> 30 seconds
>>> [1/16]: creating certificate server user
>>> [2/16]: configuring certificate server instance
>>> ipa : CRITICAL failed to configure ca instance Command
>>> '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exit
>>> status 1
>>>
>>> Your system may be partly configured.
>>> Run/usr/sbin/ipa-server-**install --uninstall to clean up.
>>>
>>
>> Can you look into /var/log/ipareplica-ca-**install.log? It should have
>> more information on what caused pkispawn to fail.
>>
>> Here's what it looks like:
>
> Storing deployment configuration into
> /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.
> Installation failed.
>
>
> 2013-08-30T07:37:24Z DEBUG stderr=pkispawn : WARNING ...... unable to
> validate security domain user/password through REST interface. Interface
> not available
> pkispawn : ERROR ...... Exception from Java Configuration Servlet:
> Failed to obtain installation token from security domain:
> java.lang.NullPointerException
>
> 2013-08-30T07:37:24Z CRITICAL failed to configure ca instance Command
> '/usr/sbin/pkispawn -s CA -f /tmp/tmpVC28HP' returned non-zero exist status
> 1
> 2013-08-30T07:37:24Z INFO File
> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line
> 619, in run_script
> return_value = main_function()
>
> File "/usr/sbin/ipa-ca-install", line 182, in main
> config, dogtag_master_ds_port, postinstall=True)
> :
> File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
> line 744, in __spawn_instance
> raise RuntimeError('Configuration of CA failed')
> 2013-08-30T07:37:24Z INFO The ipa-ca-install command failed, exception:
> RuntimeError: Configuration of CA Failed
>
>
>> Configuration of CA failed.
>>> [ipamaster2]#
>>>
>>> Which uninstallation & cleanup I did.
>>>
>>> Now, when trying to re-install the
>>> replica file:
>>>
>>> [ipamaster2]# ipa-replica-install --setup-dns --no-forwarders --setup-ca
>>> /var/lib/ipa/replica-info-**ipamaster2.foo.net.gpg
>>> Directory manager (existing master) password:
>>>
>>> Run connection check to master
>>> Check connection from replica to remote master 'ipamaster.foo.net
>>> <http://ipamaster.foo.net>':
>>>
>>> Directory Service: Unsecure port (389): OK
>>> Directory Service: Secure port (686): OK
>>> Kerberos KDC: TCP (88): OK
>>> Kerberos Kpasswd: TCP (464): OK
>>> HTTP Server: Unsecure port (80): OK
>>> HTTP Server: Secure port (443): OK
>>>
>>> The followign list of ports use UDP protocol and would need to be
>>> checked manually:
>>> Kerberos KDC: UDP (88): SKIPPED
>>> Kerberos Kpasswd: UDP (464): SKIPPED
>>>
>>> Connection from replica to master is OK.
>>> Start listening on required ports for remote master check
>>> Get credentials to log in to remote master
>>> admin at FOO.NET <mailto:admin at FOO.NET> password:
>>>
>>>
>>> Check SSH connection to remote master
>>> Execute check on remote master
>>> Check connection from master to remote replica 'ipamaster2.foo.net
>>> <http://ipamaster2.foo.net>':
>>>
>>> Directory Service: Unsecure port (389): OK
>>> Directory Service: Secure port (686): OK
>>> Kerberos KDC: TCP (88): OK
>>> Kerberos KDC: UDP (88): OK
>>> Kerberos Kpasswd: TCP (464): OK
>>> Kerberos Kpasswd: UDP (464): OK
>>> HTTP Server: Unsecure port (80): OK
>>> HTTP Server: Secure port (443): OK
>>>
>>> Connection from master to replica is OK.
>>>
>>> Connection check OK
>>> The host ipamaster2.foo.net <http://ipamaster2.foo.net> already exists
>>>
>>> on the master server.
>>> You should remove it before proceeding:
>>> % ipa host-del ipamaster2.foo.net <http://ipamaster2.foo.net>
>>>
>>> ipa : ERROR Could not resolve hostname ipamaster.foo.net
>>> <http://ipamaster.foo.net> using DNS Clients may not function properly.
>>>
>>> Please check your DNS setup. (Note that this check queries IPA DNS
>>> directly and ignores /etc/hosts.)
>>> Continue? [no]: *yes*
>>> [ipamaster2]# host ipamaster.foo.net <http://ipamaster.foo.net>
>>> ipamaster.foo.net <http://ipamaster.foo.net> has address 1.2.3.4
>>>
>>>
>>> No matter what answer I give to the "Continue?" prompt, it just exits.
>>> "nslookup" returns the same value, and I have three different
>>> nameservers configured for this host (including ipamaster and two of the
>>> older replicas).
>>>
>>
>> The error that caused the installation to fail is that ipamaster2.foo.netalready exists on the master server.
>>
>> The DNS warning and its "Continue?" prompt is unrelated, but the order of
>> the output is very confusing. I've filed ticket 3889 for this.
>> Anyway, to do this DNS resolution check you'd need to explicitly ask for
>> the IPA server:
>> $ dig @ipamaster.foo.net ipamaster2.foo.net
>>
>> Right, and that's a problem because the main reason for needing to clone
> this box was that the master server has got its tentacles in a wad and
> can't actually delete any replication agreements. It's missing some who
> have died and left the configuration, and others who are uncommunicative.
> Any new attempts to remove replicas just get stuck behind this logjam and
> add to the fun.
>
>
>> And this message is the one that has prompted me to want to delete hosts
>>> before installing in the past, Simo.
>>>
>>> Any thoughts on how best to proceed now?
>>>
>>
>> I believe you do need to delete he host at this point, but I'd rather
>> have Rob or Simo confirm.
>>
>> Instead, I believe I'll change this system's name and let it become yet
> another new replica instead. That should take less time in the long run, as
> long as I don't end up running out of IP addresses on this subnet. ;-)
>
> I dug into this job's log file earlier and found that the DNS problem was
> the script trying to resolve against a nonexistent replica (see prior
> comment). And the exit even when I tell it to continue appears to be an
> inline sys.exit(3) call on line 629 of the ipa-replica-install script,
> which exits if a replication agreement already exists. The script is
> supposed to print a message to that effect but for some reason, that
> message isn't getting to stdout before the sys.exit(3) occurs.
>
> Another reason to give this temporary box a new identity and try again.
>
> Thanks for your help!
>
>
>> *Bret Wortman*
>>>
>>> http://damascusgrp.com/
>>> http://about.me/wortmanbret
>>>
>>>
>>> On Thu, Aug 29, 2013 at 2:59 PM, Rob Crittenden <rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>> wrote:
>>>
>>> Bret Wortman wrote:
>>>
>>> Okay, I got the cacert.p12 (turns out it was taking my
>>> passphrase, but
>>> the messages looked like errors to my addled eyes). This system
>>> is on a
>>> different network, so getting the file transferred would take me
>>> about
>>> 24 hours. Is there something I can get that'll tell you what you
>>> need
>>> but is plaintext?
>>>
>>>
>>> Ok, that's fine.
>>>
>>> Try this. Set ra_plugin to dogtag in /etc/ipa/default.conf. This
>>> will let it get past the error and it should install a CA. I'm
>>> trying to think worst case scenario what it might do and I'm not
>>> coming up with anything. I think the worst that happens is that
>>> adding a CA fails later.
>>>
>>> rob
>>>
>>>
>>> I tried this and hope this subset of information is helpful:
>>>
>>> # openssl pkcs12 -in cacert.p12 -out cacert.pem.bdw -cacerts
>>> -nokeys
>>> # cat cacert.pem.bdw
>>> Bag Attributes: <No Attributes>
>>> subject=/O=FOO.NET/CN=__**Certificate<http://FOO.NET/CN=__Certificate>
>>> <http://FOO.NET/CN=Certificate**>
>>> <http://FOO.NET/CN=**Certificate__<http://FOO.NET/CN=Certificate__>>
>>> Authority/
>>> issuer=/O=FOO.NET/CN=__**Certificate<http://FOO.NET/CN=__Certificate>
>>> <http://FOO.NET/CN=Certificate**>
>>> <http://FOO.NET/CN=**Certificate__<http://FOO.NET/CN=Certificate__>>
>>> Authority
>>>
>>>
>>> -----BEGIN CERTIFICATE-----
>>> MIIDgzCCA...
>>> ...Iwk4r
>>> -----END CERTIFICATE-----
>>> # openssl pkcs12 -in cacert.p12 -out cert.pem.bdw -clcerts
>>> -nokeys
>>> # cat cert.pem.bdw
>>> Bag Attributes:
>>> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE
>>> 6C A7 2E 53 88
>>> friendlyName: caSigningCert cert-pki-ca
>>> subject=/O=FOO.NET/CN=__**Certificate<http://FOO.NET/CN=__Certificate>
>>> <http://FOO.NET/CN=Certificate**>
>>> <http://FOO.NET/CN=**Certificate__<http://FOO.NET/CN=Certificate__>>
>>> Authority
>>> issuer=/O=FOO.NET/CN=__**Certificate<http://FOO.NET/CN=__Certificate>
>>> <http://FOO.NET/CN=Certificate**>
>>> <http://FOO.NET/CN=**Certificate__<http://FOO.NET/CN=Certificate__>>
>>> Authority
>>>
>>>
>>> -----BEGIN CERTIFICATE-----
>>> MIIDgzCCA...
>>> ...Iwk4r
>>> -----END CERTIFICATE-----
>>> Bag Attributes:
>>> localKeyID: 88 BF DF 56 30 BB A9 47 12 D4 5F 7B AE 39 DC
>>> BF CF F5 92 22
>>> friendlyName: ocspSigningCert cert-pki-ca
>>> subject=/O=FOO.NET/CN=OCSP <http://FOO.NET/CN=OCSP>
>>> <http://FOO.NET/CN=OCSP> Subsystem
>>> issuer=/O=FOO.NET/CN=__**Certificate<http://FOO.NET/CN=__Certificate>
>>> <http://FOO.NET/CN=Certificate**>
>>> <http://FOO.NET/CN=**Certificate__<http://FOO.NET/CN=Certificate__>>
>>> Authority
>>>
>>>
>>> -----BEGIN CERTIFICATE-----
>>> MIIDYTCCA...
>>> ...wlr4Q=
>>> -----END CERTIFICATE-----
>>> Bag Attributes:
>>> localKeyID: B5 3B 27 CC 57 72 45 E2 8D 46 C9 5E E1 C0 50
>>> DF 2D 11 62 0E
>>> friendlyName: subsystemCert cert-pki-ca
>>> subject=/O=FOO.NET/CN=CA <http://FOO.NET/CN=CA>
>>> <http://FOO.NET/CN=CA> Subsystem
>>> issuer=/O=FOO.NET/CN=__**Certificate<http://FOO.NET/CN=__Certificate>
>>> <http://FOO.NET/CN=Certificate**>
>>> <http://FOO.NET/CN=**Certificate__<http://FOO.NET/CN=Certificate__>>
>>> Authority
>>>
>>>
>>> -----BEGIN CERTIFICATE-----
>>> MIIDaTCCA...
>>> ...BxqqA==
>>> -----END CERTIFICATE-----
>>> Bag Attributes:
>>> localKeyID: 1F 69 62 C7 88 D8 95 2A B1 7D 61 F9 10 87 14
>>> D0 76 Ba B9 44
>>> friendlyName: auditSigningCert cert-pki-ca
>>> subject=/O=FOO.NET/CN=CA <http://FOO.NET/CN=CA>
>>> <http://FOO.NET/CN=CA> Audit
>>> issuer=/O=FOO.NET/CN=__**CertificateAUthority<http://FOO.NET/CN=__CertificateAUthority>
>>> <http://FOO.NET/CN=**CertificateAUthority<http://FOO.NET/CN=CertificateAUthority>
>>> >
>>> <http://FOO.NET/CN=__**CertificateAUthority<http://FOO.NET/CN=__CertificateAUthority>
>>>
>>> <http://FOO.NET/CN=**CertificateAUthority<http://FOO.NET/CN=CertificateAUthority>
>>> >>
>>>
>>> -----BEGIN CERTIFICATE-----
>>> MIIDRDCCA...
>>> ...EAd+Ug7
>>> -----END CERTIFICATE-----
>>> # openssl pkcs12 -in cacert.p12 -out key.pem.bdw -nocerts
>>> # cat key.pem.bdw
>>> Bag Attributes
>>> localKeyID: 82 81 2D 6E 5C 13 43 9A 5F BB C8 4D F5 6B DE
>>> 6C A7 2E 53 88
>>> friendlyName: CN=Certificate Authority,O=FOO.NET
>>> <http://FOO.NET> <http://FOO.NET>
>>>
>>>
>>> Key Attributes: <No Attributes>
>>> -----BEGIN ENCRYPTED PRIVATE KEY-----
>>> MIIFDjBAB...
>>> ...XLtoD8=
>>> -----END ENCRYPTED PRIVATE KEY-----
>>> Bag Attributes
>>> localKeyID: <let me know if you need this>
>>> friendlyName: CN=OCSP Subsystem,O=FOO.NET <http://FOO.NET>
>>> <http://FOO.NET>
>>>
>>> Key Attributes: <No Attributes>
>>> -----BEGIN ENCRYPTED PRIVATE KEY-----
>>> :
>>> -----END ENCRYPTED PRIVATE KEY-----
>>> Bag Attributes
>>> localKeyID: <let me know if you need this>
>>> friendlyName: CN=CA Subsystem,O=FOO.NET <http://FOO.NET>
>>> <http://FOO.NET>
>>>
>>> Key Attributes: <No Attributes>
>>> Bag Attributes
>>> localKeyID: <let me know if you need this>
>>> friendlyName: CN=CA Audit,O=FOO.NET <http://FOO.NET>
>>> <http://FOO.NET>
>>>
>>> Key Attributes: <No Attributes>
>>>
>>> If you need to see anything else, please let me know.
>>>
>>>
>>> _
>>> _
>>> *Bret Wortman*
>>>
>>>
>>> http://damascusgrp.com/
>>> http://about.me/wortmanbret
>>>
>>>
>>> On Thu, Aug 29, 2013 at 11:58 AM, Rob Crittenden
>>> <rcritten at redhat.com <mailto:rcritten at redhat.com>
>>> <mailto:rcritten at redhat.com <mailto:rcritten at redhat.com>>>
>>> wrote:
>>>
>>> Bret Wortman wrote:
>>>
>>> On Thu, Aug 29, 2013 at 11:40 AM, Rob Crittenden
>>> <rcritten at redhat.com <mailto:rcritten at redhat.com>
>>> <mailto:rcritten at redhat.com <mailto:rcritten at redhat.com>>
>>> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>>>**>____wrote:
>>>
>>>
>>> Bret Wortman wrote:
>>>
>>> On Thu, Aug 29, 2013 at 11:10 AM, Rob
>>> Crittenden
>>> <rcritten at redhat.com
>>> <mailto:rcritten at redhat.com> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>>
>>> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>>>
>>> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>
>>> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>
>>> <mailto:rcritten at redhat.com
>>> <mailto:rcritten at redhat.com>>>**>__> wrote:
>>>
>>> Bret Wortman wrote:
>>>
>>> A bit of googling has led me to
>>> understand
>>> that we must
>>> have
>>> created the
>>> original server with --selfsign, and
>>> that
>>> locked us into
>>> something bad
>>> which is now causing us problems. I'm
>>> not sure
>>> how this
>>> happened, since
>>> we actually created our original
>>> instance on a
>>> different server,
>>> created
>>> ipamaster as a replica of that one,
>>> then ran
>>> ipa-ca-install on
>>> ipamaster
>>> to make it the new CA. How did it end
>>> up in
>>> this state?
>>>
>>> Anyway, is there ANY way around this?
>>> Can I simply
>>> ignore this,
>>> break
>>> the replication agreement as Simo
>>> suggested,
>>> rebuild
>>> ipamaster,
>>> replicate ipamaster2 to the new
>>> ipamaster, and
>>> then
>>> somehow make
>>> ipamaster be a CA using Dogtag? Will
>>> that
>>> screw up all
>>> the clients?
>>>
>>>
>>> I think we should pause and take a look
>>> at your
>>> installation.
>>>
>>> I'd check all your current masters,
>>> whether they
>>> are currently
>>> working or not. Look at the value of
>>> ra_plugin in
>>> /etc/ipa/default.conf. That controls what
>>> IPA
>>> thinks the CA is.
>>>
>>> on ipamaster: ra_plugin=dogtag
>>>
>>> and either that same value or the ra_plugin
>>> doesn't
>>> exist on the
>>> replicas. On ipamaster2, the one I just
>>> installed,
>>> there is no
>>> ra_plugin
>>> in the file.
>>>
>>> Then check to see if you have dogtag
>>> running on
>>> any of these
>>> systems. This will include a 2nd 389-ds
>>> instance,
>>> /etc/dirsrv/slapd-PKI-IPA and, depending
>>> on your
>>> distro, a PKI
>>> service like
>>> pki-tomcatd at pki-tomcat._______**_service.
>>>
>>>
>>> You can
>>>
>>> optionally
>>>
>>> see if /etc/pki/pki-tomcat exists.
>>>
>>> ipamaster definitely has a
>>> /etc/dirsrv/slapd-PKI-IPA
>>> directory, with
>>> files updated fairly recently (within the past
>>> 30 minutes -
>>> lse.ldif and
>>> lse.ldif.bak, others updated yesterday). I
>>> also have a
>>> pki-tomcatd at .service file and a
>>> pki-tomcatd.target. no
>>> /etc/pki/pki-tomcat.
>>>
>>> ipamaster2 only has /etc/dirsrv/slapd-FOO-NET.
>>> It does have
>>> pki-tomcatd.target and pki-tomcatd at .service.
>>> No
>>> /etc/pki/pki-tomcat.
>>>
>>>
>>> Ok. When you created the replica file for
>>> ipamaster2, did
>>> you create
>>> it on ipamaster? Only a replica that is a CA can
>>> create a
>>> replica
>>> with a CA.
>>>
>>> Yes. So I'm not sure what went askew.
>>>
>>>
>>> Ok. I think we need to see what's in the prepared file. It
>>> is just a
>>> gpg-encrypted tarball. Can you do something like:
>>>
>>> gpg -d replica-info-pacer.greyoak.___**_com.gpg |tar xf -
>>>
>>>
>>>
>>> This will create a realm_info subdirectory. The file
>>> cacert.p12
>>> should be in there.
>>>
>>> rob
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ______________________________**_________________
>>> Freeipa-users mailing list
>>> Freeipa-users at redhat.com
>>> https://www.redhat.com/**mailman/listinfo/freeipa-users<https://www.redhat.com/mailman/listinfo/freeipa-users>
>>>
>>>
>>
>> --
>> Petr³
>>
>>
>> ______________________________**_________________
>> Freeipa-users mailing list
>> Freeipa-users at redhat.com
>> https://www.redhat.com/**mailman/listinfo/freeipa-users<https://www.redhat.com/mailman/listinfo/freeipa-users>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20130830/0a16e7cb/attachment.htm>
More information about the Freeipa-users
mailing list