[Freeipa-devel] Multi-master replication with puppet

[James]

On Fri, 2014-06-06 at 08:51 -0400, Simo Sorce wrote:
> On Fri, 2014-06-06 at 06:38 -0400, James wrote:
> > Hi FreeIPA,
> > 
> > *intro*
> > 
> > As some of you might know, I'm currently working on deploying
> > multi-master replicas with puppet. Since it looks like there will be
> > security implications, I wanted to start off by trying to build some
> > confidence. I want to demonstrate that I am as paranoid about getting
> > the security bits right as you all are. *cough* I'm looking at you
> > Simo ;)
> 
> I am cracking my knuckles already :-)
> 
> Comments inline.
> 
> > I've just announced the first sane implementation for secret handling
> > in puppet. Since everyone does this wrong, I thought I'd do it right,
> > by pioneering a new technique. You can read about it here:
> > 
> > https://ttboj.wordpress.com/2014/06/06/securely-managing-secrets-for-freeipa-with-puppet/
> > 
> > In short, the dm_password and admin_password never get touched by
> > puppet, and are generated locally on the freeipa server. What this
> > means is that puppet doesn't know what they are, and as a result,
> > can't use them to accomplish admin tasks.
> 
> Ok let me try first to clarify what is the intent here.
> 
> Clearly puppet has root level access to the system so you do not (should
> not ?) care much about preventing access to these systems, the aim is to
> not inadvertently divulge secrets through manifests and nothing else.
> 
> Correct ?
Basically yes. Secrets should not *touch* puppet. When (if) in the
future puppet has less access to the system, it won't even be able to
snoop passwords.

> 
> > *replicas*
> > 
> > ipa-replica-prepare and ipa-replica-install work _too_ well. they're
> > the one part of freeipa that are so easy to use by a human, but nearly
> > impossible to use with puppet (without storing the password in
> > puppet). puppet can't use them as is because they need you to enter a
> > password (which i don't want it to have). Here is what I propose to do
> > for accomplishing this. Please correct me if I've made a mistake, or
> > recommend something different.
> > 
> > ipa-replica-prepare seems to need the dm_password. This seems to be
> > needed for symmetrically encrypting the bunch of files to transfer to
> > the replica. Instead, I'll patch this process to allow use of a public
> > key for asymetric encryption that is unique to the receiving replica.
> > I'll make sure that each ipa server generates it's own keypair, and
> > that the public halves are distributed to the sending hosts.
> > 
> > I don't think there is any other need for the dm_password in
> > ipa-replica-prepare. Did I miss something?
> 
> Yes, the dm_password was chosen because it is needed to actually
> initialize and install the replica, so instead of asking it twice we
> just ask for it once and use it *also* to encrypt the prepared file. But
> the password is still currently absolutely necessary.
Okay... But it's only needed for encryption on replica-prepare side,
right?

> 
> But let me ask a more important question, how do you distribute the
> public keys securely ? Is it puppet fetching them from each
> replica-to-be server and sending them to the first master server ?
Yes. That is one approach.

> 
> > ipa-replica-install needs the dm_password to decrypt the gpg file.
> > Since I'm handling that with my asymmetric key pair, we don't need the
> > password for that.
> 
> Beeep, sorry wrong answer for now :-)
Ah, I felt the electric shock...

> 
> >  mkosek was very kind in helping me dig through this
> > part a bit, and it seems that the dm_password is needed to allow the
> > replica to bind to the first server.
> 
> It is also needed to actually store the DM password on the replica
> server itself in it's own cn=Directory Manager super user entry.
> Although I think we could send the hash for that.
This is what I was thinking... I'll need to know how to extract this,
send it (encrypted), and use it instead of --password

> 
> >  I would need some help to find an
> > alternative for this process. As was suggested, perhaps a one-time
> > password could be generated and sent across the wire (with puppet) to
> > the replica. Any mechanism that doesn't need the dm_password would be
> > great.
> 
> Forget custom schemes.
> 
> The simplest way to handle this (which I wanted to do for a while) is to
> pre-create the replica's LDAP principal and keys at replica-prepare time
> so we can start right away with GSSAPI authentication for the first
> replication and create agreements directly using GSSAPI auth. This would
> in turn simplify replication agreements creation as we won't need
> anymore to do the dance where we create the agreement with a plain text
> bind and then we need to convert the agreement to use GSSAPI.

This of course sounds fine to me. As long as it meets my criteria of not
needing the password, it should work. Is this an easy patch? IOW
something that I could test a build of within the next month at the
latest?

> 
> > I don't think there is any other need for the dm_password in
> > ipa-replica-install. Did I miss something?
> 
> See above, it needs the password to actually create the DM account at DS
> setup time. *also* the DM password is used to setup some parts of the
> dogtag CA so if any of your replica is also a CA clone you go straight
> back to needing the DM password.
Which I understand we could get around by passing around the (salted?)
hashed password, correct?

> 
> > Of course if I store the dm_password in puppet, then it's easy to do
> > this peering because it will be easy to use the password on each host.
> > I could be done my puppet module already :) But I'm going out of my
> > way to do it right, so that you'll all be proud of the work too!
> 
> Ok, here is a quickest method to do what you want to do IMO:
> 
> First of all, you need a DM password for the initial server and you need
> to store it somewhere because the admin must know what it is, there is
> no way around this.
> So I guess you are planning to store it in a file under /root in the
> master server or something like that. This means puppet has access to
> the DM password w/o storing it in the manifest.
Wrong. The whole idea is I want to keep puppet's grubby hands off the
password. I generate the password on the host, encrypt it with the
admin's public key, and never store it unencrypted. More details here:
https://ttboj.wordpress.com/2014/06/06/securely-managing-secrets-for-freeipa-with-puppet/


> 
> Use the DM password to do everything as usual AND create the asymmetric
> keys in the replicas and *securely* transmit them to the master.
Keep in mind I'm only transmitting public keys, so it doesn't have to be
secure as long as I know verify the identity.

> 
> Now wrap the current replica file (yes it is already encrypted, who
> cares), AND the DM password into a new envelope encrypted with your
> public keys.
> 
> Presto! Now you transfer the blob back to the replicas, and each replica
> can be scripted, and puppet has no passwords in its manifest.

This approach won't work because the first axiom is incorrect. To do
this, I would have to keep the dm_password around on each host with a
key that could be read and used by anyone with root. So effectively I
might as well store the password in puppet.

> 
> > *symmetry*
> > 
> > It would be especially elegant and beneficial to FreeIPA and the
> > puppet module if the process of "peering" was symmetrical, that is to
> > say, similar for any host. If I could use ipa-server-install to setup
> > N hosts, and then run a secondary command to cause certain ones to
> > "join" this would make the process much more natural for puppet.
> 
> This is intrinsically not possible.
That is unfortunate. I could hack a version of this where by each host
runs the installer, but the ones that need to peer start by running:

ipa-server-install --uninstall && ipa-replica-install

but that is less elegant that the solution existing natively.

> 
> > In addition, this would ensure that the configuration management itself
> > is HA. Without this type of functionality, then if the first ipa
> > server isn't available, then config management will be blocked.
> 
> Yeah, chicken-egg issues of bootstrapping infrastructures. I currently
> do not see any way around this.

See my above idea for a dirty hack. (--uninstall)

> 
> > I would appreciate any recommendations on how to convert a previously
> > installed ipa server into a replica. Of course it's fine that it will
> > have it's data overwritten.
> 
> If you overwrite all data what is the point really ? You are still
> dependent on exactly ONE master replicating to all others first, until
> then you cannot do anything, as any operation you would do against any
> other server would be erased and wiped.
The point is that nothing in your config management process (think 100%
automatic without human intervention) can stall. Even if something
fails, there's a path by which it can continue, even if it means wiping
out a server. Highly available config management is necessary for
certain futures.

> 
> HTH,
> Simo.
> 
Thanks for reading!
James

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20140606/a836aedc/attachment.sig>