[Freeipa-devel] Client-side command in the IPA framework

Tue Mar 4 19:27:09 UTC 2014

On Tue, 2014-03-04 at 14:11 -0500, Dmitri Pal wrote:
> On 03/04/2014 02:03 PM, Nathaniel McCallum wrote:
> > On Mon, 2014-03-03 at 20:12 -0500, Dmitri Pal wrote:
> >> On 03/01/2014 10:07 PM, Adam Young wrote:
> >>> On 02/28/2014 10:21 AM, Petr Viktorin wrote:
> >>>> On 02/28/2014 04:15 PM, Alexander Bokovoy wrote:
> >>>>> On Fri, 28 Feb 2014, Nathaniel McCallum wrote:
> >>>>>> On Fri, 2014-02-28 at 16:43 +0200, Alexander Bokovoy wrote:
> >>>>>>> On Fri, 28 Feb 2014, Nathaniel McCallum wrote:
> >>>>>>>> On Fri, 2014-02-28 at 10:47 +0100, Petr Vobornik wrote:
> >>>>>>>>> On 28.2.2014 04:02, Rob Crittenden wrote:
> >>>>>>>>>> Alexander Bokovoy wrote:
> >>>>>>>>>>> On Thu, 27 Feb 2014, Nathaniel McCallum wrote:
> >>>>>>>>>>>> So the recent discussion on importing tokens led me to write a
> >>>>>>> script to
> >>>>>>>>>>>> parse RFC 6030 xml files into IPA token data. This all works
> >>>>>>> well. But
> >>>>>>>>>>>> now I need to integrate it into the IPA framework.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This command will parse one or more xml files, creating a set
> >>>>>>> of tokens
> >>>>>>>>>>>> to be added. Given that we already have otptoken-add on the
> >>>>>>> server-side,
> >>>>>>>>>>>> it seems to me that all work needs to be done on the
> >>>>>>> client-side. How do
> >>>>>>>>>>>> I create a new client-side command that calls existing
> >>>>>>> server-side API?
> >>>>>>>>>>> subclass from frontend.Local, override run() or forward()
> >>>>>>> method and
> >>>>>>>>>>> perform batch
> >>>>>>>>>>> operation of otptoken_add from there.
> >>>>>>>>>>>
> >>>>>>>>>>> See cli.help, for example.
> >>>>>>>>>> If you do an override, do forward() for cli-specific work.
> >>>>>>>>>>
> >>>>>>>>>> But you should do as little as possible for reasons you already
> >>>>>>> stated:
> >>>>>>>>>> the UI. Anything you do in forward Petr will need to implement
> >>>>>>> in the UI.
> >>>>>>>>>> Unfortunately we don't yet have a nice way to handle files.
> >>>>>>> We have
> >>>>>>>>>> tickets open at https://fedorahosted.org/freeipa/ticket/1225 and
> >>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2933
> >>>>>>>>>>
> >>>>>>>>>> If this file is something that would be pasted into a big text
> >>>>>>> field
> >>>>>>>>>> then you can probably handle it in a similarly clumsy way that
> >>>>>>> we do
> >>>>>>>>>> CSRs in the cert plugin.
> >>>>>>>>>>
> >>>>>>>>>> rob
> >>>>>>>>> +1 for parsing it on server. Otherwise every client, not just CLI
> >>>>>>> or Web
> >>>>>>>>> UI, would have to reimplement the same logic - having it on server
> >>>>>>> will
> >>>>>>>>> support better integration with third party products.
> >>>>>>>>>
> >>>>>>>>> Parsing on client would be understandable if there was some middle
> >>>>>>> step
> >>>>>>>>> which would require some action from user, i.e, pick only some
> >>>>>>> tokens to
> >>>>>>>>> import.
> >>>>>>>> If we parse on the server side, how do we handle the long-running
> >>>>>>>> operation? Think of the case of importing hundreds or thousands of
> >>>>>>>> tokens...
> >>>>>>> Why then to do it as a IPA CLI command at all?
> >>>>>>> This is an administrative task which can be done with a separate
> >>>>>>> ipa-otp-import command, designated to run on IPA masters.
> >>>>>> Agreed.
> >>>>>>
> >>>>>> 1. Is there a framework for this? Or should it just be an independent
> >>>>>> script?
> >>>>> We don't really have a framework for administrative tools. You may
> >>>>> start
> >>>>> with install/tools/ipa-adtrust-install, it is main part is relatively
> >>>>> independent of the task (which is in
> >>>>> ipaserver/install/adtrustinstance.py)
> >>>>>
> >>>> The framework is there, new tools use it, and there's a ticket to
> >>>> convert old ones: https://fedorahosted.org/freeipa/ticket/2652 (it's
> >>>> low priority in Future Releases, so not much progress is there...)
> >>>> Also see http://www.freeipa.org/page/V3/Logging_and_output
> >>>>
> >>>>
> >>>
> >>> The RESTful approach would be:
> >>>
> >>> 1. Upload a file to a specific URL (not JSON RPC)
> >>> 2.  Receive back a 202 Accepted  HTTP Request, to include an URL to
> >>> poll for status
> >>>
> >>> Not certain the right response from the URL in step 2 would be, but I
> >>> am assuming it would be 200 with the body of the message stating:
> >>> processing or completed.
> >>>
> >>> It would be really nice if the Batch command could be handled this way
> >>> as well.  The response back could be the partial responses until
> >>> processing is complete.
> >>>
> >>> It might also be nice to supply an email address for notification of
> >>> completed processing instead of polling, if it is going to be a really
> >>> long running task.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Freeipa-devel mailing list
> >>> Freeipa-devel at redhat.com
> >>> https://www.redhat.com/mailman/listinfo/freeipa-devel
> >> Yes I think that:
> >> 1) We should not limit it to server side operation only
> >> 2) Upload the whole file and then process it.
> >> 3) We should already have code to upload files, we did it for
> >> entitlements and were supposed to use for certs.
> >> 4) Make sure we have a generic upload mechanism that reads a chunk of a
> >> configurable size and asks for more (pagination by 65K might be a good
> >> default).
> >>
> >> Regarding token files specifically: they can be big but not super huge.
> >> 10-20K tokens makes sense but probably not more. More than that would be
> >> a real corner case becuase it is hard to deploy that amount of tokens at
> >> the same time. It can take months and you do not want token file to
> >> contain many tokens that would sit on the shelf. Tokens expire so it is
> >> inefficient to buy huge chunks and let them sit unused.
> >>
> >> UI you allow uploading file too and then would process it locally.
> >> The processing of the file should generate a log or report. It would be
> >> nice to get indication from the server that it is still working so may
> >> be upload protocol should be something like:
> >>
> >> client: Initialize the transfer
> >> server: ready
> >> client: here is the chunk of data
> >> server: ack
> >> ...
> >> client: here is the last chunk of data
> >> server: ack, (forks the file processing method that updates shared
> >> status data) come back in x seconds
> >> client: how are things?
> >> server: working, here is current status, come back in x seconds
> >> ...
> >> client: how are things?
> >> server: done, here is current status, have errors in a file
> >> client: start download
> >> server: here is the chunk
> >> ...
> >>
> >> I think we can short socket the command for now to fail if it is not
> >> local on the server and then build the upload mechanism but separate
> >> command as proposed in this thread would lock us in a local approach
> >> forever.
> > The problem is that we have no infrastructure for any of this. It would
> > all have to be built just for the import command. It is also a fairly
> > rare, admin-only operation. For this reason, I am leaning towards
> > implementing it as a simple script to be executed on the FreeIPA master.
> > The main drawback of this approach is that you don't get import
> > functionality in the UI.
> >
> > I also disagree that doing it this way now necessitates we do it this
> > way forever. In fact, it is more likely that if we design a fully
> > featured server implementation now, we'll get details wrong and be stuck
> > with it. Doing it on the master using the existing API frees us to add a
> > server-side API in the future.
> >
> > Nathaniel
> >
> We already have use cases for upload but we defer it again and again for 
> a better time.
> We need to upload and download files for other use cases:
> 
> Download:
> a) Keytabs (we have a ticket to do it in the UI)
> b) Reports
> c) Import logs (for tokens or users)
> ...
> 
> Upload:
> a) Token file
> b) SSH public keys
> c) Register user certificate in IPA
> ...
> 
> We need this mechanisms anyways.
> We have been deferring this for too long.

I agree.

> I would rather defer support of the hardware tokens till next version 
> and do an upload in the cli and UI than cut the corner and create yet 
> one off mechanism.
> 
> IMO we are going the wrong way about it.

So the problem isn't hardware tokens, the problem is that users can
modify tokens they shouldn't (these are most commonly hardware, but not
necessarily so). Support for this can be added trivially once I get some
of the more pressing issues off my plate. I don't see any reason to
delay this. Let's not conflate issues.

Imports are a different story. We could land an independent import
script (IIS) in 4.0. This functionality could be moved to a framework
script (FS) later. If backwards compatibility for this is a concern, the
IIS could just point to the FS. If we desire not to support the IIS at
all, we could ship it as contrib or distribute it independently.

But once we have infrastructure for uploads/downloads/long-running task
state, the code from the IIS should move to the server side with little
modification. So it isn't lost effort.

Having *some* form of import is highly desirable for 4.0.

Nathaniel