[Freeipa-devel] Bulk IPA commands

[Adam Young]

On 10/20/2010 10:42 AM, Rob Crittenden wrote:
> Adam Young wrote:
>> On 10/19/2010 06:29 PM, Dmitri Pal wrote:
>>> Rob Crittenden wrote:
>>>> Adam Young wrote:
>>>>> I think I have an approach that will work for pacaking up multiple
>>>>> commands at once.
>>>>>
>>>> Just want to mention that XML-RPC has a provision for doing multiple
>>>> requests in a single HTTP request that has a similar format. We just
>>>> don't support it currently.
>>>>
>>>> rob
>>>>
>>> The approach makes sense but is it a lot of work?
>>> I suggest we defer it till later and focus on the actual visible work.
>>> If we have time we will do the optimization.
>>> If this eliminates the need for the session cookie at least for now 
>>> I am
>>> Ok with deferring it too.
>> It might be worthwhile to do this as part of Pavel's work on controlling
>> access to the fields based on back end permissions. We are packing more
>> and more stuff into a single json_metadata call, but it will make things
>> much easier for us if we can just make a single bulk call and parse out
>> what we need.
>>
>> I'm most concerned with the approach being approved.
>
> It seems reasonable but I'm not sure I favor adding it only to json. 
> Our goal has always been to keep all interfaces equal. The XML-RPC 
> equivalent is multicall, http://docs.python.org/library/xmlrpclib.html
>
> What are your plans in the case of exceptions? Short-circuit or keep 
> going?

By default, keep going, and return the error codes.  We can provide an 
option to short circuit as well.

>
> rob
>
>>
>>
>>
>>>
>>>>> Stage 1:
>>>>>
>>>>>
>>>>> 1. We create a new plugin, we call it bulk. For now, it will speak 
>>>>> only
>>>>> JSON.
>>>>>
>>>>> 2. The json requests look like this:
>>>>>
>>>>>
>>>>> |{"method":"bulk","params":[[|
>>>>> | {"method":"user_find","params":[[""],{}]},|
>>>>> | {"method":"user_show","params":[[""],{}]},|
>>>>> | {"method":"user_add","params":[[""],{}]},|
>>>>> |{"method":"user_blah","params":[[""],{}]}|,
>>>>> |],{"all":true,"sizelimit":100}],"id":2}|
>>>>>
>>>>>
>>>>> This gets sent as a single json_request.
>>>>>
>>>>> On the server side, each request is pulled out of the params and the
>>>>> common items all=true and sizelimit=100 is added to each request. The
>>>>> resulting json request is then executed locally, without re-requiring
>>>>> the kerberos auth. THe response from the individual requests are put
>>>>> into the response array, like so:
>>>>>
>>>>> |{
>>>>> || "error": null,
>>>>> || "id": 2,
>>>>> || "result": {
>>>>> || "count": 1,
>>>>> || "result": [{
>>>>> || "error": null, //start of the response for the first nested 
>>>>> command.
>>>>> || "id": 1,
>>>>> || "result": {
>>>>> || "count": 1,
>>>>> || "result": [
>>>>> || {
>>>>> || "cn": [
>>>>> || "Administrator"
>>>>> || ],|
>>>>>
>>>>> ...
>>>>>
>>>>> Thus the plugin is responsible for processing each request, 
>>>>> marshalling
>>>>> up the return and adding it to the current response.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Stage 2: we add an asynchronous mechanism. We add an option for the
>>>>> email notification address to the bulk plugin. The response returns
>>>>> immediately. Meanwhile, the plugin hands off the processing of the
>>>>> initial request to a queue that will be handled by another thread or
>>>>> process, and email the response to the address. Alternatively, we 
>>>>> email
>>>>> back just an abbreviated status message.
>>>>>
>>>>>
>>>>> Stage 3: Store the response in a filesystem or the DirSrv, and mail
>>>>> back
>>>>> a link that allows the user to fetch the status.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I really only care about Stage 1 for now. THe json_metadata plugin is
>>>>> getting loaded up with too many unrelated calls in it. I'd like to be
>>>>> able to craft and arbitraty message that gets sent and returned with
>>>>> all
>>>>> of the information required to initialize the IPA Web UI.
>>>>>
>>>>> user-find --whoami
>>>>> json_metadata
>>>>> I18N_messages
>>>>> Effective rights
>>>>> plugin lists (to enable/disable UI features based on what is active)
>>>>>
>>>>> While we would do this initially as a JSON only call, the XML-RPC
>>>>> should
>>>>> nest the same way. I leave it to Rob and Simo to figureo ut if this
>>>>> would support the CLI, but it seems to me that it should be pretty 
>>>>> easy
>>>>> to do:
>>>>>
>>>>>
>>>>> ipa bulk
>>>>>> ipa user-mod ...
>>>>>> ipa user-del ...
>>>>>> upa group-add ...
>>>>> ^D
>>>>>
>>>>> And have that sent as a single command, with the responses again 
>>>>> parsed
>>>>> out by the cli. Then the user can do:
>>>>>
>>>>> cat mycommands | ipa bulk> bulk.response
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Freeipa-devel mailing list
>>>>> Freeipa-devel at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/freeipa-devel
>>>> _______________________________________________
>>>> Freeipa-devel mailing list
>>>> Freeipa-devel at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/freeipa-devel
>>>
>>
>