[Freeipa-devel] [PATCH] 1 Do lazy initializiation ipalib

[Martin Kosek]

On Wed, 2011-10-26 at 15:52 +0200, Jan Cholasta wrote:
> Dne 26.10.2011 15:41, Martin Kosek napsal(a):
> > On Wed, 2011-10-26 at 11:39 +0200, Jan Cholasta wrote:
> >> Dne 25.10.2011 22:30, Rob Crittenden napsal(a):
> >>> Ondrej Hamada wrote:
> >>>> On 10/25/2011 04:01 PM, Martin Kosek wrote:
> >>>>> On Tue, 2011-10-25 at 15:29 +0200, Ondrej Hamada wrote:
> >>>>>> https://fedorahosted.org/freeipa/ticket/1336
> >>>>>>
> >>>>>> Lazy initialization of ipalib plugins is used under all contexts, not
> >>>>>> only when context = cli. Every loaded plugin is pre-finalized - a flag
> >>>>>> is set, which means, that the plugin needs to be finalized. Then every
> >>>>>> call of plugin's __gettattr__ checks the flag and finalizes the plugin
> >>>>>> if necessary. The code was implemented by jcholast. Time reduction of
> >>>>>> commands execution is quite markable:
> >>>>>>
> >>>>>> patch [s] | normal [s] | command
> >>>>>> -----------------------------------------------------------------------
> >>>>>> 1.468 | 2.287 | ipa user-add jsmith --firt=john
> >>>>>> --last=smith
> >>>>>> 1.658 | 2.235 | ipa user-del jsmith
> >>>>>> 1.624 | 2.204 | ipa dnsrecord-find example.com
> >>>>>>
> >>>>> Thanks for submitting the patch. Ondra, just please provide the patch in
> >>>>> proper format (exported via command `git format-patch -M -C --patience
> >>>>> --full-index -1' which I sent you earlier).
> >>>>>
> >>>>> Martin
> >>>>>
> >>>>>
> >>>> Sorry, correct version attached
> >>>
> >>> Wow, this is very impressive, great job Ondra and Honza!
> >>>
> >>> Martin, ACK from me but I'd like a second opinion. The patch is very
> >>> straightforward and clean, just want to make sure we're not missing a
> >>> corner case.
> >>>
> >>> I ran the self-tests and didn't see any problem there.
> >>>
> >>> Before pushing please add the ticket # to the commit.
> >>>
> >>> rob
> >>>
> >>
> >> I've just remembered that special methods aren't looked up through
> >> __getattribute__ (see the note at
> >> http://docs.python.org/reference/datamodel.html#more-attribute-access-for-new-style-classes).
> >> That might possibly cause problems in Command and subclasses, because it
> >> uses __call__.
> >>
> >> Honza
> >>
> >
> > This should be investigated. All our unit tests are OK though.
> >
> > I am also thinking if we shouldn't do this optimization on CLI side only
> > as the original ticket suggests. As server side loads and finalizes all
> > plugins just once at httpd start, performance should be OK. Plus, we
> > would know right after the start that we are ready and there is no
> > problem with finalizing any component.
> >
> > Martin
> >
> 
> As we discussed earlier - are you sure it works that way?

Yes.

>  I did a few 
> Django-based applications before and I'm pretty sure all mod_wsgi does 
> is caching the bytecode, which is run from the beginning for each 
> request, like what CLI does.

I did few Django application myself too. Its all about how mod_wsgi is
configured. You may want to learn more about WSGIDaemonProcess and
WSGIImportScript directives.

In IPA, we have API already prepared and finalized for every incoming
request.

> 
> If it does work the way you described, the worst thing that could happen 
> is that the plugins would be finalized the first time they are used and 
> then stay initialized until the server terminates.
> 
> Honza
> 

I know, I was just being conservative. As this optimization has no
effect on server performance I considered it a little bit safer to load
everything at start and be sure we are OK. But its not that hard
requirement.

Martin