[libvirt] Symptoms of main loop slowing down in libvirtd

[Daniel P. Berrange]

On Tue, May 02, 2017 at 04:42:11PM +0530, Prerna wrote:
> On Tue, May 2, 2017 at 4:27 PM, Peter Krempa <pkrempa at redhat.com> wrote:
> 
> > On Tue, May 02, 2017 at 16:16:39 +0530, Prerna wrote:
> > > On Tue, May 2, 2017 at 4:07 PM, Peter Krempa <pkrempa at redhat.com> wrote:
> > >
> > > > On Tue, May 02, 2017 at 16:01:40 +0530, Prerna wrote:
> > > >
> > > > [please don't top-post on technical lists]
> > > >
> > > > > Thanks for the quick response Peter !
> > > > > This ratifies the basic approach I had in mind.
> > > > > It needs some (not-so-small) cleanup of the qemu driver code, and I
> > have
> > > > > already started cleaning up some of it. I am planning to have a
> > constant
> > > > > number of event handler threads to start with. I'll try adding this
> > as a
> > > > > configurable parameter in qemu.conf once basic functionality is
> > > > completed.
> > > >
> > > > That is wrong, since you can't guarantee that it will not lock up.
> > Since
> > > > the workers handling monitor events tend to call monitor commands
> > > > themselves it's possible that it will get stuck due to unresponsive
> > > > qemu. Without having a worst-case-scenario of a thread per VM you can't
> > > > guarantee that the pool won't be depleted.
> > > >
> > >
> > > Once a worker thread "picks" an event, it will contend on the per-VM lock
> > > for that VM. Consequently, the handling for that event will be delayed
> > > until an existing RPC call for that VM completes.
> > >
> > >
> > > >
> > > > If you want to fix this properly, you'll need a dynamic pool.
> > > >
> > >
> > > To improve the efficiency of the thread pool, we can try contending for a
> > > VM's lock for a specific time, say, N seconds, and then relinquish the
> > > lock. The same thread in the pool can then move on to process events of
> > the
> > > next VM.
> >
> > This would unnecessarily delay events which are not locked.
> >
> > > Note that this needs all VMs to be hashed to a constant number of threads
> > > in the pool, say 5. This ensures that each worker thread has a unique ,
> > > non-overlapping set of VMs to work with.
> >
> > How would this help?
> >
> > > As an example,  {VM_ID: 1, 6,11,16,21 ..} are handled by the same worker
> > > thread. If this particular worker thread cannot find the requisite VM's
> > > lock, it will move on to the event list for the next VM and so on. The
> > use
> > > of pthread_trylock() ensures that the worker thread will never be stuck
> > > forever.
> >
> > No, I think this isn't the right approach at all. You could end up
> > having all VM's handled with one thread, with others being idle. I think
> > the right approach will be to have a dynamic pool, which will handle
> > incomming events. In case when two events for a single VM should be
> > handled in parallel, the same thread should pick them up in order they
> > arrived. In that way, you will have at most a thread per VM, while
> > normally you will have only one.
> >
> 
> I agree that dynamic threadpool is helpful when there are events from
> distinct VMs that need to be processed at the same time.
> But I am also concerned about efficiently using the threads in this pool.
> If we have a few threads only contend on per-VM locks until the RPCs for
> that VM complete, it is not a very efficient use of resources. I would
> rather have this thread drop handling of this VM's events and do something
> useful while it is unable to grab this VM's lock.
> This is the reason I wanted to load-balance incoming events by VM IDs and
> hash them onto distinct threads. The idea was that a pthread always has
> something else to take up if the current Vm's lock is not available. Would
> you have some suggestions on improving the efficacy of the thread pool as a
> whole ?

Hashing VMs onto threads is really overkill and is also not providing any
kind of fairness. ie, if one VM gets stuck for a long time, any other
VM hashed onto the same thread will get delayed longer than VMs whose
events occurr much later. If we want fewer threads than VMs, then we should
handle events FIFO for fairness.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|