<div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">On Tue, May 2, 2017 at 4:27 PM, Peter Krempa <<a href="mailto:pkrempa@redhat.com" target="_blank">pkrempa@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, May 02, 2017 at 16:16:39 +0530, Prerna wrote: > On Tue, May 2, 2017 at 4:07 PM, Peter Krempa <<a href="mailto:pkrempa@redhat.com">pkrempa@redhat.com</a>> wrote: > > > On Tue, May 02, 2017 at 16:01:40 +0530, Prerna wrote: > > > > [please don't top-post on technical lists] > > > > > Thanks for the quick response Peter ! > > > This ratifies the basic approach I had in mind. > > > It needs some (not-so-small) cleanup of the qemu driver code, and I have > > > already started cleaning up some of it. I am planning to have a constant > > > number of event handler threads to start with. I'll try adding this as a > > > configurable parameter in qemu.conf once basic functionality is > > completed. > > > > That is wrong, since you can't guarantee that it will not lock up. Since > > the workers handling monitor events tend to call monitor commands > > themselves it's possible that it will get stuck due to unresponsive > > qemu. Without having a worst-case-scenario of a thread per VM you can't > > guarantee that the pool won't be depleted. > > > > Once a worker thread "picks" an event, it will contend on the per-VM lock > for that VM. Consequently, the handling for that event will be delayed > until an existing RPC call for that VM completes. > > > > > > If you want to fix this properly, you'll need a dynamic pool. > > > > To improve the efficiency of the thread pool, we can try contending for a > VM's lock for a specific time, say, N seconds, and then relinquish the > lock. The same thread in the pool can then move on to process events of the > next VM. This would unnecessarily delay events which are not locked. > Note that this needs all VMs to be hashed to a constant number of threads > in the pool, say 5. This ensures that each worker thread has a unique , > non-overlapping set of VMs to work with. How would this help? > As an example, {VM_ID: 1, 6,11,16,21 ..} are handled by the same worker > thread. If this particular worker thread cannot find the requisite VM's > lock, it will move on to the event list for the next VM and so on. The use > of pthread_trylock() ensures that the worker thread will never be stuck > forever. No, I think this isn't the right approach at all. You could end up having all VM's handled with one thread, with others being idle. I think the right approach will be to have a dynamic pool, which will handle incomming events. In case when two events for a single VM should be handled in parallel, the same thread should pick them up in order they arrived. In that way, you will have at most a thread per VM, while normally you will have only one. </blockquote></div> </div><div class="gmail_extra">I agree that dynamic threadpool is helpful when there are events from distinct VMs that need to be processed at the same time. </div><div class="gmail_extra">But I am also concerned about efficiently using the threads in this pool. If we have a few threads only contend on per-VM locks until the RPCs for that VM complete, it is not a very efficient use of resources. I would rather have this thread drop handling of this VM's events and do something useful while it is unable to grab this VM's lock. </div><div class="gmail_extra">This is the reason I wanted to load-balance incoming events by VM IDs and hash them onto distinct threads. The idea was that a pthread always has something else to take up if the current Vm's lock is not available. Would you have some suggestions on improving the efficacy of the thread pool as a whole ? </div></div>