<div dir="ltr">Hi Marc,<div>Currently the block job handling needs to be sorted out before events can assume independent handling from RPC contexts. <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sorry, I have not been able to revisit this in the past 2 months, but this is something I would very much like to fix. I will try looking up the exact dependency of the block layer so that this series could make some progress.</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">regards,</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Prerna</div></div><div class="gmail_extra"> <div class="gmail_quote">On Wed, Mar 21, 2018 at 1:24 PM, Marc Hartmayer <<a href="mailto:mhartmay@linux.vnet.ibm.com" target="_blank">mhartmay@linux.vnet.ibm.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, Oct 24, 2017 at 07:34 PM +0200, Prerna Saxena <<a href="mailto:saxenap.ltc@gmail.com">saxenap.ltc@gmail.com</a>> wrote: <div><div class="h5">> As noted in > <a href="https://www.redhat.com/archives/libvir-list/2017-May/msg00016.html" rel="noreferrer" target="_blank">https://www.redhat.com/archives/libvir-list/2017-May/msg00016.html</a> > libvirt-QEMU driver handles all async events from the main loop. > Each event handling needs the per-VM lock to make forward progress. In > the case where an async event is received for the same VM which has an > RPC running, the main loop is held up contending for the same lock. > > This impacts scalability, and should be addressed on priority. > > Note that libvirt does have a 2-step deferred handling for a few event > categories, but (1) That is insufficient since blockign happens before > the handler could disambiguate which one needs to be posted to this > other queue. > (2) There needs to be homogeniety. > > The current series builds a framework for recording and handling VM > events. > It initializes per-VM event queue, and a global event queue pointing to > events from all the VMs. Event handling is staggered in 2 stages: > - When an event is received, it is enqueued in the per-VM queue as well > as the global queues. > - The global queue is built into the QEMU Driver as a threadpool > (currently with a single thread). > - Enqueuing of a new event triggers the global event worker thread, which > then attempts to take a lock for this event's VM. > - If the lock is available, the event worker runs the function handling > this event type. Once done, it dequeues this event from the global > as well as per-VM queues. > - If the lock is unavailable(ie taken by RPC thread), the event worker > thread leaves this as-is and picks up the next event. > - Once the RPC thread completes, it looks for events pertaining to the > VM in the per-VM event queue. It then processes the events serially > (holding the VM lock) until there are no more events remaining for > this VM. At this point, the per-VM lock is relinquished. > > Patch Series status: > Strictly RFC only. No compilation issues. I have not had a chance to > (stress) test it after rebase to latest master. > Note that documentation and test coverage is TBD, since a few open > points remain. > > Known issues/ caveats: > - RPC handling time will become non-deterministic. > - An event will only be "notified" to a client once the RPC for same VM completes. > - Needs careful consideration in all cases where a QMP event is used to > "signal" an RPC thread, else will deadlock. > > Will be happy to drive more discussion in the community and completely > implement it. > > Prerna Saxena (8): > Introduce virObjectTrylock() > QEMU Event handling: Introduce async event helpers in qemu_event.[ch] > Setup global and per-VM event queues. Also initialize per-VM queues > when libvirt reconnects to an existing VM. > Events: Allow monitor to "enqueue" events to a queue. Also introduce a > framework of handlers for each event type, that can be called when > the handler is running an event. > Events: Plumb event handling calls before a domain's APIs complete. > Code refactor: Move helper functions of doCoreDump*, syncNicRxFilter*, > and qemuOpenFile* to qemu_process.[ch] > Fold back the 2-stage event implementation for a few events : > Watchdog, Monitor EOF, Serial changed, Guest panic, Nic RX filter > changed .. into single level. > Initialize the per-VM event queues in context of domain init. > > src/Makefile.am | 1 + > src/conf/domain_conf.h | 3 + > src/libvirt_private.syms | 1 + > src/qemu/qemu_conf.h | 4 + > src/qemu/qemu_driver.c | 1710 +++++++---------------------------- > src/qemu/qemu_event.c | 317 +++++++ > src/qemu/qemu_event.h | 231 +++++ > src/qemu/qemu_monitor.c | 592 ++++++++++-- > src/qemu/qemu_monitor.h | 80 +- > src/qemu/qemu_monitor_json.c | 291 +++--- > src/qemu/qemu_process.c | 2031 ++++++++++++++++++++++++++++++++++-------- > src/qemu/qemu_process.h | 88 ++ > src/util/virobject.c | 26 + > src/util/virobject.h | 4 + > src/util/virthread.c | 5 + > src/util/virthread.h | 1 + > tests/qemumonitortestutils.c | 2 +- > 17 files changed, 3411 insertions(+), 1976 deletions(-) > create mode 100644 src/qemu/qemu_event.c > create mode 100644 src/qemu/qemu_event.h > > -- > 2.9.5 > </div></div>> -- > libvir-list mailing list > <a href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/libvir-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/libvir-list</a> Hey Prerna, any updates so far? :) I’m just curious about the status of this series as it would fix a performance problem we’ve. Thank you. <div class="HOEnZb"><div class="h5"> Beste Grüße / Kind regards Marc Hartmayer IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294 </div></div></blockquote></div> </div>