[libvirt] [[RFC] 0/8] Implement async QEMU event handling in libvirtd.
Prerna
saxenap.ltc at gmail.com
Wed Nov 15 05:40:51 UTC 2017
On Wed, Nov 15, 2017 at 12:07 AM, Marc Hartmayer <
mhartmay at linux.vnet.ibm.com> wrote:
> On Tue, Oct 24, 2017 at 07:34 PM +0200, Prerna Saxena <
> saxenap.ltc at gmail.com> wrote:
> > As noted in
> > https://www.redhat.com/archives/libvir-list/2017-May/msg00016.html
> > libvirt-QEMU driver handles all async events from the main loop.
> > Each event handling needs the per-VM lock to make forward progress. In
> > the case where an async event is received for the same VM which has an
> > RPC running, the main loop is held up contending for the same lock.
>
> What's about the remaining qemuMonitorCallbacks? The main event loop can
> still be 'blocked' by e.g. qemuProcessHandleMonitorEOF if the VM is
> already locked by a worker thread. In fact, we currently have a problem
> with D-Bus which causes a D-Bus call (of a worker thread) to run into
> the timeout of 30 seconds. During these 30 seconds the main event loop
> is stuck.
>
>
EOF handling in the current series is still yet another event, and hence is
done only once worker threads complete.
It needs to go into a priority thread pool, and should wake up RPC workers.
> I tried the patch series and got a segmentation fault:
>
> Thread 1 "libvirtd" received signal SIGSEGV, Segmentation fault.
> 0x000003ff98faa452 in virEnqueueVMEvent (qlist=0x3ff908ce760,
> ev=<optimized out>) at ../../src/qemu/qemu_event.c:153 153
> vmq_entry->ev->ev_id = vmq->last->ev->ev_id + 1;
> (gdb) bt
> #0 0x000003ff98faa452 in virEnqueueVMEvent (qlist=0x3ff908ce760,
> ev=<optimized out>) at ../../src/qemu/qemu_event.c:153
> #1 0x000003ff98fc3564 in qemuProcessEnqueueEvent (mon=<optimized out>,
> vm=<optimized out>, ev=<optimized out>, opaque=0x3ff90548ec0) at
> ../../src/qemu/qemu_process.c:1864
> #2 0x000003ff98fe4804 in qemuMonitorEnqueueEvent
> (mon=mon at entry=0x3ff4c007440, ev=0x2aa1e0104c0) at
> ../../src/qemu/qemu_monitor.c:1325
> #3 0x000003ff98fe7102 in qemuMonitorEmitShutdown
> (mon=mon at entry=0x3ff4c007440, guest=<optimized out>,
> seconds=seconds at entry=1510683878, micros=micros at entry=703956) at
> ../../src/qemu/qemu_monitor.c:1365
> #4 0x000003ff98ffc19a in qemuMonitorJSONHandleShutdown
> (mon=0x3ff4c007440, data=<optimized out>, seconds=1510683878,
> micros=<optimized out>) at ../../src/qemu/qemu_monitor_json.c:552
> #5 0x000003ff98ffbb8a in qemuMonitorJSONIOProcessEvent
> (mon=mon at entry=0x3ff4c007440, obj=obj at entry=0x2aa1e012030) at
> ../../src/qemu/qemu_monitor_json.c:208
> #6 0x000003ff99002138 in qemuMonitorJSONIOProcessLine
> (mon=mon at entry=0x3ff4c007440, line=0x2aa1e010460 "{\"timestamp\":
> {\"seconds\": 1510683878, \"microseconds\": 703956}, \"event\":
> \"SHUTDOWN\"}", msg=msg at entry=0x0) at
> ../../src/qemu/qemu_monitor_json.c:237
> #7 0x000003ff990022b4 in qemuMonitorJSONIOProcess
> (mon=mon at entry=0x3ff4c007440, data=0x2aa1e014bc0 "{\"timestamp\":
> {\"seconds\": 1510683878, \"microseconds\": 703956}, \"event\":
> \"SHUTDOWN\"}\r\n", len=85, msg=msg at entry=0x0) at
> ../../src/qemu/qemu_monitor_json.c:279
> #8 0x000003ff98fe4b44 in qemuMonitorIOProcess
> (mon=mon at entry=0x3ff4c007440) at ../../src/qemu/qemu_monitor.c:443
> #9 0x000003ff98fe5d00 in qemuMonitorIO (watch=<optimized out>,
> fd=<optimized out>, events=0, opaque=0x3ff4c007440) at
> ../../src/qemu/qemu_monitor.c:697
> #10 0x000003ffa68d6442 in virEventPollDispatchHandles (nfds=<optimized
> out>, fds=0x2aa1e013990) at ../../src/util/vireventpoll.c:508
> #11 0x000003ffa68d66c8 in virEventPollRunOnce () at
> ../../src/util/vireventpoll.c:657
> #12 0x000003ffa68d44e4 in virEventRunDefaultImpl () at
> ../../src/util/virevent.c:327
> #13 0x000003ffa6a83c5e in virNetDaemonRun (dmn=0x2aa1dfe3eb0) at
> ../../src/rpc/virnetdaemon.c:838
> #14 0x000002aa1df29cc4 in main (argc=<optimized out>, argv=<optimized
> out>) at ../../daemon/libvirtd.c:1494
>
>
Thanks for trying it out. Let me look into this.
> >
> > This impacts scalability, and should be addressed on priority.
> >
> > Note that libvirt does have a 2-step deferred handling for a few event
> > categories, but (1) That is insufficient since blockign happens before
> > the handler could disambiguate which one needs to be posted to this
> > other queue.
> > (2) There needs to be homogeniety.
> >
> > The current series builds a framework for recording and handling VM
> > events.
> > It initializes per-VM event queue, and a global event queue pointing to
> > events from all the VMs. Event handling is staggered in 2 stages:
> > - When an event is received, it is enqueued in the per-VM queue as well
> > as the global queues.
> > - The global queue is built into the QEMU Driver as a threadpool
> > (currently with a single thread).
> > - Enqueuing of a new event triggers the global event worker thread, which
> > then attempts to take a lock for this event's VM.
> > - If the lock is available, the event worker runs the function
> handling
> > this event type. Once done, it dequeues this event from the global
> > as well as per-VM queues.
> > - If the lock is unavailable(ie taken by RPC thread), the event
> worker
> > thread leaves this as-is and picks up the next event.
> > - Once the RPC thread completes, it looks for events pertaining to the
> > VM in the per-VM event queue. It then processes the events serially
> > (holding the VM lock) until there are no more events remaining for
> > this VM. At this point, the per-VM lock is relinquished.
> >
> > Patch Series status:
> > Strictly RFC only. No compilation issues. I have not had a chance to
> > (stress) test it after rebase to latest master.
> > Note that documentation and test coverage is TBD, since a few open
> > points remain.
> >
> > Known issues/ caveats:
> > - RPC handling time will become non-deterministic.
> > - An event will only be "notified" to a client once the RPC for same VM
> completes.
> > - Needs careful consideration in all cases where a QMP event is used to
> > "signal" an RPC thread, else will deadlock.
> >
> > Will be happy to drive more discussion in the community and completely
> > implement it.
> >
> > Prerna Saxena (8):
> > Introduce virObjectTrylock()
> > QEMU Event handling: Introduce async event helpers in qemu_event.[ch]
> > Setup global and per-VM event queues. Also initialize per-VM queues
> > when libvirt reconnects to an existing VM.
> > Events: Allow monitor to "enqueue" events to a queue. Also introduce a
> > framework of handlers for each event type, that can be called when
> > the handler is running an event.
> > Events: Plumb event handling calls before a domain's APIs complete.
> > Code refactor: Move helper functions of doCoreDump*, syncNicRxFilter*,
> > and qemuOpenFile* to qemu_process.[ch]
> > Fold back the 2-stage event implementation for a few events :
> > Watchdog, Monitor EOF, Serial changed, Guest panic, Nic RX filter
> > changed .. into single level.
> > Initialize the per-VM event queues in context of domain init.
> >
> > src/Makefile.am | 1 +
> > src/conf/domain_conf.h | 3 +
> > src/libvirt_private.syms | 1 +
> > src/qemu/qemu_conf.h | 4 +
> > src/qemu/qemu_driver.c | 1710 +++++++----------------------------
> > src/qemu/qemu_event.c | 317 +++++++
> > src/qemu/qemu_event.h | 231 +++++
> > src/qemu/qemu_monitor.c | 592 ++++++++++--
> > src/qemu/qemu_monitor.h | 80 +-
> > src/qemu/qemu_monitor_json.c | 291 +++---
> > src/qemu/qemu_process.c | 2031 ++++++++++++++++++++++++++++++
> ++++--------
> > src/qemu/qemu_process.h | 88 ++
> > src/util/virobject.c | 26 +
> > src/util/virobject.h | 4 +
> > src/util/virthread.c | 5 +
> > src/util/virthread.h | 1 +
> > tests/qemumonitortestutils.c | 2 +-
> > 17 files changed, 3411 insertions(+), 1976 deletions(-)
> > create mode 100644 src/qemu/qemu_event.c
> > create mode 100644 src/qemu/qemu_event.h
> >
> > --
> > 2.9.5
> >
> > --
> > libvir-list mailing list
> > libvir-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/libvir-list
>
> Beste Grüße / Kind regards
> Marc Hartmayer
>
> IBM Deutschland Research & Development GmbH
> Vorsitzende des Aufsichtsrats: Martina Koederitz
> Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20171115/68238bba/attachment-0001.htm>
More information about the libvir-list
mailing list