[libvirt] [[RFC] 0/8] Implement async QEMU event handling in libvirtd.

[Prerna Saxena]

As noted in
https://www.redhat.com/archives/libvir-list/2017-May/msg00016.html
libvirt-QEMU driver handles all async events from the main loop.
Each event handling needs the per-VM lock to make forward progress. In
the case where an async event is received for the same VM which has an
RPC running, the main loop is held up contending for the same lock.

This impacts scalability, and should be addressed on priority.

Note that libvirt does have a 2-step deferred handling for a few event
categories, but (1) That is insufficient since blockign happens before
the handler could disambiguate which one needs to be posted to this
other queue.
(2) There needs to be homogeniety.

The current series builds a framework for recording and handling VM
events.
It initializes per-VM event queue, and a global event queue pointing to
events from all the VMs. Event handling is staggered in 2 stages:
- When an event is received, it is enqueued in the per-VM queue as well
  as the global queues.
- The global queue is built into the QEMU Driver as a threadpool
  (currently with a single thread).
- Enqueuing of a new event triggers the global event worker thread, which
  then attempts to take a lock for this event's VM.
    - If the lock is available, the event worker runs the function handling
      this event type. Once done, it dequeues this event from the global
      as well as per-VM queues.
    - If the lock is unavailable(ie taken by RPC thread), the event worker 
      thread leaves this as-is and picks up the next event.
- Once the RPC thread completes, it looks for events pertaining to the
  VM in the per-VM event queue. It then processes the events serially
  (holding the VM lock) until there are no more events remaining for
  this VM. At this point, the per-VM lock is relinquished.

Patch Series status:
Strictly RFC only. No compilation issues. I have not had a chance to
(stress) test it after rebase to latest master.
Note that documentation and test coverage is TBD, since a few open
points remain.

Known issues/ caveats:
- RPC handling time will become non-deterministic.
- An event will only be "notified" to a client once the RPC for same VM completes.
- Needs careful consideration in all cases where a QMP event is used to
  "signal" an RPC thread, else will deadlock.

Will be happy to drive more discussion in the community and completely
implement it.

Prerna Saxena (8):
  Introduce virObjectTrylock()
  QEMU Event handling: Introduce async event helpers in qemu_event.[ch]
  Setup global and per-VM event queues. Also initialize per-VM queues
    when libvirt reconnects to an existing VM.
  Events: Allow monitor to "enqueue" events to a queue. Also introduce a
    framework of handlers for each event type, that can be called when
    the handler is running an event.
  Events: Plumb event handling calls before a domain's APIs complete.
  Code refactor: Move helper functions of doCoreDump*, syncNicRxFilter*,
    and qemuOpenFile* to qemu_process.[ch]
  Fold back the 2-stage event implementation for a few events :
    Watchdog, Monitor EOF, Serial changed, Guest panic, Nic RX filter
    changed .. into single level.
  Initialize the per-VM event queues in context of domain init.

 src/Makefile.am              |    1 +
 src/conf/domain_conf.h       |    3 +
 src/libvirt_private.syms     |    1 +
 src/qemu/qemu_conf.h         |    4 +
 src/qemu/qemu_driver.c       | 1710 +++++++----------------------------
 src/qemu/qemu_event.c        |  317 +++++++
 src/qemu/qemu_event.h        |  231 +++++
 src/qemu/qemu_monitor.c      |  592 ++++++++++--
 src/qemu/qemu_monitor.h      |   80 +-
 src/qemu/qemu_monitor_json.c |  291 +++---
 src/qemu/qemu_process.c      | 2031 ++++++++++++++++++++++++++++++++++--------
 src/qemu/qemu_process.h      |   88 ++
 src/util/virobject.c         |   26 +
 src/util/virobject.h         |    4 +
 src/util/virthread.c         |    5 +
 src/util/virthread.h         |    1 +
 tests/qemumonitortestutils.c |    2 +-
 17 files changed, 3411 insertions(+), 1976 deletions(-)
 create mode 100644 src/qemu/qemu_event.c
 create mode 100644 src/qemu/qemu_event.h

-- 
2.9.5