[libvirt] [PATCH 4/7] libvirt: support an "embed" URI path selector for opening drivers

Thu Dec 19 11:15:50 UTC 2019

On Tue, Dec 17, 2019 at 12:41:27PM -0500, Cole Robinson wrote:
> On 12/2/19 10:03 AM, Daniel P. Berrangé wrote:
> > The driver URI scheme:
> > 
> >   "$drivername:///embed?root=/some/path"
> > 
> > enables a new way to use the drivers by embedding them directly in the
> > calling process. To use this the process must have a thread running the
> > libvirt event loop. This URI will then cause libvirt to dynamically load
> > the driver module and call its global initialization function. This
> > syntax is applicable to any driver, but only those will have been
> > modified to support a custom root directory and embed URI path will
> > successfully open.
> > 
> > The application can now make normal libvirt API calls which are all
> > serviced in-process with no RPC layer involved.
> > 
> > It is required to specify an explicit root directory, and locks will be
> > acquired on this directory to avoid conflicting with another app that
> > might accidentally pick the same directory.
> > 
> > Use of '/' is not explicitly forbidden, but note that the file layout
> > used underneath the embedded driver root does not match the file
> > layout used by system/session mode drivers. So this cannot be used as
> > a backdoor to interact with, or fake, the system/session mode drivers.
> > 
> > Libvirt will create arbitrary files underneath this root directory. The
> > root directory can be kept untouched across connection open attempts if
> > the application needs persistence. The application is responsible for
> > purging everything underneath this root directory when finally no longer
> > required.
> > 
> > Even when a virt driver is used in embedded mode, it is still possible
> > for it to in turn use functionality that calls out to other secondary
> > drivers in libvirtd. For example an embedded instance of QEMU can open
> > the network, secret or storage drivers in the system libvirtd.
> > 
> > That said, the application would typically want to at least open an
> > embedded secret driver ("secret:///embed?root=/some/path"). Note that
> > multiple different embedded drivers can use the same root prefix and
> > co-operate just as they would inside a normal libvirtd daemon.
> > 
> > A key thing to note is that for this to work, the application that links
> > to libvirt *MUST* be built with -Wl,--export-dynamic to ensure that
> > symbols from libvirt.so are exported & thus available to the dynamically
> > loaded driver module. If libvirt.so itself was dynamically loaded then
> > RTLD_GLOBAL must be passed to dlopen().
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange at redhat.com>
> > ---
> >  src/driver-state.h |  1 +
> >  src/driver.h       |  2 ++
> >  src/libvirt.c      | 72 ++++++++++++++++++++++++++++++++++++++++++++--
> >  3 files changed, 73 insertions(+), 2 deletions(-)

> It would be nice if this logic was moved to a separate function
> 
> I've hit a couple issues in testing, not sure if/where the fixes will
> live, so I'll just mention them here. Also the reviewed patches are
> pushable IMO
> 
> I tried this code:
> 
> from gi.repository import LibvirtGLib
> 
> import libvirt
> 
> 
> 
> LibvirtGLib.init(None)
> 
> LibvirtGLib.event_register()
> 
> 
> 
> conn1 = libvirt.open("qemu:///embed?root=/tmp/foo")
> 
> conn2 = libvirt.open("qemu:///embed?root=/tmp/bar")
> 
> print(conn1.listAllDomains())
> 
> print(conn2.listAllDomains())
> 
> 
> With /tmp/foo populated with a VM: both connections see the same values.
> So this should be rejected. Even trying to close conn1 fully and open a
> new embed root will only see the old root. So maybe this needs knowledge
> in the driver lookup.

Right, so because of our design with the single global driver struct, we
can only have 1 instance of the driver active per process.  So yeah, we
need to enforce this in some place or other.

It should be valid to open multiple connections to the same embedded
driver instance directory though.

> The second issue: testing with virt-manager, everything locks up with
> OpenGraphicsFD:
> 
> #0  0x00007ffff7a7f07a in pthread_cond_timedwait@@GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
> #1  0x00007fffe94be113 in virCondWaitUntil (c=c at entry=0x7fffc8071f98,
> m=m at entry=0x7fffc8071ed0, whenms=whenms at entry=1576602286073) at
> /home/crobinso/src/libvirt/src/util/virthread.c:159
> #2  0x00007fffe44fc549 in qemuDomainObjBeginJobInternal
> (driver=driver at entry=0x7fffc8004f30, obj=0x7fffc8071ec0,
> job=job at entry=QEMU_JOB_MODIFY,
> agentJob=agentJob at entry=QEMU_AGENT_JOB_NONE,
> asyncJob=asyncJob at entry=QEMU_ASYNC_JOB_NONE, nowait=nowait at entry=false)
> at /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9357
> #3  0x00007fffe4500aa1 in qemuDomainObjBeginJob
> (driver=driver at entry=0x7fffc8004f30, obj=<optimized out>,
> job=job at entry=QEMU_JOB_MODIFY) at
> /home/crobinso/src/libvirt/src/qemu/qemu_domain.c:9521
> #4  0x00007fffe4582572 in qemuDomainOpenGraphicsFD (dom=<optimized out>,
> idx=<optimized out>, flags=0) at
> /home/crobinso/src/libvirt/src/qemu/qemu_driver.c:18990
> #5  0x00007fffe968699c in virDomainOpenGraphicsFD (dom=0x7fffd0005830,
> idx=0, flags=0) at /home/crobinso/src/libvirt/src/libvirt-domain.c:10664
> #6  0x00007fffe98cc6b1 in libvirt_virDomainOpenGraphicsFD () at
> /usr/lib64/python3.7/site-packages/libvirtmod.cpython-37m-x86_64-linux-gnu.so
> 
> 
> I didn't dig into it any more than that. Otherwise in some mild testing
> I'm surprised how much things seemed to 'just work' :)

Hard to tell but my guess is that there's probably some event loop
interaction causing us problems. Obviously this stack trace is waiting
for the job lock.  I'm guessing that the job lock is held by another
API call made from the event loop thread. Will investigate it some more.

In libvirtd no libvirt APIs calls are ever made from the event loop
thread, they can only be made from worker threads. We'll need to make
it clear that apps using the embedded driver must likewise ensure that
libvirt APIs calls are *NEVER* made from the event loop thread. This
is good practice even when using traditional libvirt connections,
because the RPC calls to libvirtd can take an arbitrary  amount of
time to complete & so make the event loop response very poor.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|