[Libguestfs] [nbdkit PATCH v5 4/4] vddk: Drive library loading from libdir parameter.

[Eric Blake]

On 2/17/20 12:17 PM, Richard W.M. Jones wrote:
> Unfortunately this series doesn't work with the real VDDK library :-(
> 
> I see:
> 
>    nbdkit: debug: vddk: config_complete
>    nbdkit: debug: dlopen shim prefix set to /home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64
> 
> Then a little bit later during VDDK initialization it fails with:
> 
>    nbdkit: debug: VixDiskLib: Could not load default plugins from /home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so: Cannot open library: libcrypto.so.1.0.2: cannot open shared object file: No such file or directory.
> 
> I was curious if the replacement dlopen() was being called at all, and
> it is certainly called from plugins/vddk/vddk.c:
> 
>    nbdkit: debug: dlopen: /home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.6 flags=2
> 
> It's also called from VDDK itself:
> 
>    nbdkit: debug: dlopen: /home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so flags=257

That's nice to know...

> 
> but it's not being explicitly called for libcrypto.so because
> immediately after the above message I see the error:
> 
>    nbdkit: debug: VixDiskLib: Could not load default plugins from /home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so: Cannot open library: libcrypto.so.1.0.2: cannot open shared object file: No such file or directory.
> 
> So I guess libcrypto is actually loaded from a DT_NEEDED entry:

...but that's indeed frustrating.

> 
>    $ eu-readelf -d /home/rjones/tmp/vddk-6.7.3/vmware-vix-disklib-distrib/lib64/libdiskLibPlugin.so  | grep NEEDED
>      NEEDED            Shared library: [libpthread.so.0]
>      NEEDED            Shared library: [libstdc++.so.6]
>      NEEDED            Shared library: [libdl.so.2]
>      NEEDED            Shared library: [libcrypto.so.1.0.2]
>      NEEDED            Shared library: [libssl.so.1.0.2]
>      NEEDED            Shared library: [libexpat.so]
>      NEEDED            Shared library: [libz.so.1]
>      NEEDED            Shared library: [libvmacore.so]
>      NEEDED            Shared library: [libvmomi.so]
>      NEEDED            Shared library: [libvim-types.so]
>      NEEDED            Shared library: [librt.so.1]
>      NEEDED            Shared library: [libgcc_s.so.1]
>      NEEDED            Shared library: [libc.so.6]
>      NEEDED            Shared library: [ld-linux-x86-64.so.2]
> 
> which I suppose doesn't actually go through dlopen ...

And we were wondering if the dlopen() with an absolute file name would 
help the dynamic linker alter its search path for its dependencies. 
That may have answered our question, but not the way we wanted. :(

la_objsearch() seems like it should catch these DT_NEEDED entries, but 
given the fact that glibc does not yet support DT_AUDIT for 
single-process rather than global auditing, we're stuck with having to 
set LD_AUDIT in the environment (which is no different than having to 
set LD_LIBRARY_PATH in the environment).


Next idea: how hard would it be to re-exec nbdkit?  Here's what I'm 
envisioning:

nbdkit proper exposes some way to let a plugin query argv[] (necessary, 
so that re-exec gets the same arguments).  Re-exec is only safe at a 
point prior to the end of .config_complete, and must be done in a way 
that does not cause infinite recursion on re-exec attempts.

In vddk's .config_complete, if the key reexec= is absent, libdir=... is 
present, and dlopen()/init of libvixDiskLib.so fails, then we grab argv 
from nbdkit and re-exec 'LD_LIBRARY_PATH=libdir... nbdkit vddk 
same_args... reexec=true'.  Conversely, if key reexec= is present, then 
.config_complete calls setenv() to reset LD_LIBRARY_PATH to its former 
state.  .config_complete runs prior to the point where nbdkit fork()s 
the --run child, so the child will see the original environment rather 
than the one rewritten by the re-exec attempt, limiting our temporary 
LD_LIBRARY_PATH pollution to just the nbdkit process.

In the meantime, patch 3 (delaying from the dlopen() of libvixDiskLib.so 
from .load over to .config_complete) is worthwhile to push now, even if 
it doesn't change the status quo (I didn't change it from what you posted).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org