[libvirt] [PATCH v4 00/23] Introduce metadata locking
Bjoern Walk
bwalk at linux.ibm.com
Wed Sep 12 11:17:01 UTC 2018
Michal Privoznik <mprivozn at redhat.com> [2018-09-12, 11:32AM +0200]:
> On 09/12/2018 07:19 AM, Bjoern Walk wrote:
> > Michal Privoznik <mprivozn at redhat.com> [2018-09-10, 11:36AM +0200]:
> >> Technically, this is v4 of:
> >>
> >> https://www.redhat.com/archives/libvir-list/2018-August/msg01627.html
> >>
> >> However, this is implementing different approach than any of the
> >> previous versions.
> >>
> >> One of the problems with previous version was that it was too
> >> complicated. The main reason for that was that we could not close the
> >> connection whilst there was a file locked. So we had to invent a
> >> mechanism that would prevent that (on the client side).
> >>
> >> These patches implement different approach. They rely on secdriver's
> >> transactions which bring all the paths we want to label into one place
> >> so that they can be relabelled within different namespace.
> >> I'm extending this idea so that transactions run all the time
> >> (regardless of domain namespacing) and only at the very last moment is
> >> decided which namespace would the relabeling run in.
> >>
> >> Metadata locking is then as easy as putting lock/unlock calls around one
> >> function.
> >>
> >> You can find the patches at my github too:
> >>
> >> https://github.com/zippy2/libvirt/tree/disk_metadata_lock_v4_alt
> >
> > Hey Michal,
> >
> > is was running a quick test with this patch series with two domains
> > sharing a disk image without <shareable/> and SELinux enabled. When
> > starting the second domain, the whole libvirtd daemon hangs for almost a
> > minute until giving the error that the image is locked. I haven't
> > debugged it yet to figure out what happens.
>
> Is this on two different hosts or one?
On the same host.
> I'm unable to reproduce, so can you please attach debugger and share 't
> a a bt' output with me?
(gdb) thread apply all bt
Thread 17 (Thread 0x3ff48dff910 (LWP 193353)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff6678c5a8 in udevEventHandleThread (opaque=<optimized out>) at node_device/node_device_udev.c:1603
#3 0x000003ff8c6bad16 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 16 (Thread 0x3ff4acfe910 (LWP 193312)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 15 (Thread 0x3ff4b4ff910 (LWP 193311)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0x3ff64efd910 (LWP 193310)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0x3ff656fe910 (LWP 193309)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0x3ff65eff910 (LWP 193308)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0x3ff67fff910 (LWP 193307)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0x3ff84bf7910 (LWP 193306)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0x3ff853f8910 (LWP 193305)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x3ff85bf9910 (LWP 193304)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x3ff863fa910 (LWP 193303)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x3ff86bfb910 (LWP 193302)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x3ff873fc910 (LWP 193301)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x3ff87bfd910 (LWP 193300)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x3ff883fe910 (LWP 193299)):
#0 0x000003ff8b7113d4 in read () from /lib64/libpthread.so.0
#1 0x000003ff8c65a648 in saferead () from /lib64/libvirt.so.0
#2 0x000003ff8c65a718 in saferead_lim () from /lib64/libvirt.so.0
#3 0x000003ff8c65abe4 in virFileReadHeaderFD () from /lib64/libvirt.so.0
#4 0x000003ff8c69cb5a in virProcessRunInMountNamespace () from /lib64/libvirt.so.0
#5 0x000003ff8c767bd6 in virSecuritySELinuxTransactionCommit () from /lib64/libvirt.so.0
#6 0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0
#7 0x000003ff8c75bb70 in virSecurityStackTransactionCommit () from /lib64/libvirt.so.0
#8 0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0
#9 0x000003ff6612c492 in qemuSecuritySetAllLabel () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#10 0x000003ff660bcfd6 in qemuProcessLaunch () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#11 0x000003ff660c0916 in qemuProcessStart () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#12 0x000003ff66124a00 in qemuDomainObjStart.constprop.52 () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#13 0x000003ff66125030 in qemuDomainCreateWithFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#14 0x000003ff8c87bfca in virDomainCreate () from /lib64/libvirt.so.0
#15 0x000002aa1f2d516c in remoteDispatchDomainCreate (server=<optimized out>, msg=0x2aa4c488b00, args=<optimized out>, rerr=0x3ff883fdae8, client=<optimized out>) at remote/remote_daemon_dispatch_stubs.h:4434
#16 remoteDispatchDomainCreateHelper (server=<optimized out>, client=<optimized out>, msg=0x2aa4c488b00, rerr=0x3ff883fdae8, args=<optimized out>, ret=0x3ff78004c80) at remote/remote_daemon_dispatch_stubs.h:4410
#17 0x000003ff8c78bc80 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#18 0x000003ff8c792aba in virNetServerHandleJob () from /lib64/libvirt.so.0
#19 0x000003ff8c6bbb16 in virThreadPoolWorker () from /lib64/libvirt.so.0
#20 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#21 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#22 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x3ff88bff910 (LWP 193298)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x3ff8cbd7970 (LWP 193297)):
#0 0x000003ff8b5ee502 in poll () from /lib64/libc.so.6
#1 0x000003ff8c65629c in virEventPollRunOnce () from /lib64/libvirt.so.0
#2 0x000003ff8c654caa in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#3 0x000003ff8c7921de in virNetDaemonRun () from /lib64/libvirt.so.0
#4 0x000002aa1f2a4c12 in main (argc=<optimized out>, argv=<optimized out>) at remote/remote_daemon.c:1461
> When I try to reproduce I always get one domain running and the other
> failing to start because qemu is unable to do its locking. When I put
> <shareable/> in both, they start successfully.
Yes, I get the same (expected) behaviour, just with the timeout.
> TBH, I don't run SELinux enabled host so I'm testing DAC only, but the
> changes to the code are merely the same. So I'm wondering if this is
> really an issue in my SELinux impl or somewhere else.
>
> BTW: The one minute timeout comes from patch 16/23:
>
> #define LOCK_ACQUIRE_TIMEOUT 60
>
> Michal
It's entirely possible that my setup is broken, this was just a quick
test. If you can't reproduce it or make sense out of the backtrace then
it's fine, I will try to put some time to debug it next week. I was just
interested if this would fix a long-standing bug where this scenario
would crash the first domain because the label changed.
Bjoern
--
IBM Systems
Linux on Z & Virtualization Development
------------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Schönaicher Str. 220, 71032 Böblingen
Phone: +49 7031 16 1819
------------------------------------------------------------------------
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 902 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20180912/a8f94754/attachment-0001.sig>
More information about the libvir-list
mailing list