[libvirt-users] libvirtd hangs
Artem Likhachev
a.likhachev at marosnet.ru
Thu Jan 25 04:17:54 UTC 2018
Ján Tomko wrote 2018-01-18 14:27:
> On Thu, Jan 18, 2018 at 11:30:16AM +0700, Artem Likhachev wrote:
>> # strace -p 5786
>> read(53, "\0\0\0\34", 4) = 4
>> read(53, "keep\0\0\0\1\0\0\0\2\0\0\0\2\0\0\0\0\0\0\0\0", 24) = 24
>> poll([{fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=12,
>> events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN},
>> {fd=15,
>> events=POLLIN}, {fd=19, events=POLLIN}, {fd=23,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=21,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=27, events=POLLIN|POLLERR|POLLHUP}, {fd=25,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=22,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=24, events=POLLIN|POLLERR|POLLHUP}, {fd=26,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=29,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=31,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=33,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=32, events=POLLIN|POLLERR|POLLHUP}, {fd=36,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=35,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=39, events=POLLIN|POLLERR|POLLHUP}, {fd=40,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=41,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=44, events=POLLIN|POLLERR|POLLHUP}, {fd=42,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=43,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=48, events=POLLIN|POLLERR|POLLHUP}, {fd=49,
>> events=POLLIN|POLLERR|POLLHUP}, {fd=59,
>> events=POLLIN|POLLERR|POLLHUP},
>> {fd=46, events=POLLIN|POLLERR|POLLHUP}, {fd=50,
>> events=POLLIN|POLLERR|POLLHUP}, ...], 43, 5000
>>
>> # gdb -p 5786
>> (gdb) thread apply all bt
>>
>> Thread 17 (Thread 0x7f411a9d7700 (LWP 5788)):
>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1 0x00007f4129c2a2e6 in virCondWait (c=c at entry=0x7f412b18ebb8,
>> m=m at entry=0x7f412b18eb90) at util/virthread.c:154
>> #2 0x00007f4129c2ada3 in virThreadPoolWorker
>> (opaque=opaque at entry=0x7f412b183ab0) at util/virthreadpool.c:124
>> #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at
>> util/virthread.c:206
>> #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a9d7700) at
>> pthread_create.c:308
>> #5 0x00007f4126d6273d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>>
>> Thread 16 (Thread 0x7f411a1d6700 (LWP 5789)):
>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1 0x00007f4129c2a2e6 in virCondWait (c=c at entry=0x7f412b18ebb8,
>> m=m at entry=0x7f412b18eb90) at util/virthread.c:154
>> #2 0x00007f4129c2ada3 in virThreadPoolWorker
>> (opaque=opaque at entry=0x7f412b183a00) at util/virthreadpool.c:124
>> #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at
>> util/virthread.c:206
>> #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a1d6700) at
>> pthread_create.c:308
>> #5 0x00007f4126d6273d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>>
>> Thread 15 (Thread 0x7f41199d5700 (LWP 5790)):
>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1 0x00007f4129c2a2e6 in virCondWait (c=c at entry=0x7f412b18ebb8,
>> m=m at entry=0x7f412b18eb90) at util/virthread.c:154
>> #2 0x00007f4129c2ada3 in virThreadPoolWorker
>> (opaque=opaque at entry=0x7f412b183950) at util/virthreadpool.c:124
>> #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at
>> util/virthread.c:206
>> #4 0x00007f4127033dc5 in start_thread (arg=0x7f41199d5700) at
>> pthread_create.c:308
>> #5 0x00007f4126d6273d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>>
>> Thread 14 (Thread 0x7f41191d4700 (LWP 5791)):
>> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1 0x00007f4129c2a2e6 in virCondWait (c=c at entry=0x7f412b18ebb8,
>> m=m at entry=0x7f412b18eb90) at util/virthread.c:154
>> #2 0x00007f4129c2ada3 in virThreadPoolWorker
>> (opaque=opaque at entry=0x7f412b1838a0) at util/virthreadpool.c:124
>> #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at
>> util/virthread.c:206
>> #4 0x00007f4127033dc5 in start_thread (arg=0x7f41191d4700) at
>> pthread_create.c:308
>> #5 0x00007f4126d6273d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>>
>
> These are all idle workers.
>
>> Thread 13 (Thread 0x7f41189d3700 (LWP 5792)):
>> ---Type <return> to continue, or q <return> to quit---
>>
>
> You omitted the backtrace of the other threads.
>
> Jan
Please, specify what do you mean? What should I do? I'm not familiar
with gdb at good level.
Here is strace output on virsh:
read(6, "", 7645) = 0
close(6) = 0
getuid() = 0
geteuid() = 0
geteuid() = 0
getuid() = 0
geteuid() = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 6
connect(6, {sa_family=AF_LOCAL,
sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(6, {sa_family=AF_LOCAL, NULL}, [2]) = 0
futex(0x7fd1b09dcfe8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fcntl(6, F_GETFD) = 0
fcntl(6, F_SETFD, FD_CLOEXEC) = 0
fcntl(6, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
futex(0x7fd1b09dcee8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
pipe2([7, 8], O_CLOEXEC) = 0
write(5, "\0", 1) = 1
futex(0x7fd1b09dc280, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fd1b09dcfb0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fd1b09dcf00, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(5, "\0", 1) = 1
futex(0x7fd1b09dc280, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=6, events=POLLOUT}, {fd=7, events=POLLIN}], 2, -1) = 1
([{fd=6, revents=POLLOUT}])
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(6, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0",
28) = 28
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 2, -1
^Cstrace: Process 16202 detached
And some errors at libvirtd log sometimes:
янв 22 03:10:03 hostname.net libvirtd[22256]: 2018-01-22
00:10:03.639+0000: 22256: error : virNetSocketReadWire:1808 : End of
file while reading data: Input/output error
янв 22 03:15:03 hostname.net libvirtd[22256]: 2018-01-22
00:15:03.802+0000: 22256: error : virNetSocketReadWire:1808 : End of
file while reading data: Input/output error
янв 22 04:45:03 hostname.net libvirtd[22256]: 2018-01-22
01:45:03.210+0000: 22256: error : virNetSocketReadWire:1808 : End of
file while reading data: Input/output error
янв 22 09:10:03 hostname.net libvirtd[22256]: 2018-01-22
06:10:03.829+0000: 22256: error : virNetSocketReadWire:1808 : End of
file while reading data: Input/output error
---
Artem Likhachev
More information about the libvirt-users
mailing list