[Libguestfs] Memory corruption when testing nbdkit python plugin with nbd-tester-client?

[Carl-Daniel Hailfinger]

On 26.09.2016 14:29, Richard W.M. Jones wrote:
> On Mon, Sep 26, 2016 at 02:18:02PM +0200, Carl-Daniel Hailfinger wrote:
>> Hi,
>>
>> has anyone ever run "make check" from nbd against nbdkit with a python
>> plugin? I usually get segfaults during such a run, and sometimes various
>> other errors happen before the segfault, suggesting that some memory
>> corruption is underway.
>> AFAICS a pure python plugin should not be able to cause memory corruption.
> Correct, a python plugin should not cause memory corruption,
> and nbdkit shouldn't segfault ever.
>
> Did you get a stack trace from C (not from Python)?

The core files were useless, but I ran nbdkit in gdb and got something...

(gdb) run -n -f /software/nbdkit/plugins/python/.libs/nbdkit-python-plugin.so script=/software/nbdkit/plugins/python/ssd-ftl-wearleveling.py disksize=52428800
Starting program: /software/nbdkit/src/nbdkit -n -f /software/nbdkit/plugins/python/.libs/nbdkit-python-plugin.so script=/software/nbdkit/plugins/python/ssd-ftl-wearleveling.py disksize=52428800
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, py_config (key=0x7fffffffe2e0 "script", value=0x7fffffffe2e7 "/software/nbdkit/plugins/python/ssd-ftl-wearleveling.py") at python.c:107
107     {
(gdb) info shared
>From                To                  Syms Read   Shared Object Library
0x00007ffff7ddaae0  0x00007ffff7df5490  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7bd6ed0  0x00007ffff7bd79ce  Yes         /lib/x86_64-linux-gnu/libdl.so.2
0x00007ffff79bd9f0  0x00007ffff79ca471  Yes         /lib/x86_64-linux-gnu/libpthread.so.0
0x00007ffff7612520  0x00007ffff7757183  Yes         /lib/x86_64-linux-gnu/libc.so.6
0x00007ffff73f0000  0x00007ffff73f13e7  Yes         /software/nbdkit/plugins/python/.libs/nbdkit-python-plugin.so
0x00007ffff6ed70c0  0x00007ffff7065782  Yes (*)     /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
0x00007ffff6c73e00  0x00007ffff6c83bf8  Yes         /lib/x86_64-linux-gnu/libz.so.1
0x00007ffff6a6ff10  0x00007ffff6a70804  Yes         /lib/x86_64-linux-gnu/libutil.so.1
0x00007ffff676e610  0x00007ffff67dd056  Yes         /lib/x86_64-linux-gnu/libm.so.6
(*): Shared library is missing debugging information.
(gdb) c
Continuing.
[New Thread 0x7ffff62eb700 (LWP 24498)]
[Thread 0x7ffff62eb700 (LWP 24498) exited]
[New Thread 0x7ffff62eb700 (LWP 24516)]
[Thread 0x7ffff62eb700 (LWP 24516) exited]
[New Thread 0x7ffff62eb700 (LWP 24519)]
[Thread 0x7ffff62eb700 (LWP 24519) exited]
[New Thread 0x7ffff62eb700 (LWP 24536)]
[Thread 0x7ffff62eb700 (LWP 24536) exited]
[New Thread 0x7ffff62eb700 (LWP 24549)]
[Thread 0x7ffff62eb700 (LWP 24549) exited]
[New Thread 0x7ffff62eb700 (LWP 24561)]
[New Thread 0x7ffff4c26700 (LWP 24580)]
nbdkit: python[6]: error: ssd-ftl-wearleveling.py: callback failed: close
[Thread 0x7ffff62eb700 (LWP 24561) exited]
nbdkit: python[7]: error: invalid flush request: expecting offset and length == 0
nbdkit: python[7]: error: write reply: Connection reset by peer

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4c26700 (LWP 24580)]
0x00007ffff7015306 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
(gdb) bt
#0  0x00007ffff7015306 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#1  0x00007ffff7018059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007ffff7018059 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007ffff701954d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007ffff7017dd8 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007ffff701954d in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007ffff704e6d0 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007ffff6fbad43 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007ffff7033577 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007ffff73f0af0 in py_close (handle=0x625338) at python.c:234
#10 0x0000000000405803 in plugin_close (conn=conn at entry=0x7fffe8000910) at plugins.c:377
#11 0x00000000004037ec in free_connection (conn=0x7fffe8000910) at connections.c:147
#12 0x0000000000404476 in _handle_single_connection (sockout=<optimized out>, sockin=<optimized out>) at connections.c:99
#13 handle_single_connection (sockin=<optimized out>, sockout=<optimized out>) at connections.c:109
#14 0x0000000000405e73 in start_thread (datav=0x7fffffffdd40) at sockets.c:220
#15 0x00007ffff79c0184 in start_thread (arg=0x7ffff4c26700) at pthread_create.c:312
#16 0x00007ffff76ed37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Regards,
Carl-Daniel


>
> Rich.
>
>> Examples of nbdkit logs for running "make check" or subsets of it from
>> nbd against nbdkit:
>>
>> nbdkit -n -f python script=ssd-ftl-wearleveling.py disksize=52428800
>> nbdkit: python[6]: error: ssd-ftl-wearleveling.py: callback failed: close
>> nbdkit: python[7]: error: invalid flush request: expecting offset and
>> length == 0
>> nbdkit: python[7]: error: write reply: Connection reset by peer
>> Segmentation fault
>>
>> nbdkit -n -f python script=ssd-ftl-wearleveling.py disksize=52428800
>> Fatal Python error: GC object already tracked
>> Segmentation fault
>>
>> nbdkit -n -f python script=ssd-ftl-wearleveling.py disksize=52428800
>> Segmentation fault
>>
>> nbdkit -n -f python script=ssd-ftl-wearleveling.py disksize=52428800
>> nbdkit: python[1]: error: invalid flush request: expecting offset and
>> length == 0
>> nbdkit: python[1]: error: write reply: Broken pipe
>> nbdkit: python[3]: error: client sent NBD_OPT_ABORT to abort the connection
>> nbdkit: python[6]: error: ssd-ftl-wearleveling.py: callback failed: close
>> Traceback (most recent call last):
>>   File "ssd-ftl-wearleveling.py", line 221, in close
>>     pickle.dump(lbahist, f)
>>   File "/usr/lib/python2.7/pickle.py", line 1370, in dump
>> Segmentation fault (core dumped)
>>
>> My close function just dumps the disk image and write access history
>> with python pickle. It does not have any return statement, and because
>> of that I seriously doubt that said close function can fail.
>>
>> I'm now trying to replicate this with example.py. Will report back.
>>
>> Regards,
>> Carl-Daniel
>>
>> _______________________________________________
>> Libguestfs mailing list
>> Libguestfs at redhat.com
>> https://www.redhat.com/mailman/listinfo/libguestfs