<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> <html> <tt>Hi Castor,</tt><tt></tt> <tt>Preliminary tests look good on x86_64, and for the most part ia64.</tt><tt></tt> <tt>On ia64, some modules never load any .data, .data.rel, etc. sections,</tt> <tt>and when that happens the "p" command for their data (d) symbols</tt> <tt>doesn't work. But other modules work just fine? No regression</tt> <tt>though...</tt><tt></tt> <tt>On s390x, we don't stand a prayer, because the modules themselves</tt> <tt>don't seem to contain any debug info? It looks like this:</tt><tt></tt> <tt>crash> mod -l</tt> <tt> MODULE NAME SIZE OBJECT FILE</tt> <tt> 2081a800 dm_mirror 56184 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20826b00 sd_mod 44584 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 2082f100 dasd_fba_mod 29696 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 2084c600 jbd 100688 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20867900 dasd_mod 116892 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20880600 dasd_eckd_mod 84224 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 2088d900 dm_zero 19456 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 208c0b00 ext3 200208 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 208e9600 scsi_transport_fc 67848 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 208f4e00 dm_snapshot 41288 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 2094b900 scsi_mod 233104 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20966100 qdio 103768 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 2097f900 dm_mod 99544 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20993c00 ccwgroup 73472 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 209a5300 cu3088 67080 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 209f5a00 zfcp 236832 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20a4ed00 sg 151600 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20bcbf00 autofs4 254472 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20c67600 qeth 563600 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 20fb0000 sunrpc 1076664 (not loaded) [CONFIG_KALLSYMS]</tt> <tt> 212aa100 ipv6 2300920 (not loaded) [CONFIG_KALLSYMS]</tt> <tt>crash> mod -s ext3</tt> <tt> MODULE NAME SIZE OBJECT FILE</tt> <tt> 208c0b00 ext3 200208 /lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko</tt> <tt>crash> p ext3_file_operations</tt> <tt>ext3_file_operations = $1 = 0</tt> <tt>crash> whatis ext3_file_operations</tt> <tt><data variable, no debug info> ext3_file_operations;</tt> <tt>crash> sym ext3_file_operations</tt> <tt>208bff88 (D) ext3_file_operations</tt> <tt>crash></tt><tt></tt> <tt>The add-symbol-file line looks OK, although the address of</tt> <tt>ext3_file_operations looks like it's in the .data.rel.ro_addr</tt> <tt>section (?):</tt><tt></tt> <tt>add-symbol-file request add-symbol-file /lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko 0x208953a8 -s .exit.text 0x208adfc4 -s .rodata 0x208ae000 -s .rodata.str1.2 0x208af1b8 -s __versions 0x208b1d60 -s .data.rel.ro 0x208bfea0 -s .data.rel 0x208c01f8 -s .data.rel.local 0x208c0740 -s .gnu.linkonce.this_module 0x208c0b00 -s .bss 0x208c4e00</tt> <tt>add symbol table from file "/lib/modules/2.6.18-1.2961.el5/kernel/fs/ext3/ext3.ko" at</tt> <tt> .text_addr = 0x208953a8</tt> <tt> .exit.text_addr = 0x208adfc4</tt> <tt> .rodata_addr = 0x208ae000</tt> <tt> .rodata.str1.2_addr = 0x208af1b8</tt> <tt> __versions_addr = 0x208b1d60</tt> <tt> .data.rel.ro_addr = 0x208bfea0</tt> <tt> .data.rel_addr = 0x208c01f8</tt> <tt> .data.rel.local_addr = 0x208c0740</tt> <tt> .gnu.linkonce.this_module_addr = 0x208c0b00</tt> <tt> .bss_addr = 0x208c4e00</tt><tt></tt> <tt>Anyway, the behavior is identical to what it's always been,</tt> <tt>so there's no regression.</tt><tt></tt> <tt>I'll keep tinkering as I get time...</tt><tt></tt> <tt>Thanks,</tt> <tt> Dave</tt> <tt></tt> Castor Fu wrote: <blockquote TYPE=CITE> <div dir="ltr">Finding the overrun wasn't actually that hard. It's also fixed in the current GDB tree.</div> <div dir="ltr"></div> <div dir="ltr">I've attached a patch which fixes the problem in symfile.c. The point of this patch</div> <div dir="ltr">is to fix loading kernel modules symbol information on 2.6 for those who have not</div> <div dir="ltr">been following this.</div> <div dir="ltr"></div> <div dir="ltr">Hopefully this will work on other platforms too....</div> <div dir="ltr"></div> <div dir="ltr"> -castor</div> <div dir="ltr"></div> <div dir="ltr"></div> <div dir="ltr"> <hr tabIndex=-1></div> <div dir="ltr">From: crash-utility-bounces@redhat.com [<A HREF="mailto:crash-utility-bounces@redhat.com">mailto:crash-utility-bounces@redhat.com</A>] On Behalf Of Castor Fu Sent: Wednesday, January 17, 2007 7:49 AM To: Discussion list for crash utility usage, maintenance and development; Discussion list for crash utility usage, maintenance and development Subject: RE: [Crash-utility] test results of latest 4.0-3.16.sym.patch (ia64) </div>  Hi Dave: I reproduced the problem on an x86 system by creating a module with a bunch of sections. I then found the following in gdb-6.1/gdb/symfile.c:add_symbol_table_command() num_sec_opts = 16; with additional code for xreallocing if it turned out to have too many sections. This seems to be the code which is broken. I'm loath to figure out exactly what it is... I'll put together a patch against 4.0-3.17 which jacks this up, and probably print a warning if we exceed the count. Thanks for digging so far into this. -castor -----Original Message----- From: crash-utility-bounces@redhat.com on behalf of Dave Anderson Sent: Thu 1/4/2007 8:22 AM To: Discussion list for crash utility usage, maintenance and development Subject: Re: [Crash-utility] test results of latest 4.0-3.16.sym.patch (ia64) Hi Castor, Another FYI re: the xrealloc() crash. The problem appears to be specific to gdb. I captured the "add-symbol-file" command string and saved it in an input file. Then I brought crash up and executed the input file, which simply passes the suspect command line directly to gdb, and it crashes on its own: crash> < /tmp/junk crash> add-symbol-file /lib/modules/2.6.18-1.2767.el5/kernel/net/ipv6/ipv6.ko 0xa00000021ed605b0 -s .exit.text 0xa00000021edb49a0 -s .rodata 0xa00000021edbd4c8 -s __ksymtab_strings 0xa00000021edbdc08 -s __versions 0xa00000021edbdf98 -s .data 0xa00000021edd6a20 -s .data.rel.ro 0xa00000021edd6c00 -s __ksymtab_gpl 0xa00000021edd6df8 -s __kcrctab_gpl 0xa00000021edd6ed8 -s .data.rel 0xa00000021edd6f48 -s .data.rel.local 0xa00000021ee39940 -s .data.rel.ro.local 0xa00000021ee3a9c0 -s .data.read_mostly 0xa00000021ee3a9e0 -s __ksymtab 0xa00000021ee3aa60 -s __kcrctab 0xa00000021ee3ac30 -s .gnu.linkonce.this_module 0xa00000021ee3ad80 -s .sdata 0xa00000021ee5d730 -s .bss 0xa00000021ee5b000 -s .sbss 0xa00000021ee5e8b8 add_symbol_file_command: calling xrealloc w/argcnt: 49 arg: [0xa00000021ee5d730]... *** glibc detected *** ./crash: realloc(): invalid next size: 0x6000000001921fe0 *** ======= Backtrace: ========= /lib/libc.so.6.1[0x20000000002f2a70] /lib/libc.so.6.1(realloc-0x1cb0b0)[0x20000000002f5e20] ./crash(xmrealloc+0x1fffffffffee6e20)[0x40000000003a7d00] ./crash[0x40000000002ff500] ./crash[0x40000000004221e0] ./crash(cmd_func+0x1ffffffffff61610)[0x4000000000422500] ./crash(execute_command+0x1fffffffffee25f0)[0x40000000003a34f0] ./crash(gdb_command_funnel+0x1fffffffffe2feb0)[0x40000000002f0dc0] ./crash(gdb_interface+0x1fffffffffcd7590)[0x40000000001984b0] ./crash(gdb_pass_through+0x1fffffffffcd6cb0)[0x4000000000197be0] ./crash(cmd_gdb+0x2000000000151068)[0x400000000019bbc0] ./crash(exec_command+0x1fffffffffb99db0)[0x400000000005acf0] ./crash(exec_input_file+0x1fffffffffd86d40)[0x4000000000247c90] ./crash[0x400000000005b420] ./crash(exec_command+0x1fffffffffb99e50)[0x400000000005ad90] ./crash(main_loop+0x1fffffffffb9a2e0)[0x400000000005a8e0] ./crash(current_interp_command_loop+0x200000000001fd60)[0x40000000004e0cc0] ./crash[0x40000000003199c0] ./crash[0x400000000039f370] ./crash[0x40000000003a4260] ./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320] ./crash[0x400000000031a930] ./crash[0x400000000039f370] ./crash[0x40000000003a4260] ./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320] ./crash(gdb_main+0x1fffffffffe58960)[0x40000000003198e0] ./crash(gdb_main_entry+0x1fffffffffe589f0)[0x4000000000319980] ./crash(gdb_main_loop+0x1fffffffffcd54d0)[0x4000000000196470] ./crash(main+0x1fffffffffb99820)[0x400000000005a330] /lib/libc.so.6.1(__libc_start_main-0x2818f0)[0x200000000023f6c0] ./crash(_start+0x1fffffffffb95240)[0x4000000000056200] ======= Memory map: ======== 00000000-00004000 r--p 00000000 00:00 0 2000000000000000-2000000000038000 r-xp 00000000 fd:00 10256390 /lib/ld-2.5.so 2000000000044000-2000000000050000 rw-p 00034000 fd:00 10256390 /lib/ld-2.5.so 2000000000050000-2000000000114000 r-xp 00000000 fd:00 10256405 /lib/libm-2.5.so 2000000000114000-2000000000120000 ---p 000c4000 fd:00 10256405 /lib/libm-2.5.so 2000000000120000-2000000000124000 rw-p 000c0000 fd:00 10256405 /lib/libm-2.5.so 2000000000124000-20000000001b0000 r-xp 00000000 fd:00 10883077 /usr/lib/libncurses.so.5.5 20000000001b0000-20000000001bc000 ---p 0008c000 fd:00 10883077 /usr/lib/libncurses.so.5.5 20000000001bc000-20000000001cc000 rw-p 00088000 fd:00 10883077 /usr/lib/libncurses.so.5.5 20000000001cc000-20000000001d0000 rw-p 20000000001cc000 00:00 0 20000000001d0000-20000000001d8000 r-xp 00000000 fd:00 10256403 /lib/libdl-2.5.so 20000000001d8000-20000000001e4000 ---p 00008000 fd:00 10256403 /lib/libdl-2.5.so 20000000001e4000-20000000001e8000 rw-p 00004000 fd:00 10256403 /lib/libdl-2.5.so 20000000001e8000-200000000020c000 r-xp 00000000 fd:00 10882711 /usr/lib/libz.so.1.2.3 200000000020c000-2000000000218000 ---p 00024000 fd:00 10882711 /usr/lib/libz.so.1.2.3 2000000000218000-200000000021c000 rw-p 00020000 fd:00 10882711 /usr/lib/libz.so.1.2.3 200000000021c000-2000000000480000 r-xp 00000000 fd:00 10256397 /lib/libc-2.5.so 2000000000480000-200000000048c000 ---p 00264000 fd:00 10256397 /lib/libc-2.5.so 200000000048c000-2000000000498000 rw-p 00260000 fd:00 10256397 /lib/libc-2.5.so 2000000000498000-20000000004d8000 rw-p 2000000000498000 00:00 0 20000000004d8000-2000000003c1c000 r--p 00000000 fd:00 10882710 /usr/lib/locale/locale-archive 2000000003c1c000-2000000003c2c000 rw-p 2000000003c1c000 00:00 0 2000000003c38000-2000000003c44000 r-xp 00000000 fd:00 10256427 /lib/libthread_db-1.0.so 2000000003c44000-2000000003c50000 ---p 0000c000 fd:00 10256427 /lib/libthread_db-1.0.so 2000000003c50000-2000000003c54000 rw-p 00008000 fd:00 10256427 /lib/libthread_db-1.0.so 2000000003c54000-2000000003c58000 rw-p 2000000003c54000 00:00 0 2000000003c6c000-2000000003da0000 rw-p 2000000003c6c000 00:00 0 2000000003da0000-2000000003dbc000 r-xp 00000000 fd:00 10884674 /usr/lib/libunwind.so.7.0.0 2000000003dbc000-2000000003dc8000 ---p 0001c000 fd:00 10884674 /usr/lib/libunwind.so.7.0.0 2000000003dc8000-2000000003dcc000 rw-p 00018000 fd:00 10884674 /usr/lib/libunwind.so.7.0.0 2000000003dcc000-2000000003df0000 rw-p 2000000003dcc000 00:00 0 2000000003e00000-2000000003e08000 r--s 00000000 fd:00 10977539 /usr/lib/gconv/gconv-modules.cache 2000000003e08000-2000000003e18000 rw-p 2000000003e08000 00:00 0 2000000003e1c000-2000000006edc000 rw-p 2000000003e1c000 00:00 0 2000000006ee8000-2000000006f04000 r-xp 00000000 fd:00 10256386 /lib/libgcc_s-4.1.1-20061130.so.1 2000000006f04000-2000000006f10000 ---p 0001c000 fd:00 10256386 /lib/libgcc_s-4.1.1-20061130.so.1 2000000006f10000-2000000006f14000 rw-p 00018000 fd:00 10256386 /lib/libgcc_s-4.1.1-20061130.so.1 2000000006f14000-2000000006f24000 rw-p 2000000006f14000 00:00 0 2000000008000000-2000000008024000 rw-p 2000000008000000 00:00 0 2000000008024000-200000000c000000 ---p 2000000008024000 00:00 0 4000000000000000-40000000007e0000 r-xp 00000000 fd:00 9633909 /var/tmp/crash-4.0-3.16/crash 600000000000c000-600000000006c000 rw-p 007dc000 fd:00 9633909 /var/tmp/crash-4.0-3.16/crash 600000000006c000-6000000001fc0000 rw-p 600000000006c000 00:00 0 [heap] 60000fff7fffc000-60000fff80004000 rw-p 60000fff7fffc000 00:00 0 60000ffffe068000-60000ffffe0bc000 rw-p 60000ffffe068000 00:00 0 [stack] a000000000000000-a000000000020000 ---p 00000000 00:00 0 [vdso] Aborted <pre WRAP> <hr WIDTH="90%" SIZE=4>-- Crash-utility mailing list Crash-utility@redhat.com <a href="https://www.redhat.com/mailman/listinfo/crash-utility">https://www.redhat.com/mailman/listinfo/crash-utility</a></pre> </blockquote> </html>