[Crash-utility] crash mod -S problem
Dave Anderson
anderson at redhat.com
Thu Jan 29 18:29:56 UTC 2009
----- "Cliff Wickman" <cpw at sgi.com> wrote:
> On Thu, Jan 29, 2009 at 11:23:49AM -0500, Dave Anderson wrote:
> >
> > ----- "Cliff Wickman" <cpw at sgi.com> wrote:
> >
> > > Hi Dave, and company,
> > >
> > > I have a question. Perhaps this is a known issue -- should I be looking
> > > in an archive somewhere?
> > >
> > > The crash command "mod -S" is messing up the address list.
> > >
> > > This is crash-4.0-7.6, running on ia64.
> > >
> > > Note how the value of module symbol xfs_ioerror_alert has changed
> > > after the mod -S.
> > >
> > > Is this a crash problem? Or my problem? (there are several ?'s in
> > > the mod -S output, including on the xfs module)
> >
> > Good question -- I don't know.
> >
> > It appears that the determination of the module's text starting location
> > is being changed by the "mod -S". That can happen when the symbols.c
> > function store_load_module_symbols() calls calculate_load_order_v2(),
> > which can change the text start location that was initialized during
> > the crash session initialization by store_module_symbols_v2().
> >
> > The question is which text start address value is correct.
> >
> > Try this:
> >
> > $ crash -s
> > crash> help -s | grep mod_text_start > before
> > crash> mod -S
> > ... [ snip ] ...
> > crash> help -s | grep mod_text_start > after
> > crash> q
> >
> > $ diff before after
>
> It changed:
> $ diff before after
> 1,3c1,3
> < mod_text_start: a0000002093f0000 (0)
> < mod_text_start: a000000209410000 (0)
> < mod_text_start: a000000209430000 (0)
> ---
> > mod_text_start: a0000002093f49e0 (49e0)
> > mod_text_start: a000000209412040 (2040)
> > mod_text_start: a000000209433968 (3968)
> 5,7c5,7
> < mod_text_start: a000000209470000 (0)
> < mod_text_start: a0000002094b0000 (0)
> < mod_text_start: a000000209510000 (0)
> ---
> > mod_text_start: a000000209483c68 (13c68)
> > mod_text_start: a0000002094d31a8 (231a8)
> > mod_text_start: a00000020951c4d8 (c4d8)
> [snip]
>
>
> > Typically it won't change, and the before and after files are
> identical.
> >
> > But on your system it would appear that many (if not all) of the modules are
> > changing their mod_text_start values upon running "mod -S".
> >
> > You can also determine which value is correct by doing a "dis <function>"
> > command before and after the "mod -S".
>
> Yes, we were doing a "dis" when we noticed the problem. The addresses
> are correct before the mod -S.
>
> So you suspect the problem is in calculate_load_order_v2() ?
>
> -Cliff
If the text was correct *before* the "mod -S", then yes, the
text base address was changed by calculate_load_order_v2().
My first response started a cc: to Castor Fu, the author of
calculate_load_order_v2() function, because I'd be interested
in what he has to say on the matter. He's on west coast time.
As I understand it, calculate_load_order_v2() is taking the
first (lowest) symbol of each module section (as determined
during initialization), and searches for it in the module
object file. And when it finds it, it calls bfd_get_section_name()
to get the section name string, and based upon that updates the
fields associated with that section. For the .text section it
normally finds the first text symbol to have an offset value of 0,
and therefore no changes are made to the mod_text_offset. It looks
like it's finding the "first" text symbol at a higher offset?
If you enter "set debug 3", and then do a "mod -s <module-name>",
you'll see its machinations on a given module.
Dave
>
>
> > > crash> sym xfs_ioerror_alert
> > > a000000209955f80 (t) xfs_ioerror_alert
> > >
> > > crash> mod -S
> > > MODULE NAME SIZE OBJECT FILE
> > > a0000002093fc088 scsi_transport_spi 50672
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/scsi_transport_spi.ko
> > > mod: ioc4: last symbol is not _MODULE_END_ioc4?
> > > a000000209412fa8 ioc4 13056
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/misc/ioc4.ko
> > > mod: scsi_tgt: last symbol is not _MODULE_END_scsi_tgt?
> > > a000000209435cd0 scsi_tgt 24640
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/scsi_tgt.ko
> > > a000000209453610 sgiioc4 14656
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/ide/pci/sgiioc4.ko
> > > mod: mptbase: last symbol is not _MODULE_END_mptbase?
> > > a000000209497298 mptbase 166540
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/message/fusion/mptbase.ko
> > > a0000002094f8008 scsi_mod 299024
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/scsi_mod.ko
> > > a000000209547038 ide_core 229280
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/ide/ide-core.ko
> > > a000000209583da0 qla1280 148256
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/qla1280.ko
> > > mod: scsi_transport_fc: last symbol is not
> > > _MODULE_END_scsi_transport_fc?
> > > a0000002095b51d0 scsi_transport_fc 88156
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/scsi_transport_fc.ko
> > > a0000002095d7bd8 mptfc 32776
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/message/fusion/mptfc.ko
> > > mod: mptscsih: last symbol is not _MODULE_END_mptscsih?
> > > a000000209610618 mptscsih 68816
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/message/fusion/mptscsih.ko
> > > a000000209631208 hwmon 5304
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/hwmon/hwmon.ko
> > > mod: thermal_sys: last symbol is not _MODULE_END_thermal_sys?
> > > a000000209655c90 thermal_sys 24656
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/thermal/thermal_sys.ko
> > > a000000209679b40 thermal 41144
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/acpi/thermal.ko
> > > a0000002096b4340 processor 85008
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/acpi/processor.ko
> > > a0000002096d23c8 fan 9960
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/acpi/fan.ko
> > > a000000209718490 mptspi 34936
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/message/fusion/mptspi.ko
> > > a000000209730e00 ide_generic 4264
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/ide/ide-generic.ko
> > > a0000002097506f0 crc_t10dif 2336
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/lib/crc-t10dif.ko
> > > a000000209773430 button 14312
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/acpi/button.ko
> > > a0000002097b1380 sd_mod 72432
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/sd_mod.ko
> > > a0000002097e12a0 sg 71976
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/sg.ko
> > > a0000002098124b8 cdrom 77752
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/cdrom/cdrom.ko
> > > mod: cannot find or load object file for ide_cd_mod module
> > > mod: libphy: last symbol is not _MODULE_END_libphy?
> > > a000000209859928 libphy 40168
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/net/phy/libphy.ko
> > > mod: xfs: last symbol is not _MODULE_END_xfs?
> > > a0000002099abaf0 xfs 1233888
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/xfs/xfs.ko
> > > a0000002099ed130 mptctl 188064
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/message/fusion/mptctl.ko
> > > a000000209a04690 mmtimer 18836
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/char/mmtimer.ko
> > > a000000209a53a68 kdbm_pg 16632
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/kdb/modules/kdbm_pg.ko
> > > a000000209a70a48 kdbm_sched 3296
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/kdb/modules/kdbm_sched.ko
> > > a000000209a91b28 kdbm_task 40608
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/kdb/modules/kdbm_task.ko
> > > a000000209b1fba0 qla2xxx 463704
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/scsi/qla2xxx/qla2xxx.ko
> > > a000000209b850c0 tg3 284796
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/net/tg3.ko
> > > mod: dm_mod: last symbol is not _MODULE_END_dm_mod?
> > > a000000209ff25f8 dm_mod 143048
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/md/dm-mod.ko
> > > a00000020a016518 kdbm_vm 28112
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/kdb/modules/kdbm_vm.ko
> > > a00000020a031f20 mspec 16932
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/char/mspec.ko
> > > a00000020a05e290 numatools 59232
> > > /lib/modules/2.6.27.8-1-sgidev/up
> > > dates/numatools.ko
> > > a00000020a073f38 xp 17216
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/misc/sgi-xp/xp.ko
> > > a00000020a0b4c08 gru 87288
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/misc/sgi-gru/gru.ko
> > > a00000020a0d38e0 sgi_gpl_lib 15272
> > > /lib/modules/2.6.27.8-1-sgidev/we
> > > ak-updates/extra/gpl_lib/sgi-gpl-lib.ko
> > > a00000020a115c58 myri10ge 91048
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/net/myri10ge/myri10ge.ko
> > > mod: loop: last symbol is not _MODULE_END_loop?
> > > a00000020a138478 loop 34788
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/block/loop.ko
> > > mod: cannot find or load object file for nls_iso8859_1 module
> > > a00000020a18ac80 sgi_os_lib 111192
> > > /lib/modules/2.6.27.8-1-sgidev/we
> > > ak-updates/extra/os_lib/sgi-os-lib.ko
> > > mod: crypto_blkcipher: last symbol is not
> > > _MODULE_END_crypto_blkcipher?
> > > a00000020a1a8c00 crypto_blkcipher 36848
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/crypto/crypto_blkcipher.ko
> > > a00000020a1c6fd0 dm_crypt 29784
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/drivers/md/dm-crypt.ko
> > > a00000020a1e4cd8 binfmt_misc 20444
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/binfmt_misc.ko
> > > a00000020a25d758 xpmem 122536
> > > /lib/modules/2.6.27.8-1-sgidev/up
> > > dates/xpmem.ko
> > > a00000020a28d518 fuse 121352
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/fuse/fuse.ko
> > > mod: ipv6: last symbol is not _MODULE_END_ipv6?
> > > a00000020a358030 ipv6 627656
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/net/ipv6/ipv6.ko
> > > mod: sgi_xvm: last symbol is not _MODULE_END_sgi_xvm?
> > > a00000020a41a520 sgi_xvm 704400
> > > /lib/modules/2.6.27.8-1-sgidev/we
> > > ak-updates/extra/xvm/sgi-xvm.ko
> > > a00000020a46ba78 autofs4 48656
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/autofs4/autofs4.ko
> > > a00000020a482090 exportfs 8944
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/exportfs/exportfs.ko
> > > mod: auth_rpcgss: last symbol is not _MODULE_END_auth_rpcgss?
> > > a00000020a4b3ee0 auth_rpcgss 82696
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/net/sunrpc/auth_gss/auth_rpcgss.ko
> > > mod: nfs_acl: last symbol is not _MODULE_END_nfs_acl?
> > > a00000020a4d17b8 nfs_acl 6648
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/nfs_common/nfs_acl.ko
> > > a00000020a5b1e88 sunrpc 472176
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/net/sunrpc/sunrpc.ko
> > > mod: lockd: last symbol is not _MODULE_END_lockd?
> > > a00000020a5f1e00 lockd 141416
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/lockd/lockd.ko
> > > a00000020a6aaa98 nfs 641664
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/nfs/nfs.ko
> > > a00000020a73ea30 nfsd 523168
> > > /lib/modules/2.6.27.8-1-sgidev/ke
> > > rnel/fs/nfsd/nfsd.ko
> > >
> > > crash> sym xfs_ioerror_alert
> > > a000000209992d18 (T) xfs_ioerror_alert
> > >
> > > Thanks.
> > > -Cliff
> > >
> > > --
> > > Crash-utility mailing list
> > > Crash-utility at redhat.com
> > > https://www.redhat.com/mailman/listinfo/crash-utility
> >
> > --
> > Crash-utility mailing list
> > Crash-utility at redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Cliff Wickman
> Silicon Graphics, Inc.
> cpw at sgi.com
> (651) 683-3824
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list