<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> <html> <tt>"Jansen, Frank" wrote:</tt> <blockquote TYPE=CITE><tt>> -----Original Message-----</tt> <tt>> From: crash-utility-bounces@redhat.com</tt> <tt>> [<a href="mailto:crash-utility-bounces@redhat.com">mailto:crash-utility-bounces@redhat.com</a>] On Behalf Of Dave Anderson</tt> <tt>> Sent: Monday, May 14, 2007 12:22 PM</tt> <tt>> To: Discussion list for crash utility usage, maintenance and</tt> <tt>> development</tt> <tt>> Subject: Re: [Crash-utility] Seek error type: "tss_struct ist</tt> <tt>> array" problem on8-CPU AMD system</tt> <tt>></tt> <tt>> "Jansen, Frank" wrote:</tt> <tt>></tt> <tt>> > Looking through the changelog, I saw that the 'tss_struct ist array'</tt> <tt>> > problem on 8-CPU systems had been addressed previously.</tt> <tt>> However, I'm</tt> <tt>> > running into this issue on an AMD server with crash 4.0-4.1</tt> <tt>> and RHEL4</tt> <tt>> > Update 5 (2.6.9-55.Elsmp).</tt> <tt>> ></tt> <tt>> > The output from the crash invocation is the following:</tt> <tt>> > +++</tt> <tt>> > [root@well-rhel4564-ps3 dump]# /fpj/crash System_map.2.6.9-55.ELsmp</tt> <tt>> > vmlinux.debug.2.6.9-55.ELsmp ap3.1178895173.dmp</tt> <tt>> ></tt> <tt>> > crash 4.0-4.1</tt> <tt>> > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.</tt> <tt>> > Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C)</tt> <tt>> > 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu</tt> <tt>> > Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.</tt> <tt>> > Copyright (C) 2005 NEC Corporation</tt> <tt>> > Copyright (C) 1999, 2002 Silicon Graphics, Inc.</tt> <tt>> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.</tt> <tt>> > This program is free software, covered by the GNU General Public</tt> <tt>> > License, and you are welcome to change it and/or distribute</tt> <tt>> copies of</tt> <tt>> > it under certain conditions. Enter "help copying" to see the</tt> <tt>> > conditions.</tt> <tt>> > This program has absolutely no warranty. Enter "help warranty" for</tt> <tt>> > details.</tt> <tt>> ></tt> <tt>> > GNU gdb 6.1</tt> <tt>> > Copyright 2004 Free Software Foundation, Inc.</tt> <tt>> > GDB is free software, covered by the GNU General Public</tt> <tt>> License, and</tt> <tt>> > you are welcome to change it and/or distribute copies of it under</tt> <tt>> > certain conditions.</tt> <tt>> > Type "show copying" to see the conditions.</tt> <tt>> > There is absolutely no warranty for GDB. Type "show warranty" for</tt> <tt>> > details.</tt> <tt>> > This GDB was configured as "x86_64-unknown-linux-gnu"...</tt> <tt>> ></tt> <tt>> > crash: seek error: kernel virtual address: 10408119e84 type:</tt> <tt>> > "tss_struct ist array"</tt> <tt>> > ---</tt> <tt>> ></tt> <tt>> > The server is a 4 dual-core AMD (2.8GHz) with 64GB.</tt> <tt>> ></tt> <tt>> > Any insights into how best to troubleshoot this are much</tt> <tt>> appreciated.</tt> <tt>> ></tt> <tt>> > Thanks,</tt> <tt>> ></tt> <tt>> > Frank Jansen</tt> <tt>></tt> <tt>> I doubt this has anything to do with the 8-cpu issue.</tt> <tt>></tt> <tt>I think that you are right, as the crash -d7 seems to indicate that the</tt> <tt>dump may be incomplete(cf. attached crash -d7 output).</tt><tt></tt> <tt>> A few questions:</tt> <tt>></tt> <tt>> Is this an RHEL4 derivative kernel of some kind? I ask</tt> <tt>> because you're using a system.map file as an argument.</tt> <tt>></tt> <tt>It's a standard kernel, to which we add a couple of our (Egenera)</tt> <tt>drivers. I can read the dump without the system map argument, but was</tt> <tt>just going off the data provided to me by the person that ran into the</tt> <tt>problem.</tt><tt></tt> <tt>> Anyway, this dumpfile is Egenera's LKCD off-shoot, correct?</tt> <tt>> Since you got an "lseek" error, the question is whether (1)</tt> <tt>> the virtual address of 10408119e84 is legitimate, and (2)</tt> <tt>> whether it is included in your dumpfile.</tt><tt></tt> <tt>I think that the virtual address is legitimate, but that the dump is</tt> <tt>incomplete at this point.</tt><tt></tt> <tt>></tt> <tt>> What does "crash -d7 ..." show?</tt><tt></tt> <tt>See attached output</tt><tt></tt> <tt>></tt> <tt>> Does crash work on the live system?</tt> <tt>Yes, it works</tt></blockquote> <tt>Right -- if it works on the live system, there's a good chance that</tt> <tt>it's probably missing from the dumpfile. The tss_struct for each</tt> <tt>cpu is located in each cpu's per-cpu data area. I have seen the</tt> <tt>exact same problem with x86_64 netdump "vmcore-incomplete" dumpfiles,</tt> <tt>where the per-cpu data areas, allocated with alloc_bootmem_node(),</tt> <tt>would tend to be located in very high physical memory (beyond the</tt> <tt>end of the vmcore-incomplete contents).</tt><tt></tt> <tt>On a 64GB system, the virtual address of 10408119e84 (~16GB physical)</tt> <tt>would certainly not be out of the question. And if it can be read</tt> <tt>on the live machine (crash -d7 will show the same address access</tt> <tt>sequence), then it's probably not included in the dumpfile for</tt> <tt>whatever reason.</tt><tt></tt> <tt>In fact, looking at the -d7 output, the level_pgt pagetable pointers</tt> <tt>for each non-cpu0 cpu_pda get allocated with __get_free_pages() -- and</tt> <tt>there's a couple from the 10408xxxxxx virtual memory location:</tt><tt></tt> <tt>...</tt> <tt><readmem: ffffffff804ed700, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU0: level4_pgt: ffffffff80101000 data_offset: 10087adef60</tt> <tt><readmem: ffffffff804ed780, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU1: level4_pgt: 1040802c000 data_offset: 10487bf8d60</tt> <tt><readmem: ffffffff804ed800, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU2: level4_pgt: 10408008000 data_offset: 10887bf8d60</tt> <tt><readmem: ffffffff804ed880, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU3: level4_pgt: 10bf9ff2000 data_offset: 10c87bfbf60</tt> <tt><readmem: ffffffff804ed900, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU4: level4_pgt: 10008028000 data_offset: 10087ae6f60</tt> <tt><readmem: ffffffff804ed980, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU5: level4_pgt: 10bf9f8a000 data_offset: 10487c00d60</tt> <tt><readmem: ffffffff804eda00, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU6: level4_pgt: 100f7f08000 data_offset: 10887c00d60</tt> <tt><readmem: ffffffff804eda80, KVADDR, "cpu_pda entry", 128, (FOE), 930580></tt> <tt>CPU7: level4_pgt: 107f9f8e000 data_offset: 10c87c03f60</tt> <tt><readmem: 10008000084, KVADDR, "tss_struct ist array", 56, (FOE), 90c5b0></tt> <tt><readmem: 10408119e84, KVADDR, "tss_struct ist array", 56, (FOE), 90c5e8></tt> <tt>crash: seek error: kernel virtual address: 10408119e84 type: "tss_struct ist array"</tt><tt></tt> <tt>They weren't *read* from there at that point, but it shows that</tt> <tt>there was memory in that neighborhood. Anyway, the "seek error"</tt> <tt>from LKCD means that the physical page couldn't be found in the</tt> <tt>dumpfile by lkcd_lseek():</tt><tt></tt> <tt>/*</tt> <tt> * Read from an LKCD formatted dumpfile.</tt> <tt> */</tt> <tt>int</tt> <tt>read_lkcd_dumpfile(int fd, void *bufptr, int cnt, ulong addr, physaddr_t paddr)</tt> <tt>{</tt> <tt> set_lkcd_fp(fp);</tt><tt></tt> <tt> if (!lkcd_lseek(paddr))</tt> <tt> return SEEK_ERROR;</tt><tt></tt> <tt> if (lkcd_read((void *)bufptr, cnt) != cnt)</tt> <tt> return READ_ERROR;</tt><tt></tt> <tt> return cnt;</tt> <tt>}</tt><tt></tt> <tt>I can't really help you from that point on, though...</tt><tt></tt> <tt>Dave</tt> <tt></tt> </html>