<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=Content-Type content="text/html; charset=us-ascii"> <META content="MSHTML 6.00.2900.3086" name=GENERATOR></HEAD> <BODY> <DIV dir=ltr align=left>Dave,</DIV> <DIV dir=ltr align=left> </DIV> <DIV dir=ltr align=left>I agree that we own it from this side to figure out where the rest of the dump went.</DIV> <DIV dir=ltr align=left> </DIV> <DIV dir=ltr align=left>Thank you again for your help,</DIV> <DIV dir=ltr align=left> Frank</DIV> <BLOCKQUOTE dir=ltr style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px"> <DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left> <HR tabIndex=-1> From: crash-utility-bounces@redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Monday, May 14, 2007 3:16 PM To: Discussion list for crash utility usage, maintenance and development Subject: Re: [Crash-utility] Seek error type: "tss_struct ist array" problemon8-CPU AMD system </DIV> <DIV></DIV><TT>"Jansen, Frank" wrote:</TT> <BLOCKQUOTE TYPE="CITE"><TT>> -----Original Message-----</TT> <TT>> From: crash-utility-bounces@redhat.com</TT> <TT>> [<A href="mailto:crash-utility-bounces@redhat.com">mailto:crash-utility-bounces@redhat.com</A>] On Behalf Of Dave Anderson</TT> <TT>> Sent: Monday, May 14, 2007 12:22 PM</TT> <TT>> To: Discussion list for crash utility usage, maintenance and</TT> <TT>> development</TT> <TT>> Subject: Re: [Crash-utility] Seek error type: "tss_struct ist</TT> <TT>> array" problem on8-CPU AMD system</TT> <TT>></TT> <TT>> "Jansen, Frank" wrote:</TT> <TT>></TT> <TT>> > Looking through the changelog, I saw that the 'tss_struct ist array'</TT> <TT>> > problem on 8-CPU systems had been addressed previously.</TT> <TT>> However, I'm</TT> <TT>> > running into this issue on an AMD server with crash 4.0-4.1</TT> <TT>> and RHEL4</TT> <TT>> > Update 5 (2.6.9-55.Elsmp).</TT> <TT>> ></TT> <TT>> > The output from the crash invocation is the following:</TT> <TT>> > +++</TT> <TT>> > [root@well-rhel4564-ps3 dump]# /fpj/crash System_map.2.6.9-55.ELsmp</TT> <TT>> > vmlinux.debug.2.6.9-55.ELsmp ap3.1178895173.dmp</TT> <TT>> ></TT> <TT>> > crash 4.0-4.1</TT> <TT>> > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.</TT> <TT>> > Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C)</TT> <TT>> > 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu</TT> <TT>> > Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.</TT> <TT>> > Copyright (C) 2005 NEC Corporation</TT> <TT>> > Copyright (C) 1999, 2002 Silicon Graphics, Inc.</TT> <TT>> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.</TT> <TT>> > This program is free software, covered by the GNU General Public</TT> <TT>> > License, and you are welcome to change it and/or distribute</TT> <TT>> copies of</TT> <TT>> > it under certain conditions. Enter "help copying" to see the</TT> <TT>> > conditions.</TT> <TT>> > This program has absolutely no warranty. Enter "help warranty" for</TT> <TT>> > details.</TT> <TT>> ></TT> <TT>> > GNU gdb 6.1</TT> <TT>> > Copyright 2004 Free Software Foundation, Inc.</TT> <TT>> > GDB is free software, covered by the GNU General Public</TT> <TT>> License, and</TT> <TT>> > you are welcome to change it and/or distribute copies of it under</TT> <TT>> > certain conditions.</TT> <TT>> > Type "show copying" to see the conditions.</TT> <TT>> > There is absolutely no warranty for GDB. Type "show warranty" for</TT> <TT>> > details.</TT> <TT>> > This GDB was configured as "x86_64-unknown-linux-gnu"...</TT> <TT>> ></TT> <TT>> > crash: seek error: kernel virtual address: 10408119e84 type:</TT> <TT>> > "tss_struct ist array"</TT> <TT>> > ---</TT> <TT>> ></TT> <TT>> > The server is a 4 dual-core AMD (2.8GHz) with 64GB.</TT> <TT>> ></TT> <TT>> > Any insights into how best to troubleshoot this are much</TT> <TT>> appreciated.</TT> <TT>> ></TT> <TT>> > Thanks,</TT> <TT>> ></TT> <TT>> > Frank Jansen</TT> <TT>></TT> <TT>> I doubt this has anything to do with the 8-cpu issue.</TT> <TT>></TT> <TT>I think that you are right, as the crash -d7 seems to indicate that the</TT> <TT>dump may be incomplete(cf. attached crash -d7 output).</TT><TT></TT> <TT>> A few questions:</TT> <TT>></TT> <TT>> Is this an RHEL4 derivative kernel of some kind? I ask</TT> <TT>> because you're using a system.map file as an argument.</TT> <TT>></TT> <TT>It's a standard kernel, to which we add a couple of our (Egenera)</TT> <TT>drivers. I can read the dump without the system map argument, but was</TT> <TT>just going off the data provided to me by the person that ran into the</TT> <TT>problem.</TT><TT></TT> <TT>> Anyway, this dumpfile is Egenera's LKCD off-shoot, correct?</TT> <TT>> Since you got an "lseek" error, the question is whether (1)</TT> <TT>> the virtual address of 10408119e84 is legitimate, and (2)</TT> <TT>> whether it is included in your dumpfile.</TT><TT></TT> <TT>I think that the virtual address is legitimate, but that the dump is</TT> <TT>incomplete at this point.</TT><TT></TT> <TT>></TT> <TT>> What does "crash -d7 ..." show?</TT><TT></TT> <TT>See attached output</TT><TT></TT> <TT>></TT> <TT>> Does crash work on the live system?</TT> <TT>Yes, it works</TT></BLOCKQUOTE><TT>Right -- if it works on the live system, there's a good chance that</TT> <TT>it's probably missing from the dumpfile. The tss_struct for each</TT> <TT>cpu is located in each cpu's per-cpu data area. I have seen the</TT> <TT>exact same problem with x86_64 netdump "vmcore-incomplete" dumpfiles,</TT> <TT>where the per-cpu data areas, allocated with alloc_bootmem_node(),</TT> <TT>would tend to be located in very high physical memory (beyond the</TT> <TT>end of the vmcore-incomplete contents).</TT><TT></TT> <TT>On a 64GB system, the virtual address of 10408119e84 (~16GB physical)</TT> <TT>would certainly not be out of the question. And if it can be read</TT> <TT>on the live machine (crash -d7 will show the same address access</TT> <TT>sequence), then it's probably not included in the dumpfile for</TT> <TT>whatever reason.</TT><TT></TT> <TT>In fact, looking at the -d7 output, the level_pgt pagetable pointers</TT> <TT>for each non-cpu0 cpu_pda get allocated with __get_free_pages() -- and</TT> <TT>there's a couple from the 10408xxxxxx virtual memory location:</TT><TT></TT> <TT>...</TT> <TT><readmem: ffffffff804ed700, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU0: level4_pgt: ffffffff80101000 data_offset: 10087adef60</TT> <TT><readmem: ffffffff804ed780, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU1: level4_pgt: 1040802c000 data_offset: 10487bf8d60</TT> <TT><readmem: ffffffff804ed800, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU2: level4_pgt: 10408008000 data_offset: 10887bf8d60</TT> <TT><readmem: ffffffff804ed880, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU3: level4_pgt: 10bf9ff2000 data_offset: 10c87bfbf60</TT> <TT><readmem: ffffffff804ed900, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU4: level4_pgt: 10008028000 data_offset: 10087ae6f60</TT> <TT><readmem: ffffffff804ed980, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU5: level4_pgt: 10bf9f8a000 data_offset: 10487c00d60</TT> <TT><readmem: ffffffff804eda00, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU6: level4_pgt: 100f7f08000 data_offset: 10887c00d60</TT> <TT><readmem: ffffffff804eda80, KVADDR, "cpu_pda entry", 128, (FOE), 930580></TT> <TT>CPU7: level4_pgt: 107f9f8e000 data_offset: 10c87c03f60</TT> <TT><readmem: 10008000084, KVADDR, "tss_struct ist array", 56, (FOE), 90c5b0></TT> <TT><readmem: 10408119e84, KVADDR, "tss_struct ist array", 56, (FOE), 90c5e8></TT> <TT>crash: seek error: kernel virtual address: 10408119e84 type: "tss_struct ist array"</TT><TT></TT> <TT>They weren't *read* from there at that point, but it shows that</TT> <TT>there was memory in that neighborhood. Anyway, the "seek error"</TT> <TT>from LKCD means that the physical page couldn't be found in the</TT> <TT>dumpfile by lkcd_lseek():</TT><TT></TT> <TT>/*</TT> <TT> * Read from an LKCD formatted dumpfile.</TT> <TT> */</TT> <TT>int</TT> <TT>read_lkcd_dumpfile(int fd, void *bufptr, int cnt, ulong addr, physaddr_t paddr)</TT> <TT>{</TT> <TT> set_lkcd_fp(fp);</TT><TT></TT> <TT> if (!lkcd_lseek(paddr))</TT> <TT> return SEEK_ERROR;</TT><TT></TT> <TT> if (lkcd_read((void *)bufptr, cnt) != cnt)</TT> <TT> return READ_ERROR;</TT><TT></TT> <TT> return cnt;</TT> <TT>}</TT><TT></TT> <TT>I can't really help you from that point on, though...</TT><TT></TT> <TT>Dave</TT> <TT></TT> </BLOCKQUOTE></BODY></HTML>