<html> <head> <style></style></head> <body class='hmmessage'><div dir='ltr'><br> <BR><div id="SkyDrivePlaceholder"></div><div>> Date: Fri, 18 Jan 2013 10:23:00 -0500<br>> From: anderson@redhat.com<br>> To: crash-utility@redhat.com<br>> Subject: Re: [Crash-utility] questions about crash utility<br>> <br>> <br>> <br>> ----- Original Message -----<br>> > <br>> > <br>> > <br>> > Hi Dave:<br>> > <br>> > thank you very much for your detail answer, this really helpful.<br>> > please see my inline words. thanks.<br>> > <br>> > <br>> > > Date: Thu, 17 Jan 2013 14:17:36 -0500<br>> > > From: anderson@redhat.com<br>> > > To: crash-utility@redhat.com<br>> > > Subject: Re: [Crash-utility] questions about crash utility<br>> > <br>> > > The fact that crash gets as far as it does at least means that the<br>> > > ELF header you've created was deemed acceptable as an ARM vmcore.<br>> > > However, the error messages re: "cpu_present_mask indicates..." and<br>> > > "cannot determine base kernel version" indicate that the data<br>> > > that was read from the vmcore was clearly not the correct data.<br>> > > <br>> > > The "cpu_present_mask" value that it read contained too<br>> > > many bits -- presuming that the 32-bit ARM processor is<br>> > > still limited to only 4 cpus. (looks like upstream that<br>> > > CONFIG_NR_CPUS is still 2 in the arch/arm/configs files.)<br>> > > <br>> > > But more indicative of the wrong data being read is the second<br>> > > "cannot determine base kernel version" message, which was generated<br>> > > after it read the kernel's "init_uts_ns" uts_namespace structure.<br>> > > After reading it, it sees that the "release" string contains<br>> > > non-ASCII data, whereas it should contain the kernel version:<br>> > > <br>> > > crash> p init_uts_ns<br>> > > init_uts_ns = $3 = {<br>> > > kref = {<br>> > > refcount = {<br>> > > counter = 2<br>> > > }<br>> > > },<br>> > > name = {<br>> > > sysname =<br>> > > "Linux\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",<br>> > > nodename =<br>> > > "phenom-01.lab.bos.redhat.com\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",<br>> > > release =<br>> > > "2.6.32-313.el6.x86_64\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",<br>> > > version = "#1 SMP Thu Sep 27 16:25:19 EDT<br>> > > 2012\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",<br>> > > machine =<br>> > > "x86_64\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",<br>> > > domainname =<br>> > > "(none)\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"<br>> > > }<br>> > > }<br>> > > crash><br>> > > <br>> > > So it appears that you're reading data from the wrong<br>> > > locations in the dumpfile. You should be able to verify<br>> > > that by bringing up the crash session with the --minimal<br>> > > flag like this:<br>> > > <br>> > > $ crash --minimal vmlinux vmcore<br>> > > <br>> > > That will bypass most of the initialization, including all<br>> > > readmem() calls of the vmcore. Then do this:<br>> > > <br>> > > crash> rd linux_banner 20<br>> > > ffffffff818000a0: 65762078756e694c 2e33206e6f697372 Linux version<br>> > > 3.<br>> > > ffffffff818000b0: 63662e312d312e35 365f3638782e3731 5.1-1.fc17.x86_6<br>> > > ffffffff818000c0: 626b636f6d282034 69756240646c6975 4(mockbuild@bui<br>> > > ffffffff818000d0: 2e33322d6d76646c 6465662e32786870 ldvm-23.phx2.fed<br>> > > ffffffff818000e0: 656a6f727061726f 202967726f2e7463 oraproject.org)<br>> > > ffffffff818000f0: 7265762063636728 372e34206e6f6973 (gcc version 4.7<br>> > > ffffffff81800100: 303231303220302e 6465522820373035 .0 20120507<br>> > > (Red<br>> > > ffffffff81800110: 372e342074614820 47282029352d302e Hat 4.7.0-5) (G<br>> > > ffffffff81800120: 3123202920294343 75685420504d5320 CC) ) #1 SMP Thu<br>> > > ffffffff81800130: 3120392067754120 2033343a30353a37 Aug 9 17:50:43<br>> > > crash> rd -a linux_banner<br>> > > ffffffff818000a0: Linux version 3.5.1-1.fc17.x86_64 (mockbuild@buildvm-23.phx2<br>> > > ffffffff818000dc: .fedoraproject.org) (gcc version 4.7.0 20120507 (Red Hat 4.7<br>> > > ffffffff81800118: .0-5) (GCC) ) #1 SMP Thu Aug 9 17:50:43 UTC 2012<br>> > > crash><br>> > > <br>> > > I'm guessing that you will not see a string starting with "Linux version"<br>> > > with your dumpfile as shown above.<br>> > > <br>> > > If that's the case, then it's clear that the readmem() function is ultimately<br>> > > reading from the wrong vmcore file offset.<br>> > > <br>> > > Here's what you can try doing. Taking the linux_banner example above,<br>> > > you can check where in the dumpfile it's reading from by setting the debug<br>> > > flag, before doing a simple read -- like this example on an ARM dumpfile:<br>> > > <br>> > > crash> set debug 8<br>> > > debug: 8<br>> > > crash> rd linux_banner<br>> > > <addr: c033ea10 count: 1 flag: 488 (KVADDR)><br>> > > <readmem: c033ea10, KVADDR, "32-bit KVADDR", 4, (FOE), ff94f048><br>> > > <read_kdump: addr: c033ea10 paddr: 33ea10 cnt: 4><br>> > > read_netdump: addr: c033ea10 paddr: 33ea10 cnt: 4 offset: 33f088<br>> > > c033ea10: 756e694c Linu<br>> > > crash><br>> > > <br>> > > The linux_banner is at virtual address c033ea10 (addr). First it gets translated<br>> > > into physical address 33ea10 (paddr). Then that paddr is translated into the<br>> > > vmcore file offset of 33f088. It lseeks to vmcore file offset 33f088 and<br>> > > reads 4 bytes, which contain "756e694c", or the first 4 bytes of the<br>> > > "Linux version ..." string.<br>> > > <br>> > > Note that if I subtract the physical address from vmcore file offset<br>> > > I get this:<br>> > > <br>> > > crash> eval 33f088 - 33ea10<br>> > > hexadecimal: 678<br>> > > decimal: 1656<br>> > > octal: 3170<br>> > > binary: 00000000000000000000011001111000<br>> > > crash><br>> > > <br>> > > which would put physical address 0 at a vmcore file offset of 0x678, and<br>> > > therefore implying that that the ELF header comprises the first 0x678 bytes.<br>> > > And looking at the vmcore, that can be verified:<br>> > > <br>> > <br>> > yes you are right, here i get the result as below:<br>> > crash> set debug 8<br>> > debug: 8<br>> > crash> rd linux_banner<br>> > <addr: c065a071 count: 1 flag: 488 (KVADDR)><br>> > <readmem: c065a071, KVADDR, "32-bit KVADDR", 4, (FOE), ffdf297c><br>> > <read_kdump: addr: c065a071 paddr: 85a071 cnt: 4><br>> > read_netdump: addr: c065a071 paddr: 85a071 cnt: 4 offset: 65a0e5<br>> > c065a071: 03e59130 0...<br>> > <br>> > the virtual address is 0xc065a071 , and the physical address is <br>> > 0x85a071 , and the offset is 0x65a0e5.<br>> > my elf header is 116 bytes long, 0x65a0e5 - 116=0x65A071, which has a<br>> > gap 0x00200000 with the physical address 0x85a071.<br>> > <br>> > <br>> > > $ readelf -a vmcore<br>> > > ELF Header:<br>> > > Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00<br>> > > Class: ELF32<br>> > > Data: 2's complement, little endian<br>> > > Version: 1 (current)<br>> > > OS/ABI: UNIX - System V<br>> > > ABI Version: 0<br>> > > Type: CORE (Core file)<br>> > > Machine: ARM<br>> > > Version: 0x1<br>> > > Entry point address: 0x0<br>> > > Start of program headers: 52 (bytes into file)<br>> > > Start of section headers: 0 (bytes into file)<br>> > > Flags: 0x0<br>> > > Size of this header: 52 (bytes)<br>> > > Size of program headers: 32 (bytes)<br>> > > Number of program headers: 3<br>> > > Size of section headers: 0 (bytes)<br>> > > Number of section headers: 0<br>> > > Section header string table index: 0<br>> > > <br>> > > There are no sections in this file.<br>> > > <br>> > > There are no sections to group in this file.<br>> > > <br>> > > Program Headers:<br>> > > Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align<br>> > > NOTE 0x000094 0x00000000 0x004e345c 0x005e4 0x005e4 0<br>> > > LOAD 0x000678 0xc0000000 0x00000000 0x5600000 0x5600000 RWE 0<br>> > > LOAD 0x5600678 0xc5700000 0x05700000 0x100000 0x100000 RWE 0<br>> > > ...<br>> > > <br>> > > Note that the "Offset" value of the first PT_LOAD segment has a file offset<br>> > > value of 0x678.<br>> > > <br>> > <br>> > here i got the result as below:<br>> > Program Headers:<br>> > Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align<br>> > NOTE 0x000000 0x00000000 0x00000000 0x00000 0x00000 0<br>> > LOAD 0x000074 0xc0000000 0x00200000 0x2fe00000 0x2fe00000 RWE 0<br>> > <br>> > so the problem is i don't understand the elf header meaning<br>> > accurately. if i modify code as below, everything is ok for me:<br>> > <br>> > offset += sizeof(struct elf_phdr);<br>> > phdr->p_offset = offset+0x00200000;<br>> > phdr->p_vaddr = 0xc0000000;<br>> > phdr->p_paddr = 0x00200000;<br>> > phdr->p_filesz = phdr->p_memsz = = MEMSIZE-0x00200000;<br>> > <br>> > <br>> > although my modification can make crash utility work well, i want to<br>> > know exactly whether i am doing the right thing.<br>> > 1. our platform has the ddr address from physical address 0x0.<br>> > 2. when compiling Linux kernel, our platform set in .config file:<br>> > CONFIG_PHYS_OFFSET=0x00200000<br>> > 3. when Kernel crash, all ddr content will be dumped, from address<br>> > 0x0~768MB. but kernel data starts from 0x00200000 actually.<br>> > <br>> > my questions are:<br>> > 1. whether my setting of ELF header is correct this time? the offset,<br>> > paddr, and p_memsz?<br>> <br>> I'm not really sure. Even though you've got it to work OK, I don't<br>> understand your new phdr->p_offset and phdr->p_filesz/phdr->p_memsz <br>> settings. The phdr->p_offset value typically points to the beginning<br>> of the physical memory segment, which in your case, would be at physical<br>> address 0x0 at file offset 0x74. And the phdr->p_filesz/phdr->p_memsz<br>> values are typically equal to the full size of the physical memory <br>> segment (MEMSIZE).<br>> </div><div> </div><div>if i set p_offset to 0, the file offset seems not correct. for example, when i try to read </div><div>linux_banner, i got below result:</div><div> crash> set debug 8<br> debug: 8<br> crash> rd linux_banner<br> <addr: c065a071 count: 1 flag: 488 (KVADDR)><br> <readmem: c065a071, KVADDR, "32-bit KVADDR", 4, (FOE), ffdf297c><br> <read_kdump: addr: c065a071 paddr: 85a071 cnt: 4></div><div> read_netdump: addr: c065a071 paddr: 85a071 cnt: 4 offset: 65a0e5<br> c065a071: 03e59130 0...</div><div> </div><div>virtual address is c065a071, physical address is 85a071, this is ok.</div><div>but the tool said the file offset is 65a0e5, which is not correct. because my dump binary</div><div>contains ddr content from 0x0, the data of physical address 85a071 should be at dump file</div><div>offset 85a071+74= 85a0e5, rather than 65a0e5. </div><div> </div><div> so i guess the elf header should be modified to set phdr->p_offset =header size + 0x00200000.</div><div>i don't know how to tell crash utility it should add value 0x00200000 when read dump file.</div><div><br>> I only have one ELF ARM dumpfile sample, but it does not have any<br>> physical offset:<br>> <br>> crash> vtop c0000000<br>> VIRTUAL PHYSICAL<br>> c0000000 0<br>> <br>> PAGE DIRECTORY: c0004000<br>> PGD: c0007000 => 1140e<br>> PMD: c0007000 => 1140e<br>> PAGE: 0 (1MB)<br>> <br>> <br>> PAGE PHYSICAL MAPPING INDEX CNT FLAGS<br>> c042d000 0 0 0 0 80000<br>> crash><br>> <br>> Does "vtop c0000000" work as expected on your vmcore?</div><div> </div><div>yes i think the vtop command works well on my side:</div><div> </div><div>crash> vtop c0000000<br>VIRTUAL PHYSICAL<br>c0000000 200000</div><div>PAGE DIRECTORY: c0004000<br> PGD: c0007000 => 21140e<br> PMD: c0007000 => 21140e<br> PAGE: 200000 (1MB)</div><div><br> PAGE PHYSICAL MAPPING INDEX CNT FLAGS<br>c1370800 200000 e5d43061 42 1 80068<br></div><div><br>> <br>> Also, can you read the last physical page of memory? For example, on <br>> my ARM dump, I can check that by doing this:<br>> <br>> crash> kmem -p | tail -5<br>> c04dcf60 57fb000 0 0 1 400<br>> c04dcf80 57fc000 0 0 1 400<br>> c04dcfa0 57fd000 0 0 1 400<br>> c04dcfc0 57fe000 0 0 1 400<br>> c04dcfe0 57ff000 0 0 1 400<br>> crash> rd -p 57ff000<br>> 57ff000: ef9f0000 ....<br>> crash><br>> </div><div> </div><div>result is as below:</div><div> </div><div>crash> kmem -p |tail -5<br>c19b934c 2ccfb000 0 0 1 400<br>c19b9370 2ccfc000 0 0 1 400<br>c19b9394 2ccfd000 0 0 1 400<br>c19b93b8 2ccfe000 0 0 1 400<br>c19b93dc 2ccff000 0 0 1 400</div><div><br>crash> rd -p 2ccff000<br>2ccff000: fffdffff ....<br></div><div><br>> Also, can you confirm that your kernel's symbol list starts<br>> at c0000000, i.e., something like this:<br>> <br>> crash> sym -l<br>> c0004000 (A) swapper_pg_dir<br>> c0008000 (t) .init<br>> c0008000 (T) __init_begin<br>> c0008000 (T) _sinittext<br>> c0008000 (T) _stext<br>> c0008000 (T) stext<br>> c0008040 (t) __create_page_tables<br>> c00080e4 (t) __enable_mmu_loc<br>> c00080f0 (t) __error_a<br>> c00080f4 (t) __lookup_machine_type<br>> c0008128 (t) __lookup_machine_type_data<br>> ...<br>> <br>> I just want to make sure that the kernel symbols actually start<br>> at c000000, and not c2000000.<br>> </div><div> </div><div>yes, the symbols actually start from c0000000:</div><div> </div><div>crash> sym -l<br>c0004000 (A) swapper_pg_dir<br>c0005fb8 (A) __crc_scsi_host_get<br>c0008000 (t) .head.text<br>c0008000 (T) _text<br>c0008000 (T) stext<br>c0008050 (t) __create_page_tables<br>c0008104 (t) __turn_mmu_on_loc<br>c0008110 (T) secondary_startup<br></div><div><br>> > 2. i am wondering how does crash utility translate virtual address to<br>> > physical address before and after it get the kernel page table?<br>> > before get kernel page table, does it calculate as : (virtual_addr -<br>> > p_vaddr + p_paddr) ? after get kernel page table, does it walk<br>> > through the page table and find out the real physical address<br>> > accordingly?<br>> <br>> For kernel unity-mapped kernel virtual addresses, it's not necessary<br>> to walk the page tables. It simply does this:<br>> <br>> #define VTOP(X) \<br>> ((unsigned long)(X)-(machdep->kvbase)+(machdep->machspec->phys_base))<br>> <br>> You can check your machdep->kvbase and machdep->machspec->phys_base<br>> values by entering "help -m", for example:<br>> <br>> crash> help -m | grep -e kvbase -e phys_base<br>> kvbase: c0000000<br>> phys_base: 0<br>> crash><br>> </div><div> </div><div>my result is as below, should be ok:</div><div> </div><div>crash> help -m | grep -e kvbase -e phys_base<br> kvbase: c0000000<br> phys_base: 200000<br></div><div><br>> Certainly vmalloc (and user-space) virtual addresses require a page<br>> table walkthough, but the arm_kvtop() function does this:<br>> <br>> static int<br>> arm_kvtop(struct task_context *tc, ulong kvaddr, physaddr_t *paddr, int verbose)<br>> {<br>> if (!IS_KVADDR(kvaddr))<br>> return FALSE;<br>> <br>> if (!vt->vmalloc_start) {<br>> *paddr = VTOP(kvaddr);<br>> return TRUE;<br>> }<br>> <br>> if (!IS_VMALLOC_ADDR(kvaddr)) {<br>> *paddr = VTOP(kvaddr); <=== unity-mapped kernel virtual addresses<br>> if (!verbose)<br>> return TRUE;<br>> }<br>> <br>> return arm_vtop(kvaddr, (ulong *)vt->kernel_pgd[0], paddr, verbose);<br>> }<br>> <br>> and where vmalloc addresses fall through and arm_vtop() is called to walk<br>> the page tables.<br>> <br>> However, you can translate unity-mapped addresses using the kernel page tables<br>> with the "vtop" command, as shown in the "vtop c000000" example above.<br>> <br>> > 3. my real purpose is to get the ftrace content from dump file by<br>> > crash utility , but seem the command trace is not for this case, do<br>> > i need to compile the extension "trace" of crash utility? is there<br>> > any guide to follow?<br>> <br>> That's correct. You can do this:<br>> <br>> $ wget http://people.redhat.com/anderson/crash-6.1.2.tar.gz<br>> ...<br>> $ tar xvzmf crash-6.1.2.tar.gz<br>> ...<br>> $ cd crash-6.1.2<br>> $ make<br>> ...<br>> $ make extensions<br>> ...<br>> $ ./crash vmlinux vmcore<br>> ...<br>> crash> extend trace.so<br>> ./extensions/trace.so: shared object loaded<br>> crash> help trace<br>> ...</div><div> </div><div>i have made the trace extension work, however, trace show need trace-cmd, </div><div>but in my ubuntu PC, run "sudo apt-get install trace-cmd", i get below error:</div><div>E: Couldn't find package trace-cmd<br></div><div>by Google, i found that there is a project git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git. </div><div>maybe i can only compile this tool and use it ?<br></div><div><br>> <br>> Dave<br>> <br>> --<br>> Crash-utility mailing list<br>> Crash-utility@redhat.com<br>> https://www.redhat.com/mailman/listinfo/crash-utility<br></div> </div></body> </html>