<!doctype html public "-//w3c//dtd html 4.0 transitional//en"> <html> <tt>Magnus Damm wrote:</tt> <blockquote TYPE=CITE><tt>On Tue, 2006-10-24 at 09:45 -0400, Dave Anderson wrote:</tt> <tt>> Magnus Damm wrote:</tt> <tt>> > The idea is that the crash_notes contents in the Xen hypervisor</tt> <tt>> space</tt> <tt>> > contains registers indexed by physical cpu number.</tt> <tt>> ></tt> <tt>> > It is possible to locate the crashing physical cpu by looking up a</tt> <tt>> > global variable in hypervisor symbol, and from there it should be</tt> <tt>> > possible to backtrack and find the domain pseudo-phys to virt</tt> <tt>> mapping</tt> <tt>> > table. I say "should" because it is probably pretty hairy.</tt> <tt>> ></tt> <tt>></tt> <tt>> Actually, given that the crash utility is only interested in the</tt> <tt>> specifics of the dom0 kernel, it has no interest in physical</tt> <tt>> cpus. If you're specifically interested in debugging a crash</tt> <tt>> that occurred while operating in the xen binary, you're going</tt> <tt>> to want to use gdb on the vmcore file with xen-syms-xxx</tt> <tt>> namelist file. You can still run crash on the same vmcore to</tt> <tt>> find out what was going on in the dom0 kernel, but there's</tt> <tt>> no awareness of the xen hypervisor underpinnings; you'll</tt> <tt>> just get the state of the dom0 kernel at the time of the crash.</tt><tt></tt> <tt>Exactly. But for gdb to work we need to provide crash notes to the xen</tt> <tt>vmcore file - and with physical cpu crash notes in this case. I just</tt> <tt>hacked up some code to do this and it seems like the default number of</tt> <tt>cpus in xen-unstable is set to 32. That's 32 crash notes.</tt> <tt></tt> </blockquote> <tt>That's fine -- the crash utility will ignore the registers in the</tt> <tt>NT_PRSTATUS notes if it can determine that the vmcore is</tt> <tt>a hypervisor/dom0/kdump-type dumpfile. In my prototype, it</tt> <tt>does just that when it sees the unique NT_XEN_DOM0_CR3 note.</tt> <tt>(crash doesn't really need the NR_PRSTATUS register contents,</tt> <tt>because it can find them elsewhere if need be.)</tt> <blockquote TYPE=CITE><tt></tt> <tt>> But I would guess-timate that the majority of the crashes are</tt> <tt>> going to have occurred in the dom0 kernel, and not while</tt> <tt>> running in the hypervisor.</tt><tt></tt> <tt>Yeah, given the number of lines of code...</tt><tt></tt> <tt>> Now, given that that the crash_notes context contains registers</tt> <tt>> that are indexed by the physical cpu number, well, that's not</tt> <tt>> helpful to crash's needs with respect to dom0. That's why you</tt> <tt>> guys must have created the additional NT_XEN_DOM0_CR3</tt> <tt>> ELF note.</tt><tt></tt> <tt>I'm note sure exactly why we created it - I thought it was because you</tt> <tt>wanted it - but I'm pretty sure Moriwaka-sans tool can locate things</tt> <tt>without it.</tt> <tt></tt> </blockquote> <tt>That's exactly right -- I requested dom0's cr3 or pfn_to_mfn_list_list</tt> <tt>value, and Moriwaka-san provided me with a prototype vmcore that</tt> <tt>contained the NT_XEN_DOM0_CR3 ELF note.</tt><tt></tt> <tt>I presume that the i386 vmcore he sent me was also usable with</tt> <tt>gdb and xen-syms. Here's what the header looks like:</tt><tt></tt> <tt>$ readelf -a vmcore</tt> <tt>ELF Header:</tt> <tt> Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00</tt> <tt> Class: ELF32</tt> <tt> Data: 2's complement, little endian</tt> <tt> Version: 1 (current)</tt> <tt> OS/ABI: UNIX - System V</tt> <tt> ABI Version: 0</tt> <tt> Type: CORE (Core file)</tt> <tt> Machine: Intel 80386</tt> <tt> Version: 0x1</tt> <tt> Entry point address: 0x0</tt> <tt> Start of program headers: 52 (bytes into file)</tt> <tt> Start of section headers: 0 (bytes into file)</tt> <tt> Flags: 0x0</tt> <tt> Size of this header: 52 (bytes)</tt> <tt> Size of program headers: 32 (bytes)</tt> <tt> Number of program headers: 5</tt> <tt> Size of section headers: 0 (bytes)</tt> <tt> Number of section headers: 0</tt> <tt> Section header string table index: 0</tt><tt></tt> <tt>There are no sections in this file.</tt><tt></tt> <tt>Program Headers:</tt> <tt> Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align</tt> <tt> NOTE 0x0000d4 0x00000000 0x00000000 0x00190 0x00190 0</tt> <tt> LOAD 0x000264 0xc0000000 0x00000000 0xa0000 0xa0000 RWE 0</tt> <tt> LOAD 0x0a0264 0xc0100000 0x00100000 0x3f00000 0x3f00000 RWE 0</tt> <tt> LOAD 0x3fa0264 0xc8000000 0x08000000 0x30000000 0x30000000 RWE 0</tt> <tt> LOAD 0x33fa0264 0xffffffff 0x38000000 0x7ffb000 0x7ffb000 RWE 0</tt><tt></tt> <tt>There is no dynamic segment in this file.</tt><tt></tt> <tt>There are no relocations in this file.</tt><tt></tt> <tt>There are no unwind sections in this file.</tt><tt></tt> <tt>No version information found in this file.</tt><tt></tt> <tt>Notes at offset 0x000000d4 with length 0x00000190:</tt> <tt> Owner Data size Description</tt> <tt> CORE 0x00000090 NT_PRSTATUS (prstatus structure)</tt> <tt> Xen Domanin-0 CR3 0x00000004 Unknown note type: (0x10000001)</tt> <tt> CORE 0x00000090 NT_PRSTATUS (prstatus structure)</tt> <tt> Xen Domanin-0 CR3 0x00000004 Unknown note type: (0x10000001)</tt> <tt>$</tt> <tt></tt> <blockquote TYPE=CITE><tt></tt> <tt>> I guess I understand why you feel it's a burden to continue</tt> <tt>> the maintenance of such a thing, but given that the panic can</tt> <tt>> occur either while operating in the dom0 kernel or while in</tt> <tt>> the xen hypervisor code, it makes perfect sense (to me) to</tt> <tt>> make a minimal effort by including an indication of the dom0</tt> <tt>> cr3 or dom0 pfn_to_mfn_frame_list_list value in the vmcore.</tt> <tt>> Perhaps you consider it a case of the tail wagging the dog,</tt> <tt>> but to me, it would be more a case of accomodating the needs</tt> <tt>> of the consumer... ;-)</tt> <tt>></tt><tt></tt> <tt>We should definately have something, we just need to figure out what. =)</tt><tt></tt> <tt>> ></tt> <tt>> > Our internal interfaces are not particularly clean at the moment. We</tt> <tt>> > have code that keeps the crash_notes in the hypervisor, but passes</tt> <tt>> the</tt> <tt>> > physical addresses (or machine addresses in xen lingo) for the notes</tt> <tt>> all</tt> <tt>> > the way down to kexec-tools in dom0 user space. These addresses are</tt> <tt>> then</tt> <tt>> > used to create the ELF headers. dom0 only knows about VCPU:s, but</tt> <tt>> > because we are creating a system-wide crash dump we want to use</tt> <tt>> physical</tt> <tt>> > cpus. So down in user space we then need to create a mapping between</tt> <tt>> > physical cpu:s and VCPU:s. And can we be sure that dom0 has all cpus</tt> <tt>> > available as VCPU:s?</tt> <tt>> ></tt> <tt>></tt> <tt>> I don't care about that. All I need is a starting point for</tt> <tt>> translating dom0</tt> <tt>> kernel virtual addresses. And that is either a dom0 cr3 value or the</tt> <tt>> domain's pfn_to_mfn_frame_list_list value.</tt><tt></tt> <tt>I understand that. I'm sure we can figure out a good solution.</tt><tt></tt> <tt>> ></tt> <tt>> > > But again, there's no easy way for the crash utility to dig</tt> <tt>> > > them out of a completely foreign binary's.</tt> <tt>> ></tt> <tt>> > No, but that's because your tool is missing knowledge about the</tt> <tt>> binary</tt> <tt>> > right? =) Is there any easy way out... No! =) Or maybe there is?</tt> <tt>></tt> <tt>> No!</tt> <tt>></tt> <tt>> ></tt> <tt>> ></tt> <tt>> > I hope we can find a good balance between your code and ours. Maybe</tt> <tt>> a</tt> <tt>> > relatively fair balance could be that we provide per-physical cpu</tt> <tt>> > pointers to some virtual to physical mapping tables which should be</tt> <tt>> easy</tt> <tt>> > to parse for your tool, but in return your tool doesn't depend on</tt> <tt>> > finding register information using the note program headers in the</tt> <tt>> ELF</tt> <tt>> > header...</tt> <tt>></tt> <tt>> Now we're getting complex -- I'm pretty sure I don't know what</tt> <tt>> you're talking about here... Or how it can possibly lead to a dom0</tt> <tt>> cr3 or pfn_to_mfn_frame_list_list value?</tt><tt></tt> <tt>Let's put it like this: Your tool, does it use xen-syms today? How</tt> <tt>difficult would it be to use lookup a symbol in hypervisor space? And</tt> <tt>how do you feel about requiring xen-syms to be able to parse dom0?</tt> <tt></tt> </blockquote> <tt>Definitely not. In the interest of simplicity, the idea is to keep</tt> <tt>things the way they have always been, i.e., "crash vmlinux vmcore",</tt> <tt>and having to drag out the relevant xen-syms binary defeats that</tt> <tt>concept. And that's not to mention having to read it, figure out</tt> <tt>how to translate hypervisor virtual addresses, then navigate around</tt> <tt>the vmcore to find the location of a symbol, having to deal with the</tt> <tt>following of data structures that may change over time -- to eventually</tt> <tt>find what's needed...</tt> <blockquote TYPE=CITE><tt></tt> <tt>I'm not talking about your tool walking arch-dependent cross linked data</tt> <tt>structures in the xen hypervisor, just basic symbol lookup.</tt> <tt></tt> </blockquote> <tt>But I'm not sure what good would that do? The "domain_list" symbol</tt> <tt>is only the beginning of the structure chain...</tt> <blockquote TYPE=CITE><tt></tt> <tt>> > That's good, isn't it? If I've understood things right it's possible</tt> <tt>> to</tt> <tt>> > locate the data you need using the domain list symbol?</tt> <tt>></tt> <tt>> Yeah...</tt> <tt>></tt> <tt>> To clarify, it's possible for *you*, i.e., the kexec/kdump code, to</tt> <tt>> locate</tt> <tt>> the data that way. The crash utility, using the vmlinux/vmcore file</tt> <tt>> pair, doesn't know anything about what the "domain_list" is, the</tt> <tt>> structures that it uses/links-to. And even if it did, it wouldn't</tt> <tt>> know</tt> <tt>> how to find it in the vmcore file.</tt><tt></tt> <tt>Hm...</tt><tt></tt> <tt>> > Yeah, I agree that navigating around those structures seems rather</tt> <tt>> > painful. But OTOH, if you want to know things that only the</tt> <tt>> internals</tt> <tt>> > can tell you, you need to be able to parse them, right? But maybe</tt> <tt>> you</tt> <tt>> > only want to cover the "simple" dom0 case. (Simple yeah right)</tt> <tt>> ></tt> <tt>></tt> <tt>> That's right -- crash is *only* interested in the dom0 case; again</tt> <tt>> it's clueless about the hypervisor, and rightly so.</tt> <tt>></tt> <tt>> It's just such a unique case. It's like trying to debug "ls" using</tt> <tt>> a "cat" binary, where the core file is usable for debugging either</tt> <tt>> one.</tt><tt></tt> <tt>I will continue working on cleaning up the code. Some of the changes</tt> <tt>that have been made or are going to happen are:</tt><tt></tt> <tt>- Separate dom0 notes from hypervisor notes.</tt></blockquote> <tt>Looks like a reasonable spot to stuff cr3 (or pfn_to_mfn_frame_list_list)?</tt> <tt>(i.e. stuffing the dom0 cr3 at the end of the dom0 NT_PRSTATUS</tt> <tt>register dump(s), like your current patch talks about)</tt> <blockquote TYPE=CITE><tt></tt> <tt>- Xen vmcore files should have per physical cpu hypervisor crash_notes.</tt> <tt>- These hypervisor notes should be pointed out by the program header.</tt> <tt>- Xen vmcore files should have the hypervisor in a PT_LOAD segment.</tt> <tt>- Xen vmcore files should have PT_LOAD headers for RAM, but with VA = 0.</tt></blockquote> <tt>With respect to the PT_LOAD segments, the crash utility pretty much</tt> <tt>ignores the p_vaddr values -- it's particularly interested in the</tt> <tt>p_paddr values. And once able to translate a unity-mapped kernel</tt> <tt>virtual address into a physical address, it can find that physical</tt> <tt>address by checking the p_paddr/p_filesz values in each PT_LOAD</tt> <tt>segment. (Again -- that only works with xen kernels if the pseudo</tt> <tt>physical address can be translated into a machine address, and then</tt> <tt>a vmcore file location -- hence the pre-requisite requirement for</tt> <tt>either the dom0 cr3 or pfn_to_mfn_list_list value.)</tt><tt></tt> <tt>The one exception to "ignore-the-p_vaddr" rule is for the new</tt> <tt>relocatable x86_64 kernels, where the PT_LOAD mapping for the</tt> <tt>kernel's __START_KERNEL_map region needs to be looked at in</tt> <tt>order to figure out where the kernel was physically loaded,</tt> <tt>because in relocatable kernels, unity-mapping can no longer be</tt> <tt>done by simply stripping off the PAGE_OFFSET value. Relocatable</tt> <tt>kernels are being introduced so that there is no need for separate</tt> <tt>"first-kernel" and the secondary kexec'd "kdump-kernel".</tt><tt></tt> <tt>But AFAIK, the new kernel relocation scheme shouldn't infect xen</tt> <tt>x86_64 kernels, so I don't believe that would ever be an issue.</tt> <tt>In other words, xen kernels should remain unity-mapped in the</tt> <tt>traditional manner, albeit with pseudo-physical addresses instead</tt> <tt>of machine addresses. Again, I *believe* that to be true, since</tt> <tt>there's no reason to relocate them.</tt> <blockquote TYPE=CITE><tt></tt> <tt>- Xen vmcore files should provide crash with something like CR3.</tt></blockquote> <tt>Or this would be good... ;-)</tt> <blockquote TYPE=CITE><tt></tt> <tt>- kexec-tools will be made xen aware.</tt><tt></tt> <tt>I'm sorry if we've been going over and over about this, but I'm a bit</tt> <tt>confused by the current status, what we want to have and if that is</tt> <tt>going to work. The points above show the direction. Please shout if you</tt> <tt>think they sound bad.</tt></blockquote> <tt>Nothing sounds bad to me! What would be really helpful is if,</tt> <tt>during your development, you could provide me with sample</tt> <tt>vmlinux/vmcore pairs that I could work with? Just so I don't</tt> <tt>have to scramble at the end of it all, and only then find out</tt> <tt>that there's a "gotcha" that we're not considering.</tt><tt></tt> <tt>Thanks,</tt> <tt> Dave</tt> <tt></tt> </html>