[Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Dave Anderson
anderson at redhat.com
Wed Oct 15 13:53:12 UTC 2008
----- "Kevin Worth" <kevin.worth at hp.com> wrote:
> Hi Dave,
>
> Before you responded I noticed that a simple "make modules" didn't
> work because my kernel wasn't exporting the symbol. Rather than do
> anything risky/complex which might risk mucking up the troubleshooting
> process, I just rebuilt the kernel. It built just fine and now I can
> load crash and I see "DUMPFILE: /dev/crash" when I load up crash. Let
> me try walking through the steps that you had me do previously, this
> time using /dev/crash instead of /dev/mem and /dev/kmem
You made one small error (but not totally fatal) in the suggested steps.
See my comments below...
>
> >From my limited understanding of what's going on here, it would
> appear that the dump file is missing some data, or else crash is
> looking in the wrong place for it.
The crash utility is a slave to what is indicated in the PT_LOAD
segments of the ELF header of the kdump vmcore. In the case of
the physical memory chunk that starts at 4GB physical on your machine,
this is what's in the ELF header (from your original "crash.log" file):
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 3144876760 (bb7302d8)
p_vaddr: ffffffffffffffff
p_paddr: 100000000
p_filesz: 1073741824 (40000000)
p_memsz: 1073741824 (40000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
What that says is: for the range of physical memory starting
at 0x100000000 (p_paddr), the vmcore contains a block of
memory starting at file offset (p_offset) 3144876760/0xbb7302d8
that is 1073741824/0x40000000 (p_filesz) bytes long.
More simply put, the 1GB of physical memory from 4GB to 5GB
can be found in the vmcore file starting at file offset 3144876760.
So if a request for physical memory page 0x100000000 comes
in, the crash utility reads from vmcore file offset 3144876760.
If the next physical page were requested, i.e., at 0x100001000,
it would read from vmcore file offset 3144876760+4096. It's
as simple as that -- so when you suggest that "crash is looking
in the wrong place for it", well, there's nothing that the
crash utility can do differently.
Now, back to the test sequence:
> ---Live system---
>
> KERNEL: vmlinux-devcrash
> DUMPFILE: /dev/crash
> CPUS: 2
> DATE: Tue Oct 14 16:08:28 2008
> UPTIME: 00:02:07
> LOAD AVERAGE: 0.17, 0.08, 0.03
> TASKS: 97
> NODENAME: test-machine
> RELEASE: 2.6.20-17.39-custom2
> VERSION: #1 SMP Tue Oct 14 13:45:17 PDT 2008
> MACHINE: i686 (2200 Mhz)
> MEMORY: 5 GB
> PID: 5628
> COMMAND: "crash"
> TASK: 5d4c2560 [THREAD_INFO: f3de6000]
> CPU: 1
> STATE: TASK_RUNNING (ACTIVE)
>
> crash> p modules
> modules = $2 = {
> next = 0xf8a3ea04,
> prev = 0xf8842104
> }
>
> crash> module 0xf8a3ea00
> struct module {
> state = MODULE_STATE_LIVE,
> list = {
> next = 0xf8d10484,
> prev = 0x403c63a4
> },
> name =
> "crash\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\
> 000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\
> 000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
> mkobj = {
> kobj = {
> k_name = 0xf8a3ea4c "crash",
> name =
> "crash\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
> kref = {
> refcount = {
> counter = 3
> }
> },
> entry = {
> next = 0x403c6068,
> prev = 0xf8d104e4
> },
> parent = 0x403c6074
> ...
>
> crash> vtop 0xf8a3ea00
> VIRTUAL PHYSICAL
> f8a3ea00 116017a00
OK -- so the physical memory location of the module data structure
is at physical address 116017a00, but...
>
> PAGE DIRECTORY: 4044b000
> PGD: 4044b018 => 6001
> PMD: 6e28 => 1d51a067
> PTE: 1d51a1f0 => 116017163
> PAGE: 116017000
>
> PTE PHYSICAL FLAGS
> 116017163 116017000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL)
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> 472c02e0 116017000 0 229173 1 80000000
>
You're reading from the beginning of the page, i.e., 116017000
instead of where the module structure is at 116017a00:
> crash> rd -p 116017000 30
> 116017000: 53e58955 d089c389 4d8bca89 74c98508 U..S.......M...t
> 116017010: 01e9831f b85b0d74 ffffffea ffffba5d ....t.[.....]...
> 116017020: 03c3ffff 53132043 26b48d24 00000000 ....C .S$..&....
> 116017030: 89204389 5d5b2453 26b48dc3 00000000 .C .S$[]...&....
> 116017040: 83e58955 55892cec 08558be4 89f45d89 U....,.U..U..]..
> 116017050: 7d89f875 ffeabffc 4d89ffff 8b028be0 u..}.......M....
> 116017060: c3890452 ac0fd689 45890cf3 0ceec1ec R..........E....
> 116017070: 5589c889 89d231f0 ...U.1..
> crash>
>
So therefore you're not seeing the "crash" strings embedded in
the raw physical data. Now, although it would have been "nice"
if you could have shown the contents of the module structure via
the physical address, the fact remains that since you used the
/dev/crash driver, the "module 0xf8a3ea00" command required that
the crash utility first translate the vmalloc address into its
physical equivalent, and then read from there.
In any case, you do have a dump of physical memory from 116017000
which at least is in the same 4k page as the module data structure,
so it should not change when read from the dumpfile.
> ---Using dump file---
>
>
> please wait... (gathering module symbol data)
> WARNING: cannot access vmalloc'd module memory
>
> KERNEL: vmlinux-devcrash
> DUMPFILE: /var/crash/vmcore
> CPUS: 2
> DATE: Tue Oct 14 16:09:32 2008
> UPTIME: 00:03:12
> LOAD AVERAGE: 0.09, 0.08, 0.02
> TASKS: 97
> NODENAME: test-machine
> RELEASE: 2.6.20-17.39-custom2
> VERSION: #1 SMP Tue Oct 14 13:45:17 PDT 2008
> MACHINE: i686 (2200 Mhz)
> MEMORY: 5 GB
> PANIC: "[ 192.148000] SysRq : Trigger a crashdump"
> PID: 0
> COMMAND: "swapper"
> TASK: 403c0440 (1 of 2) [THREAD_INFO: 403f2000]
> CPU: 0
> STATE: TASK_RUNNING (SYSRQ)
>
> crash> p modules
> modules = $2 = {
> next = 0xf8a3ea04,
> prev = 0xf8842104
> }
>
> crash> module 0xf8a3ea00
> struct module {
> state = MODULE_STATE_LIVE,
> list = {
> next = 0x0,
> prev = 0x0
> },
> name =
> "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0
> 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0
> 00\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0
> 00\000",
> mkobj = {
> kobj = {
> k_name = 0x0,
> name =
> "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0
> 00\000\000",
> kref = {
> refcount = {
> counter = 0
> }
> },
> entry = {
> next = 0x0,
> prev = 0x0
> ...
>
> crash> vtop 0xf8a3ea00
> VIRTUAL PHYSICAL
> f8a3ea00 116017a00
>
> PAGE DIRECTORY: 4044b000
> PGD: 4044b018 => 6001
> PMD: 6e28 => 1d51a067
> PTE: 1d51a1f0 => 116017163
> PAGE: 116017000
>
> PTE PHYSICAL FLAGS
> 116017163 116017000 (PRESENT|RW|ACCESSED|DIRTY|GLOBAL)
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> 472c02e0 116017000 0 229173 1 80000000
>
> crash> rd -p 116017000 30
> 116017000: 00000000 00000000 00000000 00000000 ................
> 116017010: 00000000 00000000 00000000 00000000 ................
> 116017020: 00000000 00000000 00000000 00000000 ................
> 116017030: 00000000 00000000 00000000 00000000 ................
> 116017040: 00000000 00000000 00000000 00000000 ................
> 116017050: 00000000 00000000 00000000 00000000 ................
> 116017060: 00000000 00000000 00000000 00000000 ................
> 116017070: 00000000 00000000 ........
> crash>
Now we're reading the same physical address as you did on
the dumpfile, and it's returning all zeroes. And the
"module 0xf8a3ea00" above shows all zeroes from a higher
location in the page because the same vmalloc translation is
done to turn it into a physical address before reading it
from the vmcore file. But instead of using the /dev/crash driver
to access the translated physical memory, the crash utility
uses the information from the ELF header's PT_LOAD segments
to find out where to find the page data in the vmcore file.
So, anyway, the "rd -p 116017000 30" command that you did
on both the live system and the dumpfile should yield the same
data.
It seems like in all examples to date, the file data read
at the greater-than-4GB PT_LOAD segment returns zeroes.
You can verify this from the crash utility's viewpoint by
doing a "help -n" during runtime when running with the dumpfile,
which will show you both the actual contents of the ELF header,
as well as the manner in which the PT_LOAD data is stored for
its use. (It's also shown with the "crash -d7 ..." output).
So again, from your original "crash.log" file, here is what the
ELF header's PT_LOAD segment contains:
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 3144876760 (bb7302d8)
p_vaddr: ffffffffffffffff
p_paddr: 100000000
p_filesz: 1073741824 (40000000)
p_memsz: 1073741824 (40000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
And this is what the crash utility stored in its internal
data structure for that particular segment:
pt_load_segment[4]:
file_offset: bb7302d8
phys_start: 100000000
phys_end: 140000000
zero_fill: 0
And when the physical memory read request comes in, it filters
to this part of the crash utility's read_netdump() function in
netdump.c:
for (i = offset = 0; i < nd->num_pt_load_segments; i++) {
pls = &nd->pt_load_segments[i];
if ((paddr >= pls->phys_start) &&
(paddr < pls->phys_end)) {
offset = (off_t)(paddr - pls->phys_start) +
pls->file_offset;
break;
}
if (pls->zero_fill && (paddr >= pls->phys_end) &&
(paddr < pls->zero_fill)) {
memset(bufptr, 0, cnt);
return cnt;
}
}
So for any physical address request between 100000000 to 140000000,
(4GB to 5GB) it will calculate the offset to seek to by subtracting
100000000 from the incoming physical address, and adding the difference
to the starting file offset of the whole segment.
So if you wanted to, you could put debug code just prior to the "break" above
that shows the pls->file_offset for a given incoming physical address.
But this code has been in place forever, so it's hard to conceive that
somehow it's not working in the case of this dumpfile. But presuming that
it *does* go to the correct file offset location in the vmcore, and it's
getting bogus data from there, then there's nothing that the crash
utility can do about it.
Dave
More information about the Crash-utility
mailing list