[Crash-utility] Unable to switch stack frames while using crash
Shashidhara Shamaiah
Shashidhara.Shamaiah at mphasis.com
Mon Jun 27 05:22:59 UTC 2011
Hi Dave,
Thanks again for the response.
When I check the address of tty->read_buf using kmem command in crash,
below is the output. Which shows that the memory is still not freed. If
the tty_close handler had been invoked, before accessing tty->read_buf
we should not get this output. Please correct me if I am erring.
kmem 0xffff8802cbfe6000
CACHE NAME OBJSIZE ALLOCATED TOTAL
SLABS SSIZE
ffff88031f8039c0 size-4096 4096 6322 6393
6393 4k
SLAB MEMORY TOTAL ALLOCATED FREE
ffff8802ac4811c0 ffff8802cbfe6000 1 1 0
FREE / [ALLOCATED]
[ffff8802cbfe6000]
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0009c9fa50 2cbfe6000 0 0 1 200000000000080
I would like to thank every one for your time and effort in analyzing
this issue.
Thanks and Regards
Shashidhara
-----Original Message-----
From: crash-utility-bounces at redhat.com
[mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
Sent: Friday, June 24, 2011 7:10 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Unable to switch stack frames while using
crash
---- Original Message -----
>>
>> BTW, are you sure about that?
>>
>> Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown
below,
>> and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail
is 0,
>> then the statement above would be simply be reading tty->read_buf[0],
or
>> virtual address 0xffff8802cbfe6000. But the oops shows it faulting
on a
>> virtual address of "5":
Well, as it turns out, you have every reason to be sure about that...
Anyway, I don't understand why line numbers are not available with
your kernel:
> crash> dis -rl n_tty_read+0x58c
> dis: line numbers are not available
... [ cut ] ...
> 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
But nonetheless, there is only on movsbl instruction in n_tty_read(),
and looking at a RHEL6 kernel, you were correct in your original
determination of the faulting instruction:
crash> dis n_tty_read | grep movsbl
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash> dis -rl 0xffffffff812f88c9 | tail
... [ cut ] ...
/usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c:
1821
0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx
crash>
which is here:
1814 if (tty->icanon) {
1815 /* N.B. avoid overrun if nr == 0 */
1816 while (nr && tty->read_cnt) {
1817 int eol;
1818
1819 eol =
test_and_clear_bit(tty->read_tail,
1820
tty->read_flags);
1821 c =
tty->read_buf[tty->read_tail];
The tty_struct offsets are these:
crash> tty_struct -o
struct tty_struct {
... [ cut ]...
[0x250] char *read_buf;
[0x258] int read_head;
[0x25c] int read_tail;
...
And you can see in the previous instructions the tty->read_buf (0x250)
and tty->read_tail (0x25c) offsets being added to the tty_struct
base address that is located in R13:
> 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
And as you originally reported, the tty_struct address in R13
is ffff8802cbd54800:
PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
#0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
#1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
#2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
#3 [ffff88031ce75b50] no_context at ffffffff8102d801
#4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
#5 [ffff88031ce75c70] bad_area at ffffffff8102da41
#6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
#7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
[exception RIP: n_tty_read+1420]
RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
#9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
#10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
#11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
But for whatever reason -- and I cannot explain it -- after these
instructions were done:
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
the RDX register ended up with 0000000000000005, and the RAX register
with
a 0000000000000000, leading to the:
BUG: unable to handle kernel NULL pointer dereference at
0000000000000005
But when you display the tty_struct at ffff8802cbd54800, you see the
read_buf and read_tail with seemingly legitimate values:
crash> tty_struct ffff8802cbd54800
struct tty_struct { ...
magic = 21505,
driver = 0xffff88031b54ea00,
ops = 0xffffffff8130f650,
name = "pts9\000\...",
driver_data = 0xffff88029c8a9668,
icanon = 1 '\001',
read_buf = 0xffff8802cbfe6000 "",
read_head = 0,
read_tail = 0,
read_cnt = 0,
...
So everything in your analysis was correct, but how it is possible
that the RDX and RAX registers to have ended up with 0 and 5 is hard
to explain. And for that matter, since tty->read_cnt is 0 above,
your original question as to how that code path was taken to
begin with is also valid.
So I don't know -- anybody on the list ever seen anything like this?
Stumped,
Dave
----- Original Message -----
> Hi Dave,
>
> Thank you so much for your help.
>
> Below is the output of dis -rl n_tty_read+0x58c
>
> crash> dis -rl n_tty_read+0x58c
> dis: line numbers are not available
> 0xffffffff811efe27 <n_tty_read>: push %rbp
> 0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax
> 0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp
> 0xffffffff811efe34 <n_tty_read+13>: push %r15
> 0xffffffff811efe36 <n_tty_read+15>: push %r14
> 0xffffffff811efe38 <n_tty_read+17>: push %r13
> 0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13
> 0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi
> 0xffffffff811efe41 <n_tty_read+26>: push %r12
> 0xffffffff811efe43 <n_tty_read+28>: push %rbx
> 0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx
> 0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp
> 0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp)
> 0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp)
> 0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax
> 0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx
> 0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp)
> 0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp)
> 0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp)
> 0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi)
> 0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax
> 0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx
> 0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp)
> 0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx
> 0xffffffff811efe94 <n_tty_read+109>: movq
> $0xffffffff81045f84,-0x60(%rbp)
> 0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp)
> 0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx
> 0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp)
> 0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax
> 0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp)
> 0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx
> 0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp)
> 0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp)
> 0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx
> 0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp)
> 0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp)
> 0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp)
> 0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp)
> 0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp)
> 0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp)
> 0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13)
> 0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11
> <n_tty_read+234>
> 0xffffffff811eff0d <n_tty_read+230>: ud2a
> 0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f
> <n_tty_read+232>
> 0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx
> 0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax
> 0xffffffff811eff1c <n_tty_read+245>: cmpq
> $0xffffffff811ed61f,0x18(%rax)
> 0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx
> 0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax
> 0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax)
> 0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx
> 0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx
> 0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64
> <n_tty_read+317>
> 0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi
> 0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax
> 0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf
> <printk>
> 0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx
> 0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax
> 0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax)
> 0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef
> <n_tty_read+456>
> 0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax
> 0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax)
> 0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611
> <n_tty_read+2026>
> 0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax
> 0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax)
> 0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604
> <n_tty_read+2013>
> 0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611
> <n_tty_read+2026>
> 0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx
> 0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx
> 0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi
> 0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax
> 0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi
> 0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a
> <kill_pgrp>
> 0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx
> 0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax
> 0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx)
> 0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax
> 0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616
> <n_tty_read+2031>
> 0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13)
> 0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f
> <n_tty_read+488>
> 0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp)
> 0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp)
> 0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076
> <n_tty_read+591>
> 0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx
> 0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax
> 0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax
> 0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp)
> 0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx
> 0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx
> 0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp)
> 0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082
> <n_tty_read+603>
> 0xffffffff811f002e <n_tty_read+519>: test %eax,%eax
> 0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e
> <n_tty_read+535>
> 0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13)
> 0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076
> <n_tty_read+591>
> 0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx
> 0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13)
> 0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068
> <n_tty_read+577>
> 0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax
> 0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15
> 0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax
> 0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7
> <n_tty_read+656>
> 0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax
> 0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13)
> 0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15
> 0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7
> <n_tty_read+656>
> 0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15
> 0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp)
> 0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax
> 0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13)
> 0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp)
> 0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp)
> 0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15
> 0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx
> 0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx)
> 0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4
> <n_tty_read+701>
> 0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi
> 0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7
> <mutex_trylock>
> 0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax
> 0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104
> <n_tty_read+733>
> 0xffffffff811f00d4 <n_tty_read+685>: movq
> $0xfffffffffffffff5,-0xa8(%rbp)
> 0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb
> <n_tty_read+1988>
> 0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi
> 0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358
> <mutex_lock_interruptible>
> 0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax
> 0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104
> <n_tty_read+733>
> 0xffffffff811f00f4 <n_tty_read+717>: movq
> $0xfffffffffffffe00,-0xa8(%rbp)
> 0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb
> <n_tty_read+1988>
> 0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al
> 0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi
> 0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi
> 0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al
> 0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx
> 0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx
> 0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp)
> 0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b
> <add_wait_queue>
> 0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx
> 0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax
> 0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx
> 0xffffffff811f013e <n_tty_read+791>: inc %rdx
> 0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp)
> 0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp)
> 0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp)
> 0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522
> <n_tty_read+1787>
> 0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp)
> 0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef
> <n_tty_read+968>
> 0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax
> 0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax)
> 0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef
> <n_tty_read+968>
> 0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx
> 0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp)
> 0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi
> 0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8
> <_spin_lock_irqsave>
> 0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi
> 0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi
> 0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl
> 0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi)
> 0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi
> 0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1
> <_spin_unlock_irqrestore>
> 0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12
> 0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi
> 0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx
> 0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi
> 0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp)
> 0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax
> 0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp)
> 0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac
> <tty_audit_add_data>
> 0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al
> 0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx
> 0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0
> <__put_user_1>
> 0xffffffff811f01de <n_tty_read+951>: test %eax,%eax
> 0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d
> <n_tty_read+1558>
> 0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp)
> 0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax
> 0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx
> 0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx)
> 0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx
> 0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax
> 0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx
> 0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax
> 0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx
> 0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax
> 0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx
> 0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240
> <n_tty_read+1049>
> 0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx
> 0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240
> <n_tty_read+1049>
> 0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax
> 0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax
> 0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax
> 0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13)
> 0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi
> 0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3
> <tty_flush_to_ldisc>
> 0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13)
> 0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261
> <n_tty_read+1082>
> 0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13)
> 0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f
> <n_tty_read+1096>
> 0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621
> <n_tty_read+2042>
> 0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13)
> 0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621
> <n_tty_read+2042>
> 0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax
> 0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax)
> 0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp)
> 0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax
> 0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376
> <n_tty_read+1359>
> 0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax
> 0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376
> <n_tty_read+1359>
> 0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b
> <n_tty_read+1300>
> 0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi
> 0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980
> <tty_hung_up_p>
> 0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax
> 0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15
> 0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx
> 0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx)
> 0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da
> <n_tty_read+1203>
> 0xffffffff811f02ca <n_tty_read+1187>: movq
> $0xfffffffffffffff5,-0xa8(%rbp)
> 0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx
> 0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax
> 0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax)
> 0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb
> <n_tty_read+1236>
> 0xffffffff811f02eb <n_tty_read+1220>: movq
> $0xfffffffffffffe00,-0xa8(%rbp)
> 0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d
> <n_tty_read+1798>
> 0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax
> 0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax
> 0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax
> 0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324
> <n_tty_read+1277>
> 0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax
> 0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13)
> 0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324
> <n_tty_read+1277>
> 0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax
> 0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13)
> 0xffffffff811f0321 <n_tty_read+1274>: sete %al
> 0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi
> 0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13)
> 0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02
> <schedule_timeout>
> 0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15
> 0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522
> <n_tty_read+1787>
> 0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx
> 0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi
> 0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx
> 0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi
> 0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp)
> 0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp)
> 0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac
> <tty_audit_add_data>
> 0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al
> 0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx
> 0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0
> <__put_user_1>
> 0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax
> 0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d
> <n_tty_read+1558>
> 0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp)
> 0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13)
> 0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456
> <n_tty_read+1583>
> 0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a
> <n_tty_read+1619>
> 0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax
> 0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx
> 0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx)
> 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d
> 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx
> 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax
> 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi
> 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx
>
> Below is the output of bt -a command in crash
>
> bt -a
> PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper"
> #0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9
> #1 [ffff880033007e90] notifier_call_chain at ffffffff81065893
> #2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd
> #3 [ffff880033007ee0] notify_die at ffffffff8106597f
> #4 [ffff880033007f10] do_nmi at ffffffff8100dc5d
> #5 [ffff880033007f50] nmi at ffffffff812d76b0
> [exception RIP: mwait_idle+163]
> RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308
> RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290
> R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
> R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029
> #7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21
>
> PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet"
> #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486
> #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230
> #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38
> #3 [ffff88031ce75b50] no_context at ffffffff8102d801
> #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9
> #5 [ffff88031ce75c70] bad_area at ffffffff8102da41
> #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19
> #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425
> [exception RIP: n_tty_read+1420]
> RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044
> RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c
> RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044
> R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e
> #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8
> #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48
> #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2
> RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212
> RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000
> RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000
> RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0
> ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
>
> PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper"
> #0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9
> #1 [ffff880033047e90] notifier_call_chain at ffffffff81065893
> #2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd
> #3 [ffff880033047ee0] notify_die at ffffffff8106597f
> #4 [ffff880033047f10] do_nmi at ffffffff8100dc5d
> #5 [ffff880033047f50] nmi at ffffffff812d76b0
> [exception RIP: mwait_idle+163]
> RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308
> RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290
> R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029
> #7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21
>
> PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper"
> #0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9
> #1 [ffff880033067e90] notifier_call_chain at ffffffff81065893
> #2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd
> #3 [ffff880033067ee0] notify_die at ffffffff8106597f
> #4 [ffff880033067f10] do_nmi at ffffffff8100dc5d
> #5 [ffff880033067f50] nmi at ffffffff812d76b0
> [exception RIP: mwait_idle+163]
> RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308
> RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000
> R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
> --- <NMI exception stack> ---
> #6 [ffff88031e115ef8] mwait_idle at ffffffff81013029
> #7 [ffff88031e115f10] cpu_idle at ffffffff8100af21
>
> Please let me know if you need any other details.
>
> Thanks and Regards
> Shashidhara
>
>
> -----Original Message-----
> From: crash-utility-bounces at redhat.com
> [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
> Sent: Thursday, June 23, 2011 9:35 PM
> To: Discussion list for crash utility usage,maintenance and
> development
> Subject: Re: [Crash-utility] Unable to switch stack frames while using
> crash
>
>
>
> ----- Original Message -----
>
> > BTW, are you sure about that?
> >
> > Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown
> below,
> > and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail
> is 0,
> > then the statement above would be simply be reading
> > tty->read_buf[0],
> or
> > virtual address 0xffff8802cbfe6000. But the oops shows it faulting
> > on
> a
> > virtual address of "5":
> >
> > BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000005
>
> Just for my own sanity, can you either attach the
> "drivers/char/n_tty.c"
>
> from *your* specific kernel, or get the source-code/line-number data
> from
> the embedded gdb module?
>
> If you don't have the n_tty.c file readily available, you can get the
> source-code/line-number data of a particular function by doing
> something
> like this:
>
> Get the line number of the beginning of n_tty_read(), which in my
> kernel
> is at 1698 -- your's will probably be different:
>
> crash> gdb list n_tty_read
> 1695 * This code must be sure never to sleep through a hangup.
> 1696 */
> 1697
> 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
> *file,
> 1699 unsigned char __user *buf, size_t nr)
> 1700 {
> 1701 unsigned char __user *b = buf;
> 1702 DECLARE_WAITQUEUE(wait, current);
> 1703 int c;
> 1704 int minimum, time;
> crash>
>
> Then get the line number of the next function in the file, which is
> n_tty_write():
>
> crash> gdb list n_tty_write
> 1918 * lock themselves)
> 1919 */
> 1920
> 1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file
> *file,
> 1922 const unsigned char *buf, size_t nr)
> 1923 {
> 1924 const unsigned char *b = buf;
> 1925 DECLARE_WAITQUEUE(wait, current);
> 1926 int c;
> 1927 ssize_t retval = 0;
>
> And then dump the whole n_tty_read() function (plus some extra stuff):
>
> crash> gdb list 1698,1920
> 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file
> *file,
> 1699 unsigned char __user *buf, size_t nr)
> 1700 {
> 1701 unsigned char __user *b = buf;
> 1702 DECLARE_WAITQUEUE(wait, current);
> 1703 int c;
> 1704 int minimum, time;
> 1705 ssize_t retval = 0;
> 1706 ssize_t size;
> 1707 long timeout;
> 1708 unsigned long flags;
> 1709 int packet;
> 1710
> 1711 do_it_again:
> 1712
> 1713 BUG_ON(!tty->read_buf);
> 1714
> 1715 c = job_control(tty, file);
> 1716 if (c < 0)
> 1717 return c;
> 1718
> 1719 minimum = time = 0;
> 1720 timeout = MAX_SCHEDULE_TIMEOUT;
> 1721 if (!tty->icanon) {
> 1722 time = (HZ / 10) * TIME_CHAR(tty);
> 1723 minimum = MIN_CHAR(tty);
> ...
>
> And lastly, since the crash occurred at
>
> IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818
>
> Do this:
>
> crash> dis -rl n_tty_read+0x58c
> ...
>
> And then post all of that data.
>
> Dave
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
> Information transmitted by this e-mail is proprietary to MphasiS, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and
> may contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the
> intended recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster at mphasis.com and delete this mail from your records.
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility at redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at mailmaster at mphasis.com and delete this mail from your records.
More information about the Crash-utility
mailing list