kernel: INFO: task xxx:yyy blocked for more than zzz seconds.

Bob Wickline wick at bobwickline.com
Fri Jun 3 19:59:20 UTC 2011 

I have an HP DL385 running RHEL5.5 that's close to going into production.  However, we've experienced a couple of lockups in the past couple weeks that has management concerned.  I've been doing some research but I haven't found any answer to the issue.

I did see something about a possible issue with the cciss driver so I decided to run bonnie++ against one of the internal LUNs and sure enough I can duplocate the issue within minutes.

What is going on here?  Is there a fix for this issue?  Is anyone working on a solution???


>Jun  2 12:59:46 hidden kernel: INFO: task kswapd0:563 blocked for more
>than
>120 seconds.
>Jun  2 12:59:46 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 12:59:46 hidden kernel: kswapd0       D ffff810021004420     0  
>563
>185           564   560 (L-TLB)
>Jun  2 12:59:46 hidden kernel:  ffff81042efd9b00 0000000000000046
>0000000000000002 0000000000000010
>Jun  2 12:59:46 hidden kernel:  0000000000008f3f 000000000000000a
>ffff81042f5950c0 ffffffff80311b60
>Jun  2 12:59:46 hidden kernel:  0002c883e50c78ed 00000000002f2877
>ffff81042f5952a8 000000000000003d
>Jun  2 12:59:46 hidden kernel: Call Trace:
>Jun  2 12:59:46 hidden kernel:  [<ffffffff88036d8a>]
>:jbd:log_wait_commit+0xa3/0xf5
>Jun  2 12:59:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 12:59:46 hidden kernel:  [<ffffffff8002781a>]
>try_to_free_buffers+0x60/0xb9
>Jun  2 12:59:46 hidden kernel:  [<ffffffff880332da>]
>:jbd:journal_try_to_free_buffers+0x19d/0x1c0
>Jun  2 12:59:46 hidden kernel:  [<ffffffff800cd369>]
>shrink_inactive_list+0x511/0x8d8
>Jun  2 12:59:46 hidden kernel:  [<ffffffff8004817a>]
>__pagevec_release+0x19/0x22
>Jun  2 12:59:48 hidden kernel:  [<ffffffff800ccd37>]
>shrink_active_list+0x4b4/0x4c4
>Jun  2 12:59:49 hidden kernel:  [<ffffffff8001327d>]
>shrink_zone+0x127/0x18d
>Jun  2 12:59:50 hidden kernel:  [<ffffffff80057e54>] kswapd+0x33d/0x495
>Jun  2 12:59:50 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 12:59:51 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 12:59:51 hidden kernel:  [<ffffffff80057b17>] kswapd+0x0/0x495
>Jun  2 12:59:51 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 12:59:51 hidden kernel:  [<ffffffff80032afc>] kthread+0xfe/0x132
>Jun  2 12:59:52 hidden kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>Jun  2 12:59:53 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 12:59:54 hidden kernel:  [<ffffffff800329fe>] kthread+0x0/0x132
>Jun  2 12:59:54 hidden kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>Jun  2 12:59:55 hidden kernel: 
>Jun  2 12:59:56 hidden kernel: INFO: task kjournald:6071 blocked for
>more
>than 120 seconds.
>Jun  2 12:59:56 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 12:59:57 hidden kernel: kjournald     D ffff81002100caa0     0 
>6071
>185          6077  5264 (L-TLB)
>Jun  2 12:59:58 hidden kernel:  ffff810427f4bcf0 0000000000000046
>000000002dd9c000 ffff81042ff15950
>Jun  2 12:59:58 hidden kernel:  00000001ffffffff 0000000000000009
>ffff810420322080 ffff81042ff18100
>Jun  2 12:59:59 hidden kernel:  0002c889276dcdab 00000000071ac130
>ffff810420322268 0000000100000086
>Jun  2 13:00:00 hidden kernel: Call Trace:
>Jun  2 13:00:02 hidden kernel:  [<ffffffff8006ec4e>]
>do_gettimeofday+0x40/0x90
>Jun  2 13:00:02 hidden kernel:  [<ffffffff800155a2>]
>sync_buffer+0x0/0x3f
>Jun  2 13:00:03 hidden kernel:  [<ffffffff800637ca>]
>io_schedule+0x3f/0x67
>Jun  2 13:00:04 hidden kernel:  [<ffffffff800155dd>]
>sync_buffer+0x3b/0x3f
>Jun  2 13:00:05 hidden kernel:  [<ffffffff800639f6>]
>__wait_on_bit+0x40/0x6e
>Jun  2 13:00:06 hidden kernel:  [<ffffffff800155a2>]
>sync_buffer+0x0/0x3f
>Jun  2 13:00:06 hidden kernel:  [<ffffffff80063a90>]
>out_of_line_wait_on_bit+0x6c/0x78
>Jun  2 13:00:07 hidden kernel:  [<ffffffff800a2921>]
>wake_bit_function+0x0/0x23
>Jun  2 13:00:08 hidden kernel:  [<ffffffff880339a5>]
>:jbd:journal_commit_transaction+0x543/0x1066
>Jun  2 13:00:10 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:00:12 hidden kernel:  [<ffffffff8004b3fb>]
>try_to_del_timer_sync+0x7f/0x88
>Jun  2 13:00:13 hidden kernel:  [<ffffffff880375d3>]
>:jbd:kjournald+0xc1/0x213
>Jun  2 13:00:13 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:00:13 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:00:15 hidden kernel:  [<ffffffff88037512>]
>:jbd:kjournald+0x0/0x213
>Jun  2 13:00:15 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:00:17 hidden kernel:  [<ffffffff80032afc>] kthread+0xfe/0x132
>Jun  2 13:00:18 hidden kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>Jun  2 13:00:19 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:00:19 hidden kernel:  [<ffffffff800329fe>] kthread+0x0/0x132
>Jun  2 13:00:20 hidden kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>Jun  2 13:00:22 hidden kernel: 
>Jun  2 13:01:46 hidden kernel: INFO: task kswapd0:563 blocked for more
>than
>120 seconds.
>Jun  2 13:01:46 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 13:01:46 hidden kernel: kswapd0       D ffff810021004420     0  
>563
>185           564   560 (L-TLB)
>Jun  2 13:01:46 hidden kernel:  ffff81042efd9b00 0000000000000046
>0000000000000002 0000000000000010
>Jun  2 13:01:46 hidden kernel:  0000000000008f3f 000000000000000a
>ffff81042f5950c0 ffffffff80311b60
>Jun  2 13:01:46 hidden kernel:  0002c883e50c78ed 00000000002f2877
>ffff81042f5952a8 000000000000003d
>Jun  2 13:01:46 hidden kernel: Call Trace:
>Jun  2 13:01:46 hidden kernel:  [<ffffffff88036d8a>]
>:jbd:log_wait_commit+0xa3/0xf5
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8002781a>]
>try_to_free_buffers+0x60/0xb9
>Jun  2 13:01:46 hidden kernel:  [<ffffffff880332da>]
>:jbd:journal_try_to_free_buffers+0x19d/0x1c0
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800cd369>]
>shrink_inactive_list+0x511/0x8d8
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8004817a>]
>__pagevec_release+0x19/0x22
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800ccd37>]
>shrink_active_list+0x4b4/0x4c4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8001327d>]
>shrink_zone+0x127/0x18d
>Jun  2 13:01:46 hidden kernel:  [<ffffffff80057e54>] kswapd+0x33d/0x495
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff80057b17>] kswapd+0x0/0x495
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff80032afc>] kthread+0xfe/0x132
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800329fe>] kthread+0x0/0x132
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>Jun  2 13:01:46 hidden kernel: 
>Jun  2 13:01:46 hidden kernel: INFO: task kjournald:6071 blocked for
>more
>than 120 seconds.
>Jun  2 13:01:46 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 13:01:46 hidden kernel: kjournald     D ffff81002100caa0     0 
>6071
>185          6077  5264 (L-TLB)
>Jun  2 13:01:46 hidden kernel:  ffff810427f4bcf0 0000000000000046
>000000002dd9c000 ffff81042ff15950
>Jun  2 13:01:46 hidden kernel:  00000001ffffffff 0000000000000009
>ffff810420322080 ffff81042ff18100
>Jun  2 13:01:46 hidden kernel:  0002c889276dcdab 00000000071ac130
>ffff810420322268 0000000100000086
>Jun  2 13:01:46 hidden kernel: Call Trace:
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8006ec4e>]
>do_gettimeofday+0x40/0x90
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800155a2>]
>sync_buffer+0x0/0x3f
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800637ca>]
>io_schedule+0x3f/0x67
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800155dd>]
>sync_buffer+0x3b/0x3f
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800639f6>]
>__wait_on_bit+0x40/0x6e
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800155a2>]
>sync_buffer+0x0/0x3f
>Jun  2 13:01:46 hidden kernel:  [<ffffffff80063a90>]
>out_of_line_wait_on_bit+0x6c/0x78
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a2921>]
>wake_bit_function+0x0/0x23
>Jun  2 13:01:46 hidden kernel:  [<ffffffff880339a5>]
>:jbd:journal_commit_transaction+0x543/0x1066
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8004b3fb>]
>try_to_del_timer_sync+0x7f/0x88
>Jun  2 13:01:46 hidden kernel:  [<ffffffff880375d3>]
>:jbd:kjournald+0xc1/0x213
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff88037512>]
>:jbd:kjournald+0x0/0x213
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff80032afc>] kthread+0xfe/0x132
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:01:46 hidden kernel:  [<ffffffff800329fe>] kthread+0x0/0x132
>Jun  2 13:01:46 hidden kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>Jun  2 13:01:46 hidden kernel: 
>Jun  2 13:03:46 hidden kernel: INFO: task kswapd0:563 blocked for more
>than
>120 seconds.
>Jun  2 13:03:46 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 13:03:46 hidden kernel: kswapd0       D ffff810021004420     0  
>563
>185           564   560 (L-TLB)
>Jun  2 13:03:46 hidden kernel:  ffff81042efd9b00 0000000000000046
>0000000000000002 0000000000000010
>Jun  2 13:03:46 hidden kernel:  0000000000008f3f 000000000000000a
>ffff81042f5950c0 ffffffff80311b60
>Jun  2 13:03:46 hidden kernel:  0002c883e50c78ed 00000000002f2877
>ffff81042f5952a8 000000000000003d
>Jun  2 13:03:46 hidden kernel: Call Trace:
>Jun  2 13:03:46 hidden kernel:  [<ffffffff88036d8a>]
>:jbd:log_wait_commit+0xa3/0xf5
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8002781a>]
>try_to_free_buffers+0x60/0xb9
>Jun  2 13:03:46 hidden kernel:  [<ffffffff880332da>]
>:jbd:journal_try_to_free_buffers+0x19d/0x1c0
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800cd369>]
>shrink_inactive_list+0x511/0x8d8
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8004817a>]
>__pagevec_release+0x19/0x22
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800ccd37>]
>shrink_active_list+0x4b4/0x4c4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8001327d>]
>shrink_zone+0x127/0x18d
>Jun  2 13:03:46 hidden kernel:  [<ffffffff80057e54>] kswapd+0x33d/0x495
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff80057b17>] kswapd+0x0/0x495
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff80032afc>] kthread+0xfe/0x132
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800329fe>] kthread+0x0/0x132
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>Jun  2 13:03:46 hidden kernel: 
>Jun  2 13:03:46 hidden kernel: INFO: task kjournald:6071 blocked for
>more
>than 120 seconds.
>Jun  2 13:03:46 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 13:03:46 hidden kernel: kjournald     D ffff81002100caa0     0 
>6071
>185          6077  5264 (L-TLB)
>Jun  2 13:03:46 hidden kernel:  ffff810427f4bcf0 0000000000000046
>000000002dd9c000 ffff81042ff15950
>Jun  2 13:03:46 hidden kernel:  00000001ffffffff 0000000000000009
>ffff810420322080 ffff81042ff18100
>Jun  2 13:03:46 hidden kernel:  0002c889276dcdab 00000000071ac130
>ffff810420322268 0000000100000086
>Jun  2 13:03:46 hidden kernel: Call Trace:
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8006ec4e>]
>do_gettimeofday+0x40/0x90
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800155a2>]
>sync_buffer+0x0/0x3f
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800637ca>]
>io_schedule+0x3f/0x67
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800155dd>]
>sync_buffer+0x3b/0x3f
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800639f6>]
>__wait_on_bit+0x40/0x6e
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800155a2>]
>sync_buffer+0x0/0x3f
>Jun  2 13:03:46 hidden kernel:  [<ffffffff80063a90>]
>out_of_line_wait_on_bit+0x6c/0x78
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a2921>]
>wake_bit_function+0x0/0x23
>Jun  2 13:03:46 hidden kernel:  [<ffffffff880339a5>]
>:jbd:journal_commit_transaction+0x543/0x1066
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8004b3fb>]
>try_to_del_timer_sync+0x7f/0x88
>Jun  2 13:03:46 hidden kernel:  [<ffffffff880375d3>]
>:jbd:kjournald+0xc1/0x213
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff88037512>]
>:jbd:kjournald+0x0/0x213
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff80032afc>] kthread+0xfe/0x132
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a26db>]
>keventd_create_kthread+0x0/0xc4
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800329fe>] kthread+0x0/0x132
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
>Jun  2 13:03:46 hidden kernel: 
>Jun  2 13:03:46 hidden kernel: INFO: task oracle:20274 blocked for more
>than
>120 seconds.
>Jun  2 13:03:46 hidden kernel: "echo 0 >
>/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Jun  2 13:03:46 hidden kernel: oracle        D ffff810021025e20     0
>20274
>1         20276 20272 (NOTLB)
>Jun  2 13:03:46 hidden kernel:  ffff810270fedcf8 0000000000000086
>ffff810270fedce0 ffff810270fedca8
>Jun  2 13:03:46 hidden kernel:  0000000000000000 0000000000000009
>ffff81037f726820 ffff81042fe2a080
>Jun  2 13:03:46 hidden kernel:  0002c8c7206b57d6 000000000003d4c1
>ffff81037f726a08 0000000470fedd18
>Jun  2 13:03:46 hidden kernel: Call Trace:
>Jun  2 13:03:46 hidden kernel:  [<ffffffff88032002>]
>:jbd:start_this_handle+0x2e5/0x36c
>Jun  2 13:03:46 hidden kernel:  [<ffffffff800a28f3>]
>autoremove_wake_function+0x0/0x2e
>Jun  2 13:03:46 hidden kernel:  [<ffffffff88032152>]
>:jbd:journal_start+0xc9/0x100
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8805389c>]
>:ext3:ext3_create+0x49/0x102
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8003a86a>]
>vfs_create+0xe6/0x158
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8001b2d5>]
>open_namei+0x19d/0x712
>Jun  2 13:03:46 hidden kernel:  [<ffffffff80027660>]
>do_filp_open+0x1c/0x38
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8001a061>]
>do_sys_open+0x44/0xbe
>Jun  2 13:03:46 hidden kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
>Jun  2 13:03:46 hidden kernel: 

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

More information about the redhat-list mailing list