[Linux-cluster] Problem with more than one partition

[aleava at libero.it]

> aleava at libero.it wrote:
> > Hi Bob,
> > thanks for your help, but I have installed RHEL4 Update 4 AS, and the cluster doesn't work (the cluster.conf and the commands are the same).In the same manner, one node mount the shared partitions (4) without problems, but when the second node mounts the first partition, the mount command hangs.
> > P.S.: if a node mounts only one partition, the second node mounts the same partition without problems. We have problems with a number of partition =>2.
> > Any ideas?
> >
> > (At the moment, our solution is creating only a big shared partition with 4 directories, using quota...)
> >
> >
> > Regards,
> > Ale 
> >   
> Hi Ale,
> 
> This is odd.  I'm assuming that you rebooted all the cluster nodes after you
> upgraded to U4, or at least brought down the whole cluster first.
> Here are some things you can do to help us solve the problem:
> 
> (1) Please post the output from:
> 
> gfs_tool sb /dev/your_vg/lvol0 all
> 
> for both of the file systems you're trying to mount, substituting your 
> vg and lv or
> where your gfs file systems are located.
> 
> (2) Check for any unusual error messages posted in /var/log/messages 
> right after
> you recreate the hang.
> 
> (3)  Check for any unusual dmesg messages on both systems right after you
> recreate the hang.
> 
> (4) Do this command and post the output, right after the hang:
> cman_tool services
> 
> (5) Are you able to ctrl-c break out of the mount that hangs or do you 
> have to reboot?
> 
> (6) If you can't break out of mount, on the system that has its mount 
> hung, do this:
> echo "t" > /proc/sysrq-trigger
> and post the output so we can see where it's hanging.
> 
> We'll get to the bottom of this mystery!  Unfortunately, I will be out 
> of the office next
> week, along with most of the other Red Hat cluster developers.  So if I 
> don't
> hear back today or tomorrow, I won't be able to respond until the 2nd of 
> January.
> 
> Regards,
> 
> Bob Peterson
> Red Hat Cluster Suite
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 

Hi Bob, 
here are the information you requested me.
Whishing you a merry Christmas and a happy 2007 I post them 
to you waiting for a answer as soon as you can.
Regards
Ale

(1)

gfs_tool sb /dev/SDB/OPERATIONAL all

  mh_magic = 0x01161970
  mh_type = 1
  mh_generation = 0
  mh_format = 100
  mh_incarn = 0
  sb_fs_format = 1309
  sb_multihost_format = 1401
  sb_flags = 0
  sb_bsize = 4096
  sb_bsize_shift = 12
  sb_seg_size = 16
  no_formal_ino = 22
  no_addr = 22
  no_formal_ino = 23
  no_addr = 23
  no_formal_ino = 26
  no_addr = 26
  sb_lockproto = lock_dlm
  sb_locktable = alpha_cluster:OPERATIONAL
  no_formal_ino = 24
  no_addr = 24
  no_formal_ino = 25
  no_addr = 25
  sb_reserved =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


gfs_tool sb /dev/SDC/TRAINING all

  mh_magic = 0x01161970
  mh_type = 1
  mh_generation = 0
  mh_format = 100
  mh_incarn = 0
  sb_fs_format = 1309
  sb_multihost_format = 1401
  sb_flags = 0
  sb_bsize = 4096
  sb_bsize_shift = 12
  sb_seg_size = 16
  no_formal_ino = 22
  no_addr = 22
  no_formal_ino = 23
  no_addr = 23
  no_formal_ino = 26
  no_addr = 26
  sb_lockproto = lock_dlm
  sb_locktable = alpha_cluster:TRAINING
  no_formal_ino = 24
  no_addr = 24
  no_formal_ino = 25
  no_addr = 25
  sb_reserved =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


(2)

Messages posted in /var/log/messages of the first server that mounted the partitions (recorder):

Dec 22 09:54:15 recorder kernel: dlm: TRAINING: unexpected rcom msgid 4/c2386650 nodeid=1


Messages posted in /var/log/messages of the second server that mounted the partitions already mounted by recorder (data_server):

Dec 22 09:47:22 data_server kernel: Lock_Harness <CVS> (built Dec 21 2006 17:02:03) installed
Dec 22 09:47:22 data_server kernel: GFS <CVS> (built Dec 21 2006 17:02:40) installed
Dec 22 09:47:22 data_server kernel: GFS: Trying to join cluster "lock_dlm", "alpha_cluster:OPERATIONAL"
Dec 22 09:47:22 data_server kernel: Lock_DLM (built Dec 21 2006 17:02:07) installed


(3)

Messages posted in dmesg of the first server that mounted the partitions (recorder):

dlm: TRAINING: unexpected rcom msgid 4/c2386650 nodeid=1


Messages posted in dmesg of the second server that mounted the partitions already mounted by recorder (data_server):

Lock_Harness <CVS> (built Dec 21 2006 17:02:03) installed
GFS <CVS> (built Dec 21 2006 17:02:40) installed
GFS: Trying to join cluster "lock_dlm", "alpha_cluster:OPERATIONAL"
Lock_DLM (built Dec 21 2006 17:02:07) installed


(4)

cman_tool services command output from recorder:

Service          Name                              GID LID State     Code
Fence Domain:    "default"                           3   2 run       -
[2 1]

DLM Lock Space:  "OPERATIONAL"                       3   4 update    U-4,1,1
[2 1]

DLM Lock Space:  "TRAINING"                          3   6 run       -
[2]

GFS Mount Group: "OPERATIONAL"                       3   5 run       -
[2]

GFS Mount Group: "TRAINING"                          3   7 run       -
[2]

User:            "usrm::manager"                     2   3 run       -
[1 2]


cman_tool services command output from data_server:

Service          Name                              GID LID State     Code
Fence Domain:    "default"                           3   2 run       -
[1 2]

DLM Lock Space:  "OPERATIONAL"                       3   4 join      S-8,40,2
[1 2]

User:            "usrm::manager"                     2   3 run       -
[1 2]


(5)

The recorder continues to work correctly with the two mounted partitions; while on data_server is not
possible to stop the mount command with CTRL-C, kill -9 <PID> nor killall mount. Data_server is not
hanged; only the terminal from which the mount command has been launched seems to do no answer to any
request.
This is the output of the command "ps -eaf | grep mount" that we have executed on data_server while the
mount is hanged:
root      5171  4391  0 09:47 pts/0    00:00:00 mount -t gfs /dev/SDB/OPERATIONAL /DRMU_Diskarray/OPERATIONAL
Now we kill (kill -9 <PID>) the bash from which the command has been executed and launch again the ps command; 
the following is the output:
root      5171     1  0 09:47 ?        00:00:00 mount -t gfs /dev/SDB/OPERATIONAL /DRMU_Diskarray/OPERATIONAL
The mount command remains as a "ghost" on the server that continues to work correctly except for the
mounting of gfs partitions. To stop the mount process we ONLY HAVE TO REBOOT.


(6)

The following is the output of the command "echo "t" > /proc/sysrq-trigger" executed on the server
in which the mount command is hanged:
It's really very long!!!!!

Dec 22 10:07:58 data_server kernel: SysRq : Show State
Dec 22 10:07:58 data_server kernel: 
Dec 22 10:07:58 data_server kernel:                                                sibling
Dec 22 10:07:58 data_server kernel:   task             PC      pid father child younger older
Dec 22 10:07:58 data_server kernel: init          S F7E05F0C   900     1      0     2               (NOTLB)
Dec 22 10:07:58 data_server kernel: f7e05ec4 00000082 c016f070 f7e05f0c 00000000 f4a4e000 c0167430 c032a380 
Dec 22 10:07:58 data_server kernel:        c032a380 00000246 c0143fa4 c201dde0 00000000 00000f82 78122caa 00000476 
Dec 22 10:07:58 data_server kernel:        c0322a80 f7e41630 f7e4179c 00000000 00000246 00467ac4 00467ac4 00000000 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c016f070>] dput+0x34/0x1a7
Dec 22 10:07:58 data_server kernel:  [<c0167430>] link_path_walk+0x94/0xbe
Dec 22 10:07:58 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:58 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:58 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:58 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:58 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:58 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:58 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:58 data_server kernel: migration/0   S 00000000  3740     2      1             3       (L-TLB)
Dec 22 10:07:58 data_server kernel: f7e0bfb4 00000046 00000005 00000000 0011c707 f5cce730 f5d42be0 00000000 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 0000027a c1f8590f 00000471 
Dec 22 10:07:58 data_server kernel:        f5cce730 f7e40b30 f7e40c9c 00000000 00000000 c201e724 c201dde0 f7e0b000 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011f6f5>] migration_thread+0x8f/0x133
Dec 22 10:07:58 data_server kernel:  [<c011f666>] migration_thread+0x0/0x133
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: ksoftirqd/0   S C02D25F9  3732     3      1             4     2 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7e0cfc4 00000046 f7e0cfcc c02d25f9 c23b2bf0 c23b2bf0 f7e41630 f7e41630 
Dec 22 10:07:58 data_server kernel:        00000001 00000000 c201e7c0 c201dde0 00000000 00000097 c6cdbd3d 00000475 
Dec 22 10:07:58 data_server kernel:        c0322a80 f7e405b0 f7e4071c 00000000 f7e0c000 f7e0c000 f7e05f80 00000000 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:58 data_server kernel:  [<c0126e45>] ksoftirqd+0x0/0xab
Dec 22 10:07:58 data_server kernel:  [<c0126e86>] ksoftirqd+0x41/0xab
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: migration/1   S FFFFFFFF  3732     4      1             5     3 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7e0dfb4 00000046 00000000 ffffffff 0011c702 f70911d0 f5cf1140 00000000 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 c20267c0 c2025de0 00000001 00000241 c0916f39 00000471 
Dec 22 10:07:58 data_server kernel:        f70911d0 f7e40030 f7e4019c 00000000 00000000 c2026724 c2025de0 f7e0d000 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011f6f5>] migration_thread+0x8f/0x133
Dec 22 10:07:58 data_server kernel:  [<c011f666>] migration_thread+0x0/0x133
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: ksoftirqd/1   S C02D25F9  3748     5      1             6     4 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7e56fc4 00000046 f7e56fcc c02d25f9 c23b2bf0 c23b2bf0 f7e40030 f7e41630 
Dec 22 10:07:58 data_server kernel:        00000001 00000000 c20267c0 c2025de0 00000001 00000080 b5227b6b 00000476 
Dec 22 10:07:58 data_server kernel:        f7e410b0 f7c03650 f7c037bc 00000000 f7e56000 f7e56000 f7e05f5c 00000001 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:58 data_server kernel:  [<c0126e45>] ksoftirqd+0x0/0xab
Dec 22 10:07:58 data_server kernel:  [<c0126e86>] ksoftirqd+0x41/0xab
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: events/0      S 00000246  2988     6      1     8       7     5 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7feff68 00000046 f7e91ea0 00000246 00000246 00000246 c201fa38 c0129e39 
Dec 22 10:07:58 data_server kernel:        00000000 f7c08030 00000003 c201dde0 00000000 00002d08 aca96f3b 00000476 
Dec 22 10:07:58 data_server kernel:        c0322a80 f7c030d0 f7c0323c f7feffa0 f7c08018 c201fa20 f7c08010 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0129e39>] __mod_timer+0x101/0x10b
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01482c0>] cache_reap+0x0/0x1a1
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: events/1      S F916F672  3424     7      1  2227      11     6 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7feef68 00000046 c01c2ff1 f916f672 00000246 00000246 c2027a38 c0129e39 
Dec 22 10:07:58 data_server kernel:        00000000 f7c080b0 00000003 c2025de0 00000001 00001dd7 c34a226d 00000476 
Dec 22 10:07:58 data_server kernel:        f7e410b0 f7c02b50 f7c02cbc f7feefa0 f7c08098 c2027a20 f7c08090 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c01c2ff1>] __delay+0x9/0xa
Dec 22 10:07:58 data_server kernel:  [<f916f672>] e1000_read_phy_reg_ex+0x33/0x92 [e1000]
Dec 22 10:07:58 data_server kernel:  [<c0129e39>] __mod_timer+0x101/0x10b
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01482c0>] cache_reap+0x0/0x1a1
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: khelper       S C013075E  3352     8      6             9       (L-TLB)
Dec 22 10:07:58 data_server kernel: f7fecf68 00000046 00000000 c013075e 00000000 f5ec88f0 00000000 00000000 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 000012c3 1f98dffc 00000357 
Dec 22 10:07:58 data_server kernel:        f5ec88f0 f7c025d0 f7c0273c f7fecfa0 f7c0c018 f4899c64 f7c0c010 00000202 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c013075e>] wait_for_helper+0x0/0x78
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01307d6>] __call_usermodehelper+0x0/0x41
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: kblockd/0     S F7C0B980  3568     9      6            10     8 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7c1bf68 00000046 c0129e39 f7c0b980 f7d6e1a0 f5cbd790 f7c0b980 f7d87808 
Dec 22 10:07:58 data_server kernel:        c022ba21 00000000 c201e7c0 c201dde0 00000000 0000978b 51f935b6 000001d9 
Dec 22 10:07:58 data_server kernel:        f5cbd790 f7c02050 f7c021bc f7c1bfa0 f7cbe018 f7d87914 f7cbe010 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0129e39>] __mod_timer+0x101/0x10b
Dec 22 10:07:58 data_server kernel:  [<c022ba21>] cfq_next_request+0x2e/0x35
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c02243b3>] blk_unplug_work+0x0/0x9
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: kblockd/1     S F7C0B980  3484    10      6            52     9 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7c1cf68 00000046 c0129e39 f7c0b980 f7d6e1a0 c2376650 f7c0b980 f7d87808 
Dec 22 10:07:58 data_server kernel:        c022ba21 00000000 c20267c0 c2025de0 00000001 00000413 be1d2988 00000471 
Dec 22 10:07:58 data_server kernel:        c2376650 f7cc1670 f7cc17dc f7c1cfa0 f7cbe098 f7d87914 f7cbe090 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0129e39>] __mod_timer+0x101/0x10b
Dec 22 10:07:58 data_server kernel:  [<c022ba21>] cfq_next_request+0x2e/0x35
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c02243b3>] blk_unplug_work+0x0/0x9
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: khubd         S 00000000  3716    11      1            54     7 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7eb1fa4 00000046 00000000 00000000 f704d190 f7e405b0 c0131505 00000001 
Dec 22 10:07:58 data_server kernel:        c0327e40 00000000 c0416120 c2025de0 00000001 00000b87 359f0396 00000009 
Dec 22 10:07:58 data_server kernel:        f7e410b0 f7cc10f0 f7cc125c c0349c58 c0120420 c0258d80 00000000 f7eb1fc0 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0131505>] attach_pid+0x1f/0xa1
Dec 22 10:07:58 data_server kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Dec 22 10:07:58 data_server kernel:  [<c0258d80>] hub_thread+0x0/0xe4
Dec 22 10:07:58 data_server kernel:  [<c0258e1f>] hub_thread+0x9f/0xe4
Dec 22 10:07:58 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:58 data_server kernel:  [<c0258d80>] hub_thread+0x0/0xe4
Dec 22 10:07:58 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: pdflush       S C011D2E4  3764    52      6            53    10 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7eb4f98 00000046 c03bfe80 c011d2e4 00000000 f7cc05f0 f7c03320 f7c030d0 
Dec 22 10:07:58 data_server kernel:        c201dde0 00000000 c0416120 c201dde0 00000000 000009db 0af5a415 00000000 
Dec 22 10:07:58 data_server kernel:        f7cc05f0 f7cc0b70 f7cc0cdc 00000297 c201de1c f7eb4000 f7eb4fb0 f7eb4fbc 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:58 data_server kernel:  [<c0145b04>] pdflush+0x0/0x1e
Dec 22 10:07:58 data_server kernel:  [<c0145a0a>] __pdflush+0x86/0x180
Dec 22 10:07:58 data_server kernel:  [<c0145b1e>] pdflush+0x1a/0x1e
Dec 22 10:07:58 data_server kernel:  [<c0145b04>] pdflush+0x0/0x1e
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: pdflush       S C0145B04  2620    53      6            55    52 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7cc6f98 00000046 000024fe c0145b04 c01452e3 0045e50e f7e41880 00000000 
Dec 22 10:07:58 data_server kernel:        00000000 f7cc6f4c 000003ff c2025de0 00000001 00001373 b1fed268 00000475 
Dec 22 10:07:58 data_server kernel:        f7e410b0 f7cc05f0 f7cc075c 0000003c 00000000 f7cc6000 f7cc6fb0 f7cc6fbc 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0145b04>] pdflush+0x0/0x1e
Dec 22 10:07:58 data_server kernel:  [<c01452e3>] wb_kupdate+0xd8/0xde
Dec 22 10:07:58 data_server kernel:  [<c0145b04>] pdflush+0x0/0x1e
Dec 22 10:07:58 data_server kernel:  [<c0145a0a>] __pdflush+0x86/0x180
Dec 22 10:07:58 data_server kernel:  [<c0145b1e>] pdflush+0x1a/0x1e
Dec 22 10:07:58 data_server kernel:  [<c0145b04>] pdflush+0x0/0x1e
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: aio/0         S 00000000  3752    55      6            56    53 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7d26f68 00000046 0dde85ef 00000000 00000001 f7d57110 0af884f1 f7d26f40 
Dec 22 10:07:58 data_server kernel:        c011cdb9 00000000 c0416120 c201dde0 00000000 00000d87 0af8ec13 00000000 
Dec 22 10:07:58 data_server kernel:        f7d57110 f7d57690 f7d577fc f7d26fa0 f7d54018 f7d26000 f7d26f80 f7d26f94 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011cdb9>] activate_task+0x88/0x95
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: aio/1         S 00000000  3752    56      6           381    55 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7d28f68 00000046 0ddf10a6 00000000 f7d28000 00000001 f7d28f94 c0116cab 
Dec 22 10:07:58 data_server kernel:        00000286 f7d28f80 c02d5122 c2025de0 00000001 00002325 0af931c3 00000000 
Dec 22 10:07:58 data_server kernel:        f7e410b0 f7d57110 f7d5727c f7d28fa0 f7d54098 f7d28000 f7d28f80 f7d28f94 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0116cab>] smp_call_function_interrupt+0x3a/0x77
Dec 22 10:07:58 data_server kernel:  [<c02d5122>] call_function_interrupt+0x1a/0x20
Dec 22 10:07:58 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:58 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:58 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: kswapd0       S F7CD7F50  3692    54      1           200    11 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7cd7f94 00000046 00000286 f7cd7f50 c011f5b3 00000000 f7cc0070 00000003 
Dec 22 10:07:58 data_server kernel:        00000286 c012acff c0327e40 c201dde0 00000000 000025b5 0afcdd40 00000000 
Dec 22 10:07:58 data_server kernel:        c0322a80 f7cc0070 f7cc01dc c032cbd0 c0120420 f7cd7fc0 c032cbd0 f7cd7fb4 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011f5b3>] set_cpus_allowed+0x11a/0x125
Dec 22 10:07:58 data_server kernel:  [<c012acff>] free_uid+0x11/0x5a
Dec 22 10:07:58 data_server kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Dec 22 10:07:58 data_server kernel:  [<c014ac78>] kswapd+0xb8/0xcc
Dec 22 10:07:58 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:58 data_server kernel:  [<c02d4622>] ret_from_fork+0x6/0x14
Dec 22 10:07:58 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:58 data_server kernel:  [<c014abc0>] kswapd+0x0/0xcc
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: kseriod       S 00000000  3792   200      1           321    54 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7d48fa0 00000046 00000000 00000000 00000000 f7e405b0 c0131505 00000001 
Dec 22 10:07:58 data_server kernel:        c0327e40 00000000 c0416120 c201dde0 00000000 00002b80 17786f4e 00000000 
Dec 22 10:07:58 data_server kernel:        f7e405b0 f7d56b90 f7d56cfc c034145c c0120420 00000000 00000000 f7d48fbc 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c0131505>] attach_pid+0x1f/0xa1
Dec 22 10:07:58 data_server kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Dec 22 10:07:58 data_server kernel:  [<c0217c65>] serio_thread+0xbe/0x131
Dec 22 10:07:58 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:58 data_server kernel:  [<c02d4622>] ret_from_fork+0x6/0x14
Dec 22 10:07:58 data_server kernel:  [<c0217ba7>] serio_thread+0x0/0x131
Dec 22 10:07:58 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:58 data_server kernel:  [<c0217ba7>] serio_thread+0x0/0x131
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: scsi_eh_0     S C03BFE80  3676   321      1           327   200 (L-TLB)
Dec 22 10:07:58 data_server kernel: c2281f7c 00000046 c22cf130 c03bfe80 c03bfe80 00000096 c03bfe80 c011d2e4 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 f7e41300 c2025de0 00000001 00001163 339826c3 00000001 
Dec 22 10:07:58 data_server kernel:        f7e410b0 c22ce630 c22ce79c 00000000 00000003 c2281fd0 c2281fc8 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:58 data_server kernel:  [<c02d1ccf>] __down_interruptible+0xbc/0x115
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c02d1d3b>] __down_failed_interruptible+0x7/0xc
Dec 22 10:07:58 data_server kernel:  [<f8839116>] .text.lock.scsi_error+0x2d/0x33 [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<f8838dfa>] scsi_error_handler+0x0/0x15a [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: scsi_eh_1     S C03BFE80  3676   327      1           333   321 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7dd7f7c 00000046 c22cf130 c03bfe80 c03bfe80 00000096 c03bfe80 c011d2e4 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 f7e41300 c2025de0 00000001 00000ee3 eeacc924 00000002 
Dec 22 10:07:58 data_server kernel:        f7e410b0 c23776d0 c237783c 00000000 00000003 f7dd7fd0 f7dd7fc8 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:58 data_server kernel:  [<c02d1ccf>] __down_interruptible+0xbc/0x115
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c02d1d3b>] __down_failed_interruptible+0x7/0xc
Dec 22 10:07:58 data_server kernel:  [<f8839116>] .text.lock.scsi_error+0x2d/0x33 [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<f8838dfa>] scsi_error_handler+0x0/0x15a [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: scsi_eh_2     S C03BFE80  3724   333      1           334   327 (L-TLB)
Dec 22 10:07:58 data_server kernel: c234bf7c 00000046 c22cf130 c03bfe80 00000001 c22cf130 c03bfe80 c011d2e4 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 000036bf aa8ff06a 00000004 
Dec 22 10:07:58 data_server kernel:        c22cf130 c2377150 c23772bc 00000000 00000003 c234bfd0 c234bfc8 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:58 data_server kernel:  [<c02d1ccf>] __down_interruptible+0xbc/0x115
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c02d1d3b>] __down_failed_interruptible+0x7/0xc
Dec 22 10:07:58 data_server kernel:  [<f8839116>] .text.lock.scsi_error+0x2d/0x33 [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<f8838dfa>] scsi_error_handler+0x0/0x15a [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: aacraid       S 00000000  3492   334      1           344   333 (L-TLB)
Dec 22 10:07:58 data_server kernel: c2367f10 00000046 c011e83e 00000000 00000246 c23a1e3c c02d1c04 00000001 
Dec 22 10:07:58 data_server kernel:        c2376bd0 c011e794 00100100 c201dde0 00000000 00000620 b50b01f5 0000034a 
Dec 22 10:07:58 data_server kernel:        c0322a80 c2376bd0 c2376d3c 00000000 00000246 004e358a 004e358a f7c0bbd8 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011e83e>] __wake_up_locked+0x11/0x13
Dec 22 10:07:58 data_server kernel:  [<c02d1c04>] __down+0xcc/0xdb
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:58 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:58 data_server kernel:  [<f88b48aa>] aac_command_thread+0x608/0x6bc [aacraid]
Dec 22 10:07:58 data_server kernel:  [<f88b4909>] aac_command_thread+0x667/0x6bc [aacraid]
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c02d4622>] ret_from_fork+0x6/0x14
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<f88b42a2>] aac_command_thread+0x0/0x6bc [aacraid]
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:58 data_server kernel: scsi_eh_3     S C03BFE80  3724   344      1           345   334 (L-TLB)
Dec 22 10:07:58 data_server kernel: f7ddbf7c 00000046 c22cf130 c03bfe80 00000001 c22cf130 c03bfe80 c011d2e4 
Dec 22 10:07:58 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 00003b8a 3b4a5024 00000006 
Dec 22 10:07:58 data_server kernel:        c22cf130 c22ce0b0 c22ce21c 00000000 00000003 f7ddbfd0 f7ddbfc8 00000246 
Dec 22 10:07:58 data_server kernel: Call Trace:
Dec 22 10:07:58 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:58 data_server kernel:  [<c02d1ccf>] __down_interruptible+0xbc/0x115
Dec 22 10:07:58 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:58 data_server kernel:  [<c02d1d3b>] __down_failed_interruptible+0x7/0xc
Dec 22 10:07:58 data_server kernel:  [<f8839116>] .text.lock.scsi_error+0x2d/0x33 [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<f8838dfa>] scsi_error_handler+0x0/0x15a [scsi_mod]
Dec 22 10:07:58 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: qla2300_3_dpc S C03BFE80  3744   345      1           393   344 (L-TLB)
Dec 22 10:07:59 data_server kernel: c2369f74 00000046 c22cf130 c03bfe80 00000001 c22cf130 c03bfe80 c011d2e4 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 0000051f 7c5d6c29 00000006 
Dec 22 10:07:59 data_server kernel:        c22cf130 c22cebb0 c22ced1c 00000000 00000003 c2369fc8 c2369fc0 00000246 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:59 data_server kernel:  [<c02d1ccf>] __down_interruptible+0xbc/0x115
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c02d1d3b>] __down_failed_interruptible+0x7/0xc
Dec 22 10:07:59 data_server kernel:  [<f888a1c0>] .text.lock.qla_os+0x5/0x21 [qla2xxx]
Dec 22 10:07:59 data_server kernel:  [<f8888509>] qla2x00_do_dpc+0x0/0x134 [qla2xxx]
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: kmirrord      S 00000000  3752   381      6          2261    56 (L-TLB)
Dec 22 10:07:59 data_server kernel: f7047f68 00000046 3a0c5222 00000000 00000001 c22cf130 81eabef7 f7047f40 
Dec 22 10:07:59 data_server kernel:        c011cdb9 00000000 c201e7c0 c201dde0 00000000 00000e13 81eae0b9 00000006 
Dec 22 10:07:59 data_server kernel:        c22cf130 f705e130 f705e29c f7047fa0 c23fa018 f7047000 f7047f80 f7047f94 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c011cdb9>] activate_task+0x88/0x95
Dec 22 10:07:59 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:59 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:59 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:59 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: kjournald     D F7C0B980  2356   393      1          2035   345 (L-TLB)
Dec 22 10:07:59 data_server kernel: f7dddd90 00000046 f7d87808 f7c0b980 f7d87808 c022ba21 00000000 f7d87808 
Dec 22 10:07:59 data_server kernel:        00000000 00000001 00000a55 d243a0ea 00000476 f7e410b0 c22cf130 c22cf29c 
Dec 22 10:07:59 data_server kernel:        c0224397 f7d83ae0 c2025de0 c0443900 f492695c f7dddd90 c02d2d62 f6829ee8 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c022ba21>] cfq_next_request+0x2e/0x35
Dec 22 10:07:59 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:59 data_server kernel:  [<c0120502>] autoremove_wake_function+0xd/0x2d
Dec 22 10:07:59 data_server kernel:  [<c02d2d62>] io_schedule+0x26/0x30
Dec 22 10:07:59 data_server kernel:  [<c02243b2>] blk_backing_dev_unplug+0xf/0x10
Dec 22 10:07:59 data_server kernel:  [<c011e7d6>] __wake_up_common+0x36/0x51
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<f88d21ed>] journal_commit_transaction+0xfb6/0xfc1 [jbd]
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c011de3f>] find_busiest_group+0xdd/0x295
Dec 22 10:07:59 data_server kernel:  [<c011cde3>] deactivate_task+0x1d/0x27
Dec 22 10:07:59 data_server kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Dec 22 10:07:59 data_server kernel:  [<f88d3f87>] kjournald+0x1a5/0x219 [jbd]
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c011d68a>] schedule_tail+0x31/0xae
Dec 22 10:07:59 data_server kernel:  [<f88d3ddc>] commit_timeout+0x0/0x5 [jbd]
Dec 22 10:07:59 data_server kernel:  [<f88d3de2>] kjournald+0x0/0x219 [jbd]
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: udevd         S 0804AE24  2980  2035      1          2308   393 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5c8dec4 00000082 c014dab0 0804ae24 bffc30dc 00000000 00000000 c032a400 
Dec 22 10:07:59 data_server kernel:        c032a380 00000246 c0143fa4 c2025de0 00000001 00016ecd 54f1b208 00000358 
Dec 22 10:07:59 data_server kernel:        f7e410b0 f705f730 f705f89c 00000001 00000000 f5cb8ec0 7fffffff 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c014dab0>] do_wp_page+0x358/0x371
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c027dce9>] datagram_poll+0x25/0xcc
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c0125c09>] sys_waitpid+0x13/0x17
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: kauditd       S C2025DE0  3764  2227      7          4480       (L-TLB)
Dec 22 10:07:59 data_server kernel: f5d59f80 00000046 c011cdb9 c2025de0 f7c02b50 f5e4b330 00000001 00000096 
Dec 22 10:07:59 data_server kernel:        c03bfe80 00000000 c20267c0 c2025de0 00000001 0000079f 707cff6d 00000009 
Dec 22 10:07:59 data_server kernel:        f5e4b330 f5d66770 f5d668dc f5d59fa8 c0328d50 f5d59000 f705dc9c f5d59fa8 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c011cdb9>] activate_task+0x88/0x95
Dec 22 10:07:59 data_server kernel:  [<c013b59c>] kauditd_thread+0x0/0x16a
Dec 22 10:07:59 data_server kernel:  [<c013b6ed>] kauditd_thread+0x151/0x16a
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:59 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: kmpathd/0     S 00000000  3752  2261      6          2262   381 (L-TLB)
Dec 22 10:07:59 data_server kernel: f5da6f68 00000046 8d1a823e 00000000 00000001 f5df3850 b81d7806 f5da6f40 
Dec 22 10:07:59 data_server kernel:        c011cdb9 00000000 c201e7c0 c201dde0 00000000 00000b7d b81dc7d2 0000000a 
Dec 22 10:07:59 data_server kernel:        f5df3850 f5daad10 f5daae7c f5da6fa0 f5aac018 f5da6000 f5da6f80 f5da6f94 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c011cdb9>] activate_task+0x88/0x95
Dec 22 10:07:59 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:59 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:59 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:59 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: kmpathd/1     S 00000000  3656  2262      6          5184  2261 (L-TLB)
Dec 22 10:07:59 data_server kernel: f7040f68 00000046 8d1aef24 00000000 00000001 00000001 f7040000 c0116cab 
Dec 22 10:07:59 data_server kernel:        00000286 f7040f80 c02d5122 c2025de0 00000001 0000418c b81e2b85 0000000a 
Dec 22 10:07:59 data_server kernel:        f7e410b0 f5df3850 f5df39bc f7040fa0 f5aac098 f7040000 f7040f80 f7040f94 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0116cab>] smp_call_function_interrupt+0x3a/0x77
Dec 22 10:07:59 data_server kernel:  [<c02d5122>] call_function_interrupt+0x1a/0x20
Dec 22 10:07:59 data_server kernel:  [<c0130bc4>] worker_thread+0xc9/0x1d5
Dec 22 10:07:59 data_server kernel:  [<c02d25f9>] schedule+0x86d/0x8db
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c0130afb>] worker_thread+0x0/0x1d5
Dec 22 10:07:59 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:07:59 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: kjournald     S C0416A1F  3608  2308      1          3162  2035 (L-TLB)
Dec 22 10:07:59 data_server kernel: f5d3cf64 00000046 00000002 c0416a1f 00003ee1 f5dab290 c01227a7 00003ee1 
Dec 22 10:07:59 data_server kernel:        00000246 00000000 c201e7c0 c201dde0 00000000 0000470d eacd94da 0000000a 
Dec 22 10:07:59 data_server kernel:        f5dab290 f5e38290 f5e383fc f7e19284 c0120420 f7e19214 f7e19200 00000001 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c01227a7>] call_console_drivers+0xb6/0xd8
Dec 22 10:07:59 data_server kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Dec 22 10:07:59 data_server kernel:  [<f88d3f87>] kjournald+0x1a5/0x219 [jbd]
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c011d68a>] schedule_tail+0x31/0xae
Dec 22 10:07:59 data_server kernel:  [<f88d3ddc>] commit_timeout+0x0/0x5 [jbd]
Dec 22 10:07:59 data_server kernel:  [<f88d3de2>] kjournald+0x0/0x219 [jbd]
Dec 22 10:07:59 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:07:59 data_server kernel: syslogd       D C01490FD  2488  3162      1          3166  2308 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5d8becc 00000086 00000001 c01490fd 0000000e f5e6e850 c1f7c380 f5d8bf5c 
Dec 22 10:07:59 data_server kernel:        c0178a5b 00000000 c20267c0 c2025de0 00000001 00001281 d5bbff22 00000476 
Dec 22 10:07:59 data_server kernel:        f5e6e850 c2376650 c23767bc 00000000 00000246 00466d61 00466d61 f49e9294 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c01490fd>] pagevec_lookup_tag+0x1b/0x22
Dec 22 10:07:59 data_server kernel:  [<c0120420>] prepare_to_wait+0x12/0x4c
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<f88d06cd>] journal_stop+0x1bf/0x1ed [jbd]
Dec 22 10:07:59 data_server kernel:  [<f88d0717>] journal_force_commit+0x1c/0x21 [jbd]
Dec 22 10:07:59 data_server kernel:  [<f88ea848>] ext3_force_commit+0x1f/0x20 [ext3]
Dec 22 10:07:59 data_server kernel:  [<c0177266>] write_inode+0x30/0x37
Dec 22 10:07:59 data_server kernel:  [<c01772dd>] __sync_single_inode+0x70/0x1c1
Dec 22 10:07:59 data_server kernel:  [<c0177a6e>] sync_inode+0x2f/0x40
Dec 22 10:07:59 data_server kernel:  [<f88e0e96>] ext3_sync_file+0xa6/0xb0 [ext3]
Dec 22 10:07:59 data_server kernel:  [<c014015d>] filemap_fdatawrite+0xf/0x13
Dec 22 10:07:59 data_server kernel:  [<c015c46b>] sys_fsync+0x8c/0xc8
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: klogd         S C011CD26  2720  3166      1          3177  3162 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5d90d9c 00000086 f5d90d6c c011cd26 c2376650 c2376650 0011db18 b1078dee 
Dec 22 10:07:59 data_server kernel:        f5eece00 00000000 c20267c0 c2025de0 00000001 00000732 d6a0e68d 00000476 
Dec 22 10:07:59 data_server kernel:        f7e410b0 f5e6e850 f5e6e9bc c03bfe80 c011d2e4 00000001 7fffffff f5d90df4 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c011cd26>] recalc_task_prio+0x128/0x133
Dec 22 10:07:59 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012046c>] prepare_to_wait_exclusive+0x12/0x4b
Dec 22 10:07:59 data_server kernel:  [<c02cda3e>] unix_wait_for_peer+0x93/0xae
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c027d004>] memcpy_fromiovec+0x65/0xb0
Dec 22 10:07:59 data_server kernel:  [<c02ce43e>] unix_dgram_sendmsg+0x373/0x45d
Dec 22 10:07:59 data_server kernel:  [<c027815b>] sock_aio_write+0x106/0x113
Dec 22 10:07:59 data_server kernel:  [<c015afd4>] do_sync_write+0x97/0xc9
Dec 22 10:07:59 data_server kernel:  [<c012242d>] do_syslog+0x1d5/0x3bf
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server last message repeated 2 times
Dec 22 10:07:59 data_server kernel:  [<c015b0cc>] vfs_write+0xc6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b186>] sys_write+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: irqbalance    S 00000000  2720  3177      1          3196  3166 (NOTLB)
Dec 22 10:07:59 data_server kernel: c23f6f5c 00000082 00000001 00000000 c1f82fa0 00000000 c23f6f74 b7ff8000 
Dec 22 10:07:59 data_server kernel:        f5dfd90c b7ff9000 c014cac5 c201dde0 00000000 00015926 378f5666 00000476 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5ec8370 f5ec84dc 00000000 00000246 00468a14 00468a14 00002713 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c014cac5>] unmap_vmas+0xf1/0x205
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c012a7f6>] sys_nanosleep+0x103/0x1a1
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel:  [<c02d007b>] packet_rcv+0x242/0x307
Dec 22 10:07:59 data_server kernel: portmap       S C032A380  2604  3196      1          3216  3177 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5dfbf28 00000082 00000000 c032a380 00000000 000000d0 c032cb80 c0144080 
Dec 22 10:07:59 data_server kernel:        00000001 00000000 f704c690 c201dde0 00000000 00005d4f 50e93899 0000000c 
Dec 22 10:07:59 data_server kernel:        c0322a80 f704c690 f704c7fc f5b8daa0 c029d709 00000000 7fffffff f5dfbfa0 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0144080>] __alloc_pages+0xb4/0x29d
Dec 22 10:07:59 data_server kernel:  [<c029d709>] tcp_poll+0x31/0x146
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c0278689>] sock_poll+0x12/0x14
Dec 22 10:07:59 data_server kernel:  [<c016bc1b>] do_pollfd+0x59/0x81
Dec 22 10:07:59 data_server kernel:  [<c016bcd1>] do_poll+0x8e/0xac
Dec 22 10:07:59 data_server kernel:  [<c016bebc>] sys_poll+0x1cd/0x281
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: rpc.statd     S F5D29E70  2420  3216      1          3247  3196 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5d29ec4 00000082 c01204f5 f5d29e70 c032a400 c2376650 00000246 c0143fa4 
Dec 22 10:07:59 data_server kernel:        c16a9220 00000000 c20267c0 c2025de0 00000001 00005fcf 51117c94 0000000c 
Dec 22 10:07:59 data_server kernel:        c2376650 f5e818f0 f5e81a5c 00000010 00000000 f5e08f60 7fffffff 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c01204f5>] autoremove_wake_function+0x0/0x2d
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c029d709>] tcp_poll+0x31/0x146
Dec 22 10:07:59 data_server kernel:  [<c027dce9>] datagram_poll+0x25/0xcc
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c0279871>] sys_socketcall+0x151/0x1fb
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: rpc.idmapd    S 00000000  3176  3247      1          3365  3216 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5ca4f28 00000082 f5e29010 00000000 c0322a80 f5d67270 c229c5e0 00000001 
Dec 22 10:07:59 data_server kernel:        00000000 00000001 c03ee000 c201dde0 00000000 00000ccc f713bcd0 00000475 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5dac7b0 f5dac91c 00000000 00000246 0046724e 0046724e 00000246 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c017ce6f>] ep_poll+0xd3/0x147
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c017c26b>] sys_epoll_wait+0x73/0x86
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: cupsd         S C0143FA4  2236  3365      1          3443  3247 (NOTLB)
Dec 22 10:07:59 data_server kernel: f7051ec4 00000086 00000246 c0143fa4 c16933a0 000000d0 00000000 c032a380 
Dec 22 10:07:59 data_server kernel:        00000000 000000d0 c032cb80 c201dde0 00000000 0005c739 dd31e285 0000046d 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5dad830 f5dad99c 00000000 00000246 004a6e8a 004a6e8a 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c01c2fc8>] atomic_dec_and_lock+0x20/0x40
Dec 22 10:07:59 data_server kernel:  [<c012614f>] sys_time+0xf/0x58
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: sshd          S C0143FA4  2720  3443      1  4389    3472  3365 (NOTLB)
Dec 22 10:07:59 data_server kernel: c23dfec4 00000086 00000246 c0143fa4 c1692e60 f5ea2e10 00000000 c032a380 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 000079d2 bdd4bb95 00000471 
Dec 22 10:07:59 data_server kernel:        f5ea2e10 f7d56610 f7d5677c 00000000 c23dff58 f5ddde20 7fffffff 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c029d709>] tcp_poll+0x31/0x146
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: xinetd        S 7FB05025  2504  3472      1          3482  3443 (NOTLB)
Dec 22 10:07:59 data_server kernel: f7c07ec4 00000082 f5dd1784 7fb05025 00000000 fffcf050 c014e50d c032a380 
Dec 22 10:07:59 data_server kernel:        c032a380 00000246 c0143fa4 c201dde0 00000000 0000de51 8b32796b 0000000d 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5cab770 f5cab8dc 00000001 00000000 f5cb3380 7fffffff 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c014e50d>] do_no_page+0x2da/0x2f9
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c029d709>] tcp_poll+0x31/0x146
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c0279871>] sys_socketcall+0x151/0x1fb
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: gpm           S 00000000  2720  3482      1          3513  3472 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4c3bec4 00000082 00000101 00000000 f5c33009 f7e97920 c22cc63c c032a380 
Dec 22 10:07:59 data_server kernel:        c032a380 00000246 c0143fa4 c201dde0 00000000 0001dd14 85fd6c07 0000000d 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5d661f0 f5d6635c 00000000 00000246 0522b874 0522b874 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c0126229>] sys_gettimeofday+0x53/0xac
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: htt           S 00000001  3360  3513      1  3514    3524  3482 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5482f28 00000086 00000000 00000001 f5d4fa14 f704cc10 f5d699c0 f5d699f0 
Dec 22 10:07:59 data_server kernel:        f5d4fa14 00000000 c201e7c0 c201dde0 00000000 00006295 b9c761a4 0000000d 
Dec 22 10:07:59 data_server kernel:        f704cc10 f7091750 f70918bc f5482f70 f7091858 f70917f4 fffffe00 f7091750 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0125985>] do_wait+0x26e/0x449
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c0125bf3>] sys_wait4+0x27/0x2a
Dec 22 10:07:59 data_server kernel:  [<c0125c09>] sys_waitpid+0x13/0x17
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: htt_server    S 00000000  2160  3514   3513                     (NOTLB)
Dec 22 10:07:59 data_server kernel: f5d1df28 00000086 000000d0 00000000 c032a380 00000000 000000d0 c032cb80 
Dec 22 10:07:59 data_server kernel:        c0144080 00000001 00000000 c201dde0 00000000 0000a0de e2c496ab 0000000d 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5caac70 f5caaddc f4caa00c f5ce1778 00000000 7fffffff f5d1dfa0 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0144080>] __alloc_pages+0xb4/0x29d
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c02d5142>] apic_timer_interrupt+0x1a/0x20
Dec 22 10:07:59 data_server kernel:  [<c016bcd1>] do_poll+0x8e/0xac
Dec 22 10:07:59 data_server kernel:  [<c016bebc>] sys_poll+0x1cd/0x281
Dec 22 10:07:59 data_server kernel:  [<c016f070>] dput+0x34/0x1a7
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: crond         S 000081A4  2504  3524      1          3562  3513 (NOTLB)
Dec 22 10:07:59 data_server kernel: f57f1f5c 00000082 001301b6 000081a4 00000001 00000000 00000000 00000000 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 000000ff c2025de0 00000001 000015da 62f12f6e 00000469 
Dec 22 10:07:59 data_server kernel:        f7e410b0 f5df2d50 f5df2ebc 00000000 00000246 004675ff 004675ff 0000ea6b 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c012a7f6>] sys_nanosleep+0x103/0x1a1
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: xfs           S 7D192025  2236  3562      1          3581  3524 (NOTLB)
Dec 22 10:07:59 data_server kernel: c23c9ec4 00000086 f5dcf1e4 7d192025 c0322a80 c032a380 c032a380 00000246 
Dec 22 10:07:59 data_server kernel:        c0143fa4 c169ce60 000000d0 c201dde0 00000000 000050c7 1006b22e 0000046b 
Dec 22 10:07:59 data_server kernel:        c0322a80 f705f1b0 f705f31c 00000000 00000246 004ecf79 004ecf79 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c01c31ca>] direct_strncpy_from_user+0x3e/0x5d
Dec 22 10:07:59 data_server kernel:  [<c0126229>] sys_gettimeofday+0x53/0xac
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: atd           S 00000246  2720  3581      1          3591  3562 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5487f5c 00000086 f88e0555 00000246 00000000 f4e3d0f8 f7e89400 f7e89200 
Dec 22 10:07:59 data_server kernel:        f88d06ef f4c098b8 00000000 c201dde0 00000000 000177a8 4779ac17 0000046b 
Dec 22 10:07:59 data_server kernel:        c0322a80 c23b20f0 c23b225c 00000000 00000246 004a3f6b 004a3f6b 0004940f 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<f88e0555>] ext3_readdir+0x8c/0x3a0 [ext3]
Dec 22 10:07:59 data_server kernel:  [<f88d06ef>] journal_stop+0x1e1/0x1ed [jbd]
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c012a7f6>] sys_nanosleep+0x103/0x1a1
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: dbus-daemon-1 S 00000000  2980  3591      1          3604  3581 (NOTLB)
Dec 22 10:07:59 data_server kernel: f5d18f28 00000082 000000d0 00000000 c23b2bf0 c23b2bf0 000000d0 c032cb80 
Dec 22 10:07:59 data_server kernel:        c0144080 00000000 c201e7c0 c201dde0 00000000 00000124 1291ed23 0000000f 
Dec 22 10:07:59 data_server kernel:        c23b2bf0 f5d67270 f5d673dc f4e87060 f5cc7758 00000000 7fffffff f5d18fa0 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0144080>] __alloc_pages+0xb4/0x29d
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c0278689>] sock_poll+0x12/0x14
Dec 22 10:07:59 data_server kernel:  [<c016bc1b>] do_pollfd+0x59/0x81
Dec 22 10:07:59 data_server kernel:  [<c016bcd1>] do_poll+0x8e/0xac
Dec 22 10:07:59 data_server kernel:  [<c016bebc>] sys_poll+0x1cd/0x281
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c0126229>] sys_gettimeofday+0x53/0xac
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel:  [<c02d007b>] packet_rcv+0x242/0x307
Dec 22 10:07:59 data_server kernel: cups-config-d S 00000000  2396  3604      1          3615  3591 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4e68f28 00000082 000000d0 00000000 c032a380 00000000 000000d0 c032cb80 
Dec 22 10:07:59 data_server kernel:        c0144080 00000001 00000000 c201dde0 00000000 000317f0 70062892 0000000e 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5e38d90 f5e38efc f4e6f00c f5cc7278 00000000 7fffffff f4e68fa0 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0144080>] __alloc_pages+0xb4/0x29d
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c0278689>] sock_poll+0x12/0x14
Dec 22 10:07:59 data_server kernel:  [<c016bc1b>] do_pollfd+0x59/0x81
Dec 22 10:07:59 data_server kernel:  [<c016bcd1>] do_poll+0x8e/0xac
Dec 22 10:07:59 data_server kernel:  [<c016bebc>] sys_poll+0x1cd/0x281
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: hald          S C1692360  2504  3615      1          3624  3604 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4e86f28 00000082 c0143fa4 c1692360 000000d0 f7c03650 c032a380 00000000 
Dec 22 10:07:59 data_server kernel:        000000d0 00000000 c20267c0 c2025de0 00000001 00002481 b52276e7 00000476 
Dec 22 10:07:59 data_server kernel:        f7c03650 c23b2bf0 c23b2d5c 00000000 00000246 0046730c 0046730c f4e86fa0 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:07:59 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:07:59 data_server kernel:  [<c016bcd1>] do_poll+0x8e/0xac
Dec 22 10:07:59 data_server kernel:  [<c016bebc>] sys_poll+0x1cd/0x281
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c0126229>] sys_gettimeofday+0x53/0xac
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: mingetty      S C23B5E94  2428  3624      1          3625  3615 (NOTLB)
Dec 22 10:07:59 data_server kernel: c23b5e90 00000086 00000013 c23b5e94 000001ff f5ea2310 00000206 00000206 
Dec 22 10:07:59 data_server kernel:        c212d000 00000000 c201e7c0 c201dde0 00000000 00031e73 76558449 0000000e 
Dec 22 10:07:59 data_server kernel:        f5ea2310 f704c110 f704c27c 00000000 c21fec00 f5d6f000 7fffffff f5d6fc04 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c02016ab>] read_chan+0x31b/0x759
Dec 22 10:07:59 data_server kernel:  [<c01203e9>] remove_wait_queue+0xf/0x34
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c01fd01c>] tty_read+0xcf/0x127
Dec 22 10:07:59 data_server kernel:  [<c015af11>] vfs_read+0xb6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b124>] sys_read+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: mingetty      S F4E71E94  2980  3625      1          3626  3624 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4e71e90 00000086 00000013 f4e71e94 000001ff f704d190 f70432a6 00000246 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 c20267c0 c2025de0 00000001 00041a3f 8326e780 0000000e 
Dec 22 10:07:59 data_server kernel:        f704d190 f5ea2310 f5ea247c 00000000 c21fec00 f4e67000 7fffffff f4e67c04 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c02016ab>] read_chan+0x31b/0x759
Dec 22 10:07:59 data_server kernel:  [<c01203e9>] remove_wait_queue+0xf/0x34
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c01fd01c>] tty_read+0xcf/0x127
Dec 22 10:07:59 data_server kernel:  [<c015af11>] vfs_read+0xb6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b124>] sys_read+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: mingetty      S F4E72E94  2980  3626      1          3627  3625 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4e72e90 00000086 00000013 f4e72e94 000001ff c23b2bf0 f5d602a6 00000246 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 00061c3a 7f5abc8b 0000000e 
Dec 22 10:07:59 data_server kernel:        c23b2bf0 f5cbcc90 f5cbcdfc 00000000 c21fec00 f5486000 7fffffff f5486c04 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c02016ab>] read_chan+0x31b/0x759
Dec 22 10:07:59 data_server kernel:  [<c01203e9>] remove_wait_queue+0xf/0x34
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c01fd01c>] tty_read+0xcf/0x127
Dec 22 10:07:59 data_server kernel:  [<c015af11>] vfs_read+0xb6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b124>] sys_read+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: mingetty      S F4E6BE94  2980  3627      1          3628  3626 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4e6be90 00000082 00000013 f4e6be94 000001ff f5ccf230 f5dc62a6 00000246 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 c20267c0 c2025de0 00000001 0004dfc3 8d15c85d 0000000e 
Dec 22 10:07:59 data_server kernel:        f5ccf230 f704d190 f704d2fc 00000000 c21fec00 f5c98000 7fffffff f5c98c04 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c02016ab>] read_chan+0x31b/0x759
Dec 22 10:07:59 data_server kernel:  [<c01203e9>] remove_wait_queue+0xf/0x34
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c01fd01c>] tty_read+0xcf/0x127
Dec 22 10:07:59 data_server kernel:  [<c015af11>] vfs_read+0xb6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b124>] sys_read+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel:  [<c02d007b>] packet_rcv+0x242/0x307
Dec 22 10:07:59 data_server kernel: mingetty      S F53FDE94  2980  3628      1          3629  3627 (NOTLB)
Dec 22 10:07:59 data_server kernel: f53fde90 00000082 00000013 f53fde94 000001ff f7e405b0 c237f2a6 00000246 
Dec 22 10:07:59 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 00049caa 8a15f27a 0000000e 
Dec 22 10:07:59 data_server kernel:        f7e405b0 f5e6edd0 f5e6ef3c 00000000 c21fec00 f5cd5000 7fffffff f5cd5c04 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c02016ab>] read_chan+0x31b/0x759
Dec 22 10:07:59 data_server kernel:  [<c01203e9>] remove_wait_queue+0xf/0x34
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c01fd01c>] tty_read+0xcf/0x127
Dec 22 10:07:59 data_server kernel:  [<c015af11>] vfs_read+0xb6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b124>] sys_read+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: mingetty      S F4E62E94  2720  3629      1          4435  3628 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4e62e90 00000086 00000013 f4e62e94 000001ff 00000800 f5d502a6 00000246 
Dec 22 10:07:59 data_server kernel:        00000000 c0326b34 f5478000 c2025de0 00000001 00050b86 8f9206df 0000000e 
Dec 22 10:07:59 data_server kernel:        f7e410b0 f5ccf230 f5ccf39c 00000000 c21fec00 f5478000 7fffffff f5478c04 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c02016ab>] read_chan+0x31b/0x759
Dec 22 10:07:59 data_server kernel:  [<c01203e9>] remove_wait_queue+0xf/0x34
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c01fd01c>] tty_read+0xcf/0x127
Dec 22 10:07:59 data_server kernel:  [<c015af11>] vfs_read+0xb6/0xe2
Dec 22 10:07:59 data_server kernel:  [<c015b124>] sys_read+0x3c/0x62
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: sshd          S C0143FA4  2236  4389   3443          4740       (NOTLB)
Dec 22 10:07:59 data_server kernel: f482cec4 00000086 00000246 c0143fa4 c16a7700 000000d0 00000000 c032a380 
Dec 22 10:07:59 data_server kernel:        00000246 00000246 f5d7e000 c201dde0 00000000 00013c34 58c8d1de 00000438 
Dec 22 10:07:59 data_server kernel:        c0322a80 f5e802f0 f5e8045c f482ced4 c011e81a f5e20920 7fffffff 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: ccsd          S C0143FA4  2316  4435      1          4436  3629 (NOTLB)
Dec 22 10:07:59 data_server kernel: f4f94ec4 00000082 00000246 c0143fa4 c16942c0 000000d0 00000000 c032a380 
Dec 22 10:07:59 data_server kernel:        00000000 000000d0 c032cb80 c2025de0 00000001 000014e4 77c7b2df 00000475 
Dec 22 10:07:59 data_server kernel:        f7e410b0 f70906d0 f709083c 00000000 f4f94f58 f5e084c0 7fffffff 00000000 
Dec 22 10:07:59 data_server kernel: Call Trace:
Dec 22 10:07:59 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:07:59 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:07:59 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:07:59 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:07:59 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:07:59 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:07:59 data_server kernel:  [<c0170afe>] destroy_inode+0x3d/0x4c
Dec 22 10:07:59 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:07:59 data_server kernel: ccsd          S C0143FA4  2236  4436      1          4478  4435 (NOTLB)
Dec 22 10:08:00 data_server kernel: f4f40ec4 00000082 00000246 c0143fa4 c169e700 000000d0 00000000 c032a380 
Dec 22 10:08:00 data_server kernel:        00000000 000000d0 c032cb80 c201dde0 00000000 0003955a bbdf3388 000000bf 
Dec 22 10:08:00 data_server kernel:        c0322a80 f7090c50 f7090dbc 00000000 f4f40f58 f5c7d560 7fffffff 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:08:00 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:08:00 data_server kernel:  [<c012038d>] add_wait_queue+0x12/0x30
Dec 22 10:08:00 data_server kernel:  [<c027dce9>] datagram_poll+0x25/0xcc
Dec 22 10:08:00 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:08:00 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:08:00 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:08:00 data_server kernel:  [<c01507f5>] unmap_vma_list+0xe/0x17
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel: cman_comms    S F5D14CC0  2600  4478      1          4479  4436 (L-TLB)
Dec 22 10:08:00 data_server kernel: f485ffb0 00000046 c0389000 f5d14cc0 000005dc 00000000 00000000 00000000 
Dec 22 10:08:00 data_server kernel:        00000000 00000000 00000000 c2025de0 00000001 00000182 22316f6c 00000476 
Dec 22 10:08:00 data_server kernel:        f7e410b0 f705e6b0 f705e81c 00000000 fffffff5 f485f000 f4c16960 f931c80c 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<f92ff26d>] cluster_kthread+0xfd/0x316 [cman]
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<f92ff170>] cluster_kthread+0x0/0x316 [cman]
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: cman_serviced S 00000004  2660  4480      7                2227 (L-TLB)
Dec 22 10:08:00 data_server kernel: f4edcfc8 00000046 00000001 00000004 00000002 f705e6b0 37373736 2e393132 
Dec 22 10:08:00 data_server kernel:        2e342e31 00000000 c20267c0 c2025de0 00000001 00003df1 235947dd 00000357 
Dec 22 10:08:00 data_server kernel:        f705e6b0 f5e6f350 f5e6f4bc 00000000 f930a22c f4edc000 f4803da0 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<f930a22c>] serviced+0x0/0x146 [cman]
Dec 22 10:08:00 data_server kernel:  [<f930a22c>] serviced+0x0/0x146 [cman]
Dec 22 10:08:00 data_server kernel:  [<f930a34b>] serviced+0x11f/0x146 [cman]
Dec 22 10:08:00 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:08:00 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: cman_memb     S 00000000  2056  4479      1          4481  4478 (L-TLB)
Dec 22 10:08:00 data_server kernel: f48a3fb0 00000046 f93064f0 00000000 f48a3f7c 00000000 f48a3f84 f4eed120 
Dec 22 10:08:00 data_server kernel:        f9308d8c f931cb88 000005d4 c2025de0 00000001 0000014b 20901a00 00000476 
Dec 22 10:08:00 data_server kernel:        f7e410b0 f5e6f8d0 f5e6fa3c f48a3fc4 f4eed138 f5e6f8d0 f5e6f8d0 f48a3fd8 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<f93064f0>] do_membership_packet+0x6b/0x1cd [cman]
Dec 22 10:08:00 data_server kernel:  [<f9308d8c>] dispatch_messages+0x9f/0xcc [cman]
Dec 22 10:08:00 data_server kernel:  [<f930524f>] membership_kthread+0x163/0x3aa [cman]
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<f93050ec>] membership_kthread+0x0/0x3aa [cman]
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: cman_hbeat    S 00000001  2612  4481      1          4495  4479 (L-TLB)
Dec 22 10:08:00 data_server kernel: f481dfd4 00000046 00000008 00000001 f7e410b0 f705e6b0 00000008 00000000 
Dec 22 10:08:00 data_server kernel:        00000000 00000000 c20267c0 c2025de0 00000001 0000068e 22316053 00000476 
Dec 22 10:08:00 data_server kernel:        f705e6b0 c23b36f0 c23b385c c02d2cb2 00000000 f481d000 c23b36f0 f931cb44 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c02d2cb2>] __cond_resched+0x14/0x39
Dec 22 10:08:00 data_server kernel:  [<f9305000>] hello_kthread+0xc2/0x13f [cman]
Dec 22 10:08:00 data_server kernel:  [<f9304f3e>] hello_kthread+0x0/0x13f [cman]
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: fenced        S 00000000  2688  4495      1          4509  4481 (NOTLB)
Dec 22 10:08:00 data_server kernel: f4f48f9c 00000082 bfeeae3c 00000000 bfeeacc8 f5d66cf0 bfeead20 00656ff4 
Dec 22 10:08:00 data_server kernel:        00000008 00000000 c201e7c0 c201dde0 00000000 0000aaa5 c081f46a 000000bf 
Dec 22 10:08:00 data_server kernel:        f5d66cf0 f5e38810 f5e3897c 00000000 f4f48000 f4f48000 f4f48fac bfeead20 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0105157>] sys_rt_sigsuspend+0xed/0x108
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel: clurgmgrd     S 00000000  2688  4509      1  4510    5171  4495 (NOTLB)
Dec 22 10:08:00 data_server kernel: f4f39f28 00000082 f4813018 00000000 f5e2cbcc f5dad2b0 f5e6cf40 f5e6cf70 
Dec 22 10:08:00 data_server kernel:        f5e2cbcc 00000000 c201e7c0 c201dde0 00000000 00034415 010a5245 000000c0 
Dec 22 10:08:00 data_server kernel:        f5dad2b0 f5dac230 f5dac39c f4f39f70 f5dac338 f5dac2d4 fffffe00 f5dac230 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0125985>] do_wait+0x26e/0x449
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<c012dcac>] sys_rt_sigaction+0xdd/0xf2
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<c0125bf3>] sys_wait4+0x27/0x2a
Dec 22 10:08:00 data_server kernel:  [<c0125c09>] sys_waitpid+0x13/0x17
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel: clurgmgrd     S F4F68EF8  2216  4510   4509          4728       (NOTLB)
Dec 22 10:08:00 data_server kernel: f4f68ec4 00000082 00000000 f4f68ef8 c032a380 f5cbd790 00000246 c0143fa4 
Dec 22 10:08:00 data_server kernel:        c169ffa0 00000000 c201e7c0 c201dde0 00000000 00002123 77c8b9dc 00000475 
Dec 22 10:08:00 data_server kernel:        f5cbd790 f5dad2b0 f5dad41c 00000000 00000246 00467d7d 00467d7d 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:08:00 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:08:00 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:08:00 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:08:00 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:08:00 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel: clurgmgrd     S C0143FA4  2956  4728   4509                4510 (NOTLB)
Dec 22 10:08:00 data_server kernel: f4fbdec4 00000082 00000246 c0143fa4 c169eca0 000000d0 00000000 c032a380 
Dec 22 10:08:00 data_server kernel:        00000000 000000d0 c032cb80 c2025de0 00000001 00000f11 dd503962 00000476 
Dec 22 10:08:00 data_server kernel:        f7e410b0 f5dacd30 f5dace9c 00000000 00000246 004671c8 004671c8 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:08:00 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:08:00 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:08:00 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:08:00 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:08:00 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel:  [<c02d007b>] packet_rcv+0x242/0x307
Dec 22 10:08:00 data_server kernel: sshd          S C0143FA4  2532  4740   3443  4743    5379  4389 (NOTLB)
Dec 22 10:08:00 data_server kernel: f5c51ec4 00000082 00000246 c0143fa4 c1693f80 f704cc10 00000000 c032a380 
Dec 22 10:08:00 data_server kernel:        00000246 00000000 c201e7c0 c201dde0 00000000 00001774 c5e08f22 00000476 
Dec 22 10:08:00 data_server kernel:        f704cc10 c22cf6b0 c22cf81c f5c51ed4 c011e81a f5ead600 7fffffff 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:08:00 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:08:00 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:08:00 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:08:00 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:08:00 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:08:00 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel:  [<c02d007b>] packet_rcv+0x242/0x307
Dec 22 10:08:00 data_server kernel: bash          R running  2628  4743   4740                     (NOTLB)
Dec 22 10:08:00 data_server kernel: mount         D F933398D  2244  5171      1                4509 (NOTLB)
Dec 22 10:08:00 data_server kernel: f4899c8c 00000086 f5ec9616 f933398d f5ec93f0 f4899c68 c011cd26 00000009 
Dec 22 10:08:00 data_server kernel:        f5e6f350 f4899c78 c011cd26 c201dde0 00000000 00001b4d 2334d96f 00000357 
Dec 22 10:08:00 data_server kernel:        c0322a80 f5cbc710 f5cbc87c 00000000 f7e9ce60 f7e9ce8c f7e9ce88 f4899ca8 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c011cd26>] recalc_task_prio+0x128/0x133
Dec 22 10:08:00 data_server kernel:  [<c011cd26>] recalc_task_prio+0x128/0x133
Dec 22 10:08:00 data_server kernel:  [<c02d26fb>] wait_for_completion+0x94/0xcb
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<f930e3d9>] kcl_join_service+0x135/0x13d [cman]
Dec 22 10:08:00 data_server kernel:  [<f93268da>] new_lockspace+0x50d/0x583 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f93269b3>] dlm_new_lockspace+0x63/0x86 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f91b54fc>] init_gdlm+0x11/0x29 [lock_dlm]
Dec 22 10:08:00 data_server kernel:  [<f91b5737>] lm_dlm_mount+0x50/0xbc [lock_dlm]
Dec 22 10:08:00 data_server kernel:  [<f91af2db>] lm_mount+0x148/0x1a0 [lock_harness]
Dec 22 10:08:00 data_server kernel:  [<f97527f6>] gfs_lm_mount+0x1be/0x2bf [gfs]
Dec 22 10:08:00 data_server kernel:  [<f9752624>] lm_cb+0x0/0x14 [gfs]
Dec 22 10:08:00 data_server kernel:  [<f975ec70>] fill_super+0x4c4/0xf65 [gfs]
Dec 22 10:08:00 data_server kernel:  [<c01c1284>] radix_tree_gang_lookup+0x39/0x4e
Dec 22 10:08:00 data_server kernel:  [<c0140976>] find_get_pages+0x23/0x50
Dec 22 10:08:00 data_server kernel:  [<c014099c>] find_get_pages+0x49/0x50
Dec 22 10:08:00 data_server kernel:  [<c01490dc>] pagevec_lookup+0x17/0x1d
Dec 22 10:08:00 data_server kernel:  [<c0149515>] invalidate_mapping_pages+0xb3/0xc6
Dec 22 10:08:00 data_server kernel:  [<c0116c13>] smp_call_function+0xbf/0xc9
Dec 22 10:08:00 data_server kernel:  [<c015d2df>] invalidate_bh_lru+0x0/0x36
Dec 22 10:08:00 data_server kernel:  [<c015d305>] invalidate_bh_lru+0x26/0x36
Dec 22 10:08:00 data_server kernel:  [<c016129a>] set_blocksize+0x77/0x7c
Dec 22 10:08:00 data_server kernel:  [<f975f840>] gfs_get_sb+0x114/0x159 [gfs]
Dec 22 10:08:00 data_server kernel:  [<c0173054>] alloc_vfsmnt+0x78/0x9f
Dec 22 10:08:00 data_server kernel:  [<c0160f1b>] do_kern_mount+0x8a/0x147
Dec 22 10:08:00 data_server kernel:  [<c017400b>] do_new_mount+0x61/0x90
Dec 22 10:08:00 data_server kernel:  [<c0174658>] do_mount+0x178/0x190
Dec 22 10:08:00 data_server kernel:  [<c0144080>] __alloc_pages+0xb4/0x29d
Dec 22 10:08:00 data_server kernel:  [<c01749af>] sys_mount+0x91/0x108
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel: dlm_astd      S F49D8FCC  3744  5184      6          5185  2262 (L-TLB)
Dec 22 10:08:00 data_server kernel: f49d8fc0 00000046 c02d25c9 f49d8fcc c0322a80 f7e41630 00000003 f5cbc710 
Dec 22 10:08:00 data_server kernel:        f7c030d0 00000246 00000246 c201dde0 00000000 000001d9 4df1527b 00000475 
Dec 22 10:08:00 data_server kernel:        c0322a80 f5ec88f0 f5ec8a5c f931fa88 f931fb61 f49d8000 00000000 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c02d25c9>] schedule+0x83d/0x8db
Dec 22 10:08:00 data_server kernel:  [<f931fa88>] process_deadlockqueue+0x20/0xe2 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f931fb61>] dlm_astd+0x0/0x1a9 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f931fb61>] dlm_astd+0x0/0x1a9 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f931fc70>] dlm_astd+0x10f/0x1a9 [dlm]
Dec 22 10:08:00 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:08:00 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: dlm_recvd     S 00000000  3352  5185      6          5186  5184 (L-TLB)
Dec 22 10:08:00 data_server kernel: f4a27fc0 00000046 00000019 00000000 00001000 f5ec93f0 00000fe7 f4ecd000 
Dec 22 10:08:00 data_server kernel:        00000000 00000000 c201e7c0 c201dde0 00000000 00002546 2358bd14 00000357 
Dec 22 10:08:00 data_server kernel:        f5ec93f0 f5ec9970 f5ec9adc f933b604 f4a05610 f4a27000 00000000 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<f9328270>] dlm_recvd+0x0/0x95 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f93282ec>] dlm_recvd+0x7c/0x95 [dlm]
Dec 22 10:08:00 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:08:00 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: dlm_sendd     S C027B0F3  3148  5186      6          5187  5185 (L-TLB)
Dec 22 10:08:00 data_server kernel: f49bbfc0 00000046 f5b8d640 c027b0f3 00000019 00000246 00000019 c029e682 
Dec 22 10:08:00 data_server kernel:        c16921a0 f4a05680 00000286 c201dde0 00000000 00004303 2355513e 00000357 
Dec 22 10:08:00 data_server kernel:        c0322a80 f5e39310 f5e3947c f933b614 00000246 f49bb000 00000000 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c027b0f3>] release_sock+0xf/0x4f
Dec 22 10:08:00 data_server kernel:  [<c029e682>] tcp_sendpage+0x56/0x5e
Dec 22 10:08:00 data_server kernel:  [<f9328358>] dlm_sendd+0x0/0x9a [dlm]
Dec 22 10:08:00 data_server kernel:  [<f93283d4>] dlm_sendd+0x7c/0x9a [dlm]
Dec 22 10:08:00 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:08:00 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: dlm_recoverd  S 00000000  3500  5187      6                5186 (L-TLB)
Dec 22 10:08:00 data_server kernel: f495efbc 00000046 00000001 00000000 00000001 00000000 0000000f 00000286 
Dec 22 10:08:00 data_server kernel:        00000000 f7067200 f5b1fb40 c201dde0 00000000 00000e9e 2358eb7e 00000357 
Dec 22 10:08:00 data_server kernel:        c0322a80 f5ec93f0 f5ec955c f932f27b f5b1fb40 f495e000 00000000 f7067200 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<f932f27b>] do_ls_recovery+0x331/0x337 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f932f281>] dlm_recoverd+0x0/0x57 [dlm]
Dec 22 10:08:00 data_server kernel:  [<f932f2a9>] dlm_recoverd+0x28/0x57 [dlm]
Dec 22 10:08:00 data_server kernel:  [<c01341ed>] kthread+0x73/0x9b
Dec 22 10:08:00 data_server kernel:  [<c013417a>] kthread+0x0/0x9b
Dec 22 10:08:00 data_server kernel:  [<c01041f5>] kernel_thread_helper+0x5/0xb
Dec 22 10:08:00 data_server kernel: sshd          S C0143FA4  2532  5379   3443  5382          4740 (NOTLB)
Dec 22 10:08:00 data_server kernel: f4a43ec4 00000082 00000246 c0143fa4 c1694580 000000d0 00000000 c032a380 
Dec 22 10:08:00 data_server kernel:        00000246 00000246 f4a4f000 c2025de0 00000001 0003ae31 e561e44f 00000476 
Dec 22 10:08:00 data_server kernel:        f7e410b0 f5ea2e10 f5ea2f7c f4a43ed4 c011e81a f5c7d380 7fffffff 00000000 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c0143fa4>] buffered_rmqueue+0x17d/0x1a5
Dec 22 10:08:00 data_server kernel:  [<c011e81a>] __wake_up+0x29/0x3c
Dec 22 10:08:00 data_server kernel:  [<c02d2e55>] schedule_timeout+0xb9/0x154
Dec 22 10:08:00 data_server kernel:  [<c01fc52d>] tty_ldisc_deref+0x50/0x5f
Dec 22 10:08:00 data_server kernel:  [<c016b740>] do_select+0x291/0x2c6
Dec 22 10:08:00 data_server kernel:  [<c016b375>] __pollwait+0x0/0x95
Dec 22 10:08:00 data_server kernel:  [<c016ba68>] sys_select+0x2e0/0x43a
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel:  [<c02d007b>] packet_rcv+0x242/0x307
Dec 22 10:08:00 data_server kernel: bash          S 00000001  2804  5382   5379  5415               (NOTLB)
Dec 22 10:08:00 data_server kernel: f4f5bf28 00000082 c03bfe80 00000001 00000086 f705ec30 c011d2e4 f5dd0c70 
Dec 22 10:08:00 data_server kernel:        f5c6a754 00000000 c201e7c0 c201dde0 00000000 0002fdda 4299db0d 00000475 
Dec 22 10:08:00 data_server kernel:        f705ec30 f70911d0 f709133c f705ecdc f705ec30 f7091274 fffffe00 f70911d0 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c011d2e4>] try_to_wake_up+0x28e/0x299
Dec 22 10:08:00 data_server kernel:  [<c0125985>] do_wait+0x26e/0x449
Dec 22 10:08:00 data_server kernel:  [<c011e7d6>] __wake_up_common+0x36/0x51
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<c011e794>] default_wake_function+0x0/0xc
Dec 22 10:08:00 data_server kernel:  [<c0125bf3>] sys_wait4+0x27/0x2a
Dec 22 10:08:00 data_server kernel:  [<c0125c09>] sys_waitpid+0x13/0x17
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb
Dec 22 10:08:00 data_server kernel: tail          S 00008180  2980  5415   5382                     (NOTLB)
Dec 22 10:08:00 data_server kernel: f4a30f5c 00000082 00708032 00008180 00000001 f5ea2e10 00000000 00000000 
Dec 22 10:08:00 data_server kernel:        00000000 00000000 c20267c0 c2025de0 00000001 00020eac e5094bf9 00000476 
Dec 22 10:08:00 data_server kernel:        f5ea2e10 f705ec30 f705ed9c 00000000 00000246 0046724b 0046724b 000003ea 
Dec 22 10:08:00 data_server kernel: Call Trace:
Dec 22 10:08:00 data_server kernel:  [<c02d2ed5>] schedule_timeout+0x139/0x154
Dec 22 10:08:00 data_server kernel:  [<c012a6de>] process_timeout+0x0/0x5
Dec 22 10:08:00 data_server kernel:  [<c012a7f6>] sys_nanosleep+0x103/0x1a1
Dec 22 10:08:00 data_server kernel:  [<c02d4703>] syscall_call+0x7/0xb






------------------------------------------------------
Passa a Infostrada. ADSL e Telefono senza limiti e senza canone Telecom
http://click.libero.it/infostrada22dic06