[Linux-cluster] dlm_sendd 100% CPU

[David Ayre]

Any ideas why dlm_sendd is taking up 100% CPU ?

We have this intermittent problem which seems to happen when a group  
of users log into our servers at the same time, possibly opening/ 
writing files from the same mount point.

Should dlm_sendd ever take up 100% of the CPU ?

Any help would be greatly appreciated.... i mean greatly.... like sell  
my soul "greatly".  If there is any additional diagnostic info i can  
provide, please let me know.    /var/log/messages just shows failed  
SMB pipes/write failures due to the high load/lockups.

Version info:
----------------------------
Kernel:   2.6.9-55.0.6.ELsmp #1 SMP
GFS:  2.6.9-72.2.0.7
SMB: 3.0.10-1.4E.12.2

Top:
----------------------------
top - 15:48:44 up 53 days, 13:35,  3 users,  load average: 57.40,  
48.47, 37.00
Tasks: 257 total,  64 running, 193 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.3% us, 99.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,   
0.0% si
Cpu1  :  0.0% us, 100.0% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,   
0.0% si
Cpu2  :  0.3% us, 99.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,   
0.0% si
Cpu3  :  0.0% us, 100.0% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,   
0.0% si
Mem:   4147164k total,  1089108k used,  3058056k free,     5200k buffers
Swap:  2031608k total,      224k used,  2031384k free,   795340k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
4218 root       9 -10     0    0    0 R  100  0.0   1066:10 dlm_sendd
16846 root      22   0     0    0    0 R    9  0.0 173:36.82 nfsd
16848 root      25   0     0    0    0 R    9  0.0 171:54.71 nfsd
6169 jsalkeld  25   0  8400 3072 2384 R    9  0.1   0:12.12 imapd
8407 bjeon     25   0 14048 3112 2212 R    8  0.1   0:04.80 smbd
16847 root      25   0     0    0    0 R    8  0.0 174:12.58 nfsd
16850 root      25   0     0    0    0 R    8  0.0 172:22.35 nfsd
8116 bjeon     25   0 14048 3212 2312 R    8  0.1   0:14.97 smbd
8236 bschuetz  25   0 14048 3396 2492 R    8  0.1   0:18.72 smbd
8308 wuj       25   0 14048 3152 2252 R    8  0.1   0:08.19 smbd
8402 jacekim   25   0 14048 3340 2416 R    8  0.1   0:03.98 smbd
8443 wuj       25   0 14064 3172 2272 R    8  0.1   0:03.60 smbd
8355 wuj       25   0 14048 3260 2344 R    7  0.1   0:09.02 smbd
8386 ashatzko  25   0 14048 3300 2396 R    7  0.1   0:05.78 smbd
8417 wuj       25   0 14064 3204 2304 R    7  0.1   0:04.58 smbd
16845 root      25   0     0    0    0 R    7  0.0 172:15.61 nfsd
8432 rkhare    25   0 14064 3280 2360 R    7  0.1   0:03.95 smbd
8108 wuj       25   0 14048 3300 2380 R    7  0.1   0:30.34 smbd
8211 yjlee     25   0 14048 3380 2476 R    7  0.1   0:08.29 smbd
8297 bjeon     25   0 14048 3316 2380 R    7  0.1   0:13.02 smbd
8343 ssing     25   0 14048 3232 2320 R    7  0.1   0:08.49 smbd
16844 root      25   0     0    0    0 R    7  0.0 171:46.80 nfsd
8112 mtaghavi  25   0 14048 3408 2504 R    6  0.1   0:25.97 smbd
8182 wuj       25   0 14048 3200 2300 R    6  0.1   0:09.37 smbd
8438 atweedie  25   0 14064 3316 2412 R    6  0.1   0:03.69 smbd
8447 ssing     25   0 14072 3308 2384 R    6  0.1   0:01.38 smbd
8121 cmeire    25   0 14048 3396 2492 R    6  0.1   0:15.93 smbd
8223 akapur    25   0 14048 3372 2468 R    6  0.1   0:06.73 smbd
8241 jfromyr   25   0 14048 3376 2472 R    6  0.1   0:13.39 smbd
8345 jwallace  25   0 14140 3428 2476 R    6  0.1   0:05.99 smbd


~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~_~
David Ayre
Programmer/Analyst - Information Technlogy Services
Emily Carr Institute of Art and Design
Vancouver, B.C.   Canada
604-844-3875 /  david at eciad.ca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20080226/6ce1f68b/attachment.htm>