[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Poor thread performance on Linux vs. Solaris



On Sunday 07 September 2003 11:47 pm, you wrote:

> On Thu, 4 Sep 2003, Bill Soudan wrote:
>
> have you tried oprofile? It should be able to profile your app too.

I have already, it was one of the tools I used to track the system time to 
the futex spinlock.  I've also used Intel's proprietary vtune tool, which 
points to the same culprit.

> How does 'vmstat 1' output look before, during and after such a freeze?
> Does it show a sharp increase in context-switching? Is there a sharp
> increase in runqueue length? [nr of tasks running]

To my untrained eye, context switching appears very high and volatile for 
the duration of the test... the only time it drops is during the 
mysterious freezes, during which system time spikes up to 80-100%.  I've 
included the vmstat output for the duration of the test.  Notice the 
particularly bad freeze near the end of the test, 99% system time for 2 
seconds.

Runqueue length is usually 0 for the duration of the test.

These results are with arjan's 2.6.0-test4 kernel + Boris Hu's 
futex_q_lock patch.  For comparison, I've also attached the vmstat output 
from a run on a 2way 450mhz Ultra-60 running Solaris 5.8.

> the system overhead alone is not a problem, Linux uses a kernel
> mechanism for which other OSs often use userspace spinlocks. 

Interesting you mention this, I'd suspected this but I wasn't certain.  So 
concievably we could be doing just as much lock thrashing on Solaris, we 
just wouldn't notice it since it'd be lumped in with the user time.  I'd 
really like to find a tool that can measure lock contention.

> But obviously the 'freezing' of your app and the low throughput is a 
> bad thing.

Currently, the throughput is only about 4x of a Sun Ultra-60 box (dual 
450mHz UltraSparc II).  I don't believe we're I/O bound at all, so with a 
rough guess on our 4way 2.668ghz Xeon I'd expect at least an 8-10x 
performance increase.  If I could translate the system time into user 
time, gain 30% more throughput, then I'll be at 6x.  Whether I can get 
this by tweaking/fixing the kernel or fixing our app is the question.

At the very least, the system time spikes look like a kernel problem to 
me.

> one thing looks interesting - your top output shows ~10% idle time, is
> that correct? Could you try the latest -mm kernel from:
>
> 	kernel.org/pub/linux/kernel/people/akpm/patches
>
> this also has some scheduler enhancements. (that might as well be a
> degradation for your app though.)

Yes, the idle time is correct.  I'll give the mm kernel a shot later today 
or first thing tomorrow.   Next on my list is to spend some more time 
with oprofile now that I have Boris' patch applied.

Bill


STATEMENT OF CONFIDENTIALITY

The information contained in this electronic message and any attachments
to this message are intended for the exclusive use of the addressee(s)
and may contain confidential or privileged information. If you are not
the intended recipient, please notify SunGard Trading Systems immediately
at (201) 499-5900 and destroy all copies of this message and any
attachments.

/data/build/bsoudan/taap_UMA27_devel/bin> vmstat 1
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0      0 3571524   7908  57916    0    0    34     9  336   723 15 10 75
 0  0  0      0 3571524   7908  57916    0    0     0     0 1006    18  0  0 100
 0  0  0      0 3571524   7908  57916    0    0     0     0 1007    16  0  0 100
 0  0  0      0 3571540   7908  57916    0    0     0     0 1007    18  0  0 100
 0  0  0      0 3571556   7916  57908    0    0     0    36 1031    42  0  1 99
 0  0  0      0 3571556   7916  57908    0    0     0     0 1007    14  0  0 100
 0  0  0      0 3571556   7916  57908    0    0     0     0 1008    18  0  0 100
 0  0  1      0 3571396   7916  58044    0    0     0     0 2662 10165 43 16 41
 0  0  1      0 3571380   7916  58044    0    0     0     0 3309 13028 68 21 11
 0  0  0      0 3571348   7924  58036    0    0     0   228 3292 13393 64 20 16
 0  0  0      0 3571348   7924  58104    0    0     0     0 3265 13002 69 20 11
 0  0  0      0 3571348   7924  58172    0    0     0     0 3352 14240 67 22 11
 0  0  0      0 3571348   7924  58172    0    0     0     0 3354 13652 69 20 11
 0  0  0      0 3571268   7924  58172    0    0     0     0 3390 17100 67 21 11
 0  0  2      0 3571268   7932  58164    0    0     0    68 3352 17269 65 21 14
 0  0  1      0 3571268   7932  58164    0    0     0     0 3430 27358 66 24 11
 0  0  1      0 3571268   7932  58164    0    0     0    40 3416 18307 65 22 13
 0  0  0      0 3571268   7932  58164    0    0     0     0 3425 16091 67 21 12
 0  0  0      0 3571268   7932  58164    0    0     0     0 3416 16119 64 24 12
 1  0  0      0 3571204   7940  58156    0    0     0    68 3401 15631 64 26 10
 0  0  1      0 3571204   7940  58224    0    0     0     0 3442 15894 66 25  9
 0  0  0      0 3571204   7940  58224    0    0     0     0 3420 16503 67 20 13
 0  0  1      0 3571204   7940  58224    0    0     0     0 3394 23001 63 25 12
 0  0  1      0 3571204   7940  58224    0    0     0     0 3413 25372 62 24 13
 1  0  0      0 3571204   7948  58216    0    0     0    68 3421 21533 65 24 11
 0  0  1      0 3571204   7948  58284    0    0     0     0 3429 16453 65 22 13
 0  0  0      0 3570628   7948  58284    0    0     0     0 3422 16711 67 22 11
 0  0  1      0 3566100   7948  58284    0    0     0     0 3539 17518 63 25 12
 0  0  1      0 3561492   7948  58284    0    0     0     0 3514 16667 64 25 11
 1  0  0      0 3555684   7956  58276    0    0     0    68 3374 17230 64 25 12
 0  0  0      0 3552996   7956  58276    0    0     0     0 3203 20928 65 20 15
 0  0  0      0 3549668   7956  58276    0    0     0     0 2532 10802 69 16 14
 0  0  0      0 3546148   7956  58276    0    0     0     0 1366  4829 66 21 13
 0  0  0      0 3544292   7956  58276    0    0     0     0 1322  4920 61 26 14
 1  0  0      0 3543908   7964  58268    0    0     0    64 1314  6799 41 48 11
 0  0  0      0 3542884   7964  58268    0    0     0     0 1340  5767 43 45 12
 0  0  0      0 3541604   7964  58268    0    0     0     0 1376  6473 72 15 13
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0      0 3540772   7964  58268    0    0     0     0 1376  6319 64 23 13
 0  0  0      0 3540260   7964  58268    0    0     0     0 1122  3750 21 73  6
 1  0  0      0 3540260   7972  58260    0    0     0    56 1115  2674  9 89  2
 0  0  0      0 3540260   7972  58260    0    0     0     0 1350  9071 35 56 10
 0  0  0      0 3540260   7972  58260    0    0     0    16 1191  5283 20 76  4
 0  0  0      0 3540260   7972  58260    0    0     0     0 1032   747  3 96  0
 0  0  0      0 3540260   7972  58260    0    0     0     0 1139  3428 14 81  5
 1  0  0      0 3539812   7980  58320    0    0     0    40 1312  6989 37 52 12
 0  0  0      0 3538916   7980  58320    0    0     0     0 1360 11262 54 30 16
 0  0  0      0 3538852   7980  58320    0    0     0     0 1147  3152 15 82  3
 0  0  0      0 3537956   7980  58320    0    0     0     0 1338  7848 42 45 13
 0  0  0      0 3537828   7980  58320    0    0     0     0 1354  9777 50 34 15
 1  0  0      0 3537636   7988  58312    0    0     0    56 1296  7845 43 46 11
 0  0  0      0 3537316   7988  58312    0    0     0     0 1261  7192 31 61  8
 0  0  0      0 3537124   7988  58312    0    0     0     0 1266  6786 47 41 12
 0  0  0      0 3536996   7988  58312    0    0     0     0 1350  6681 53 35 12
 0  0  0      0 3536932   7988  58312    0    0     0     0 1331  7851 52 35 13
 0  0  0      0 3536804   7996  58372    0    0     0    60 1318  7346 40 50 11
 0  0  0      0 3536612   7996  58372    0    0     0     0 1338  7608 57 32 11
 0  0  0      0 3536228   7996  58440    0    0     0     0 1207  3261 29 66  6
 0  0  0      0 3536228   7996  58440    0    0     0     0 1098  2481 11 87  2
 0  0  0      0 3536420   7996  58440    0    0     0     0 1339  9067 41 45 14
 0  0  0      0 3536676   8004  58432    0    0     0    56 1255  8130 45 42 14
 0  0  0      0 3536548   8004  58432    0    0     0     0 1379  6564 37 53 10
 0  0  0      0 3536548   8004  58432    0    0     0     0 1306  5329 43 48  9
 0  0  0      0 3536548   8004  58432    0    0     0     0 1322  6817 66 20 14
 0  0  0      0 3536548   8004  58432    0    0     0     0 1106  1137 10 88  2
 0  0  0      0 3537252   8012  58424    0    0     0    60 1388  7836 73 12 15
 0  0  0      0 3537188   8012  58424    0    0     0     0 1347  7703 72 12 16
 0  0  0      0 3537188   8012  58424    0    0     0     0 1368  7177 69 16 15
 0  0  0      0 3537188   8012  58424    0    0     0     0 1371  6058 58 26 16
 0  0  0      0 3537188   8012  58424    0    0     0     0 1362  6481 65 17 18
 0  0  0      0 3536228   8020  58416    0    0     0    64 1194  3321 28 67  5
 0  0  0      0 3535588   8020  58416    0    0     0     0 1381  7399 63 19 18
 0  0  0      0 3535332   8020  58416    0    0     0     0 1342  6313 55 28 18
 0  0  0      0 3535268   8020  58416    0    0     0     0 1010    36  1 99  0
 0  0  0      0 3535268   8020  58416    0    0     0     0 1008    19  1 99  0
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0      0 3535268   8028  58408    0    0     0    48 1396  7179 55 27 18
 0  0  0      0 3536484   8028  58408    0    0     0     0 1377  6094 60 12 28
 0  0  0      0 3537444   8028  58476    0    0     0    20 1364   324 27  2 71
 0  0  0      0 3537444   8028  58476    0    0     0     0 1353   123  4  2 94
 0  0  0      0 3537764   8028  58476    0    0     0     0 1359   108  1  2 98
 0  0  0      0 3537764   8044  58460    0    0     0    80 1368   142  1  2 98

 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 0 0 0 6702816 230264 0   0  0  0  0  0  0  0  0  0  1  478 3400 1050  0  6 94
 0 0 0 6702816 230264 0   0  0  0  0  0  0  0  0  0  0  441  422  361  0  0 100
 0 0 0 6702816 230264 2 106  0  0  0  0  0  0  0  0  0  449  594  377  0  0 99
 0 0 0 6702816 230264 0   0  0  0  0  0  0  0  0  0  0  581  406  628  0  0 100
 3 0 0 6702376 228584 0 216  0  0  0  0  0  0  0  0  0 3897 7449 5107 49 12 39
 3 0 0 6702376 228536 0  13  0  0  0  0  0  1  1  1 18 3698 7337 2222 73 27  0
 3 0 0 6702376 228400 0   7  0  0  0  0  0  0  0  0  0 4721 4860 2673 84 15  0
 4 0 0 6702376 228392 0   0  0  0  0  0  0  0  0  0  0 4347 4785 2502 86 13  0
 4 0 0 6702376 228392 0   1  0  0  0  0  0  0  0  0  0 4606 5762 3581 87 13  0
 3 0 0 6702376 228392 0   0  0  0  0  0  0  0  0  0  0 5317 8362 5968 80 19  0
 6 0 0 6702304 228376 0  12  0  0  0  0  0  0  0  0  0 5668 14403 8708 74 25 1
 6 0 0 6702304 228216 0  46  0  0  0  0  0  0  0  0  0 4695 13084 8693 88 11 1
 5 0 0 6700792 226728 0 347  0  0  0  0  0  0  0  0  0 4223 6723 4240 85 13  1
 4 0 0 6698560 224504 0 359  0  0  0  0  0  0  0  0  0 4185 6376 3852 84 13  2
 3 0 0 6696400 222280 0 365  0  0  0  0  0  0  0  0  0 4823 8043 5558 77 21  2
 3 0 0 6693592 219576 0 501  0  0  0  0  0  0  0  0  4 4775 11036 5653 77 20 2
 3 0 0 6689848 215920 0 482  0  0  0  0  0  0  0  0  0 4011 4977 2615 86 13  1
 5 0 0 6687472 213544 0 252  0  0  0  0  0  0  0  0  0 3481 4607 1800 83 14  2
 3 0 0 6685744 211760 0 254  0  0  0  0  0  0  0  0  0 3390 4483 1763 83 13  3
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 3 0 0 6684736 210752 0   5  0  0  0  0  0  0  0  0  0 2882 3455 1187 92  8  0
 3 0 0 6684736 210744 0   0  0  0  0  0  0  0  0  0  2 2842 5586 1386 91  9  0
 3 0 0 6684736 210720 1  16  0  0  0  0  0  0  0  0  0 2725 3482 1059 91  9  0
 3 0 0 6684736 210632 0   1  0  0  0  0  0  0  0  0  0 2780 3522  983 95  5  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2755 3510 1070 89 11  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2995 3323 1185 91  9  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  2 2847 4517 1245 85 15  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2853 4576 1330 91  9  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2760 3438  971 93  7  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2856 5459 1110 87 13  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 3121 3657 1541 90 10  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  1  1  1  3 2764 3855 1076 85 14  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2920 5627 1564 87 13  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2808 3381 1080 93  7  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2840 3400 1090 92  8  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2760 3510  979 94  6  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  1  1  1 20 2668 2918 1027 71 29  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 3163 6525 2046 83 17  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2915 3537 1341 93  7  0
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2824 7378 1143 89 11  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2837 3351 1023 91  9  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  1  1  1  5 2940 3451 1353 85 15  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2805 5956 1357 87 13  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2884 3327 1027 93  7  0
 3 0 0 6684736 210632 0   0  0  0  0  0  0  0  0  0  0 2890 3660 1245 91  9  0
 2 0 0 6684880 210776 0   0  0  0  0  0  0  0  0  0  0 2780 5400 1005 89 10  0
 3 0 0 6684880 210776 0   0  0  0  0  0  0  0  0  0  4 2972 3542 1473 85 12  2
 4 0 0 6684880 210776 0   0  0  0  0  0  0  0  0  0  0 2894 5346 1434 85 13  2
 3 0 0 6684880 210776 16  0 128 0  0  0  0  1  0  1  0 2831 3389 1040 92  7  1
 2 0 0 6684880 210888 0   0  0  0  0  0  0  0  0  0  0 2797 3361  933 94  6  0
 3 0 0 6684880 210888 2 106  0  0  0  0  0  0  0  0  0 2899 3679 1056 87 13  0
 3 0 0 6684880 210888 17 255 56 0  0  0  0  0  0  0  7 2926 3602 1297 89 11  0
 3 1 0 6683720 209816 25 255 80 0  0  0  0  0  0  0  9 2618 6652 1193 87 13  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2722 3500  943 89 10  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2732 3424  907 90 10  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2711 3553  973 89 10  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  1 2950 3529 1174 88 12  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2989 5437 1305 86 14  0
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2834 5519  923 93  7  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3004 3637 1062 90 10  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3013 3632 1177 88 11  0
 4 0 0 6684848 210880 1  52  0  0  0  0  0  8  8  8  1 3185 3821 1298 89 11  0
 3 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3025 5283 1361 92  8  0
 2 0 0 6684848 210880 4 214  0  0  0  0  0  0  0  0  0 2909 3656 1072 89 11  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2729 3387  946 90 10  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2748 3289  887 92  8  0
 2 0 0 6684848 210880 0   1  0  0  0  0  0 17 17 17 44 2786 2492  856 66 33  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2891 5141 1377 85 14  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2859 3408 1104 87 13  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2741 3346  916 91  9  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2717 3381  977 91  9  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2899 3359 1169 91  8  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3004 6185 1596 91  9  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2817 4003 1112 89 11  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2780 3355  950 93  7  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  5 2732 3281  952 91  9  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2849 3434 1066 93  7  0
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  1 3124 6069 1786 88 11  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2781 3432 1016 89 11  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2789 3483 1024 89 10  1
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  2 2720 3347  932 93  7  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2764 3317  944 91  9  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3141 6137 1888 86 14  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2750 3412  936 94  6  0
 3 0 0 6682944 209056 48 511 64 8  8  0  0  0  0  0  9 2498 4785  826 89 10  0
 2 0 0 6684848 210880 3  30 16  0  0  0  0  0  1  1  3 2579 3300  846 84 14  2
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3024 3645 1082 90 10  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3281 6243 1784 87 12  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2997 5717  978 89 11  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3163 3893 1057 87 12  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  3 3074 3833  996 88 12  0
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3127 3856 1024 87 11  2
 2 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3458 6582 1718 83 16  0
 1 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3037 3694 1121 90  9  1
 1 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3029 3882 1103 89 11  0
 0 0 0 6684848 210880 0   0  0  0  0  0  0  3  3  3  6 2644 2987  806 79 16  4
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 0 0 0 6684848 210880 0   0  0  0  0  0  0 14 14 14 38 2873 2862  813 84 15  1
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2962 4926 1597 87 11  2
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2717 3109  877 94  6  0
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2697 5082  868 89 10  0
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  5 2705 3006  849 86 13  1
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2670 3062  842 90  9  1
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2754 4862 1186 87 10  3
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2950 3281 1118 86 12  2
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3010 5001  970 86 11  3
 0 0 0 6684848 210880 45 411 40 8  8  0  0  0  0  0  9 2805 5455 1004 82 16  2
 1 0 0 6684376 210432 7 126 24  0  0  0  0  0  0  0  3 2678 3765  719 87 12  0
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3218 5373 1592 84 11  5
 1 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 3068 3573 1018 93  6  1
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2286 2305  430 46  7 48
 0 0 0 6684848 210880 1  53  0  0  0  0  0  0  0  0  0 2280 2393  387 48  5 47
 0 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2293 2288  401 47  3 50
 1 0 0 6684848 210880 0   0  0  0  0  0  0  0  0  0  0 2353 4079  971 48  4 49
 0 0 0 6683280 209376 57 548 64 0  0  0  0  0  0  0  8 2349 4559  610 54 11 35
 0 0 0 6684496 210576 0   0  0  0  0  0  0  0  0  0  0 1375 1321  458 19  4 76
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 s0   in   sy   cs us sy id
 0 0 0 6684496 210576 0   0  0  0  0  0  0  0  0  0  1  471  556  423  0  2 98
 0 0 0 6684496 210576 0   0  0  0  0  0  0  0  0  0  0  454  442  397  0  0 100


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]