[Cluster-devel] Re: Why the gfs2 performance regressed?
Steven Whitehouse
swhiteho at redhat.com
Wed Jan 2 10:19:28 UTC 2008
Hi,
On Wed, 2008-01-02 at 15:53 +0800, Cheng Renquan wrote:
> hello, Steven:
>
> I have tested gfs2 in the out-of-box RHEL51 and in the latest gfs2-nmw
> git repository, finding that the latest gfs2 performance regressed.
>
> the testing environment is RHEL5.1, default kernel + gfs2 + samba, I
> use the preinstalled samba-3.0.25b to share a gfs2 mouted direcotry,
> on Windows clients I've tested its performance is very good! it could
> be up to 100MB/s on writing, almost up to the limit of the local
> Gigabyte ether network,
>
> Now I want to test the latest gfs2 from gfs2-nmw.git repository, I
> just compiled the gfs2-nmw kernel to replace the default, using the
> .config from /boot/config-$(uname -r)
>
> But I found the performance is bad:
>
> Writers on Windows can only up to 6 MB/s, at the time I noticed on the linux:
>
> 1. from this I know the smbd thread 3627 is serving the Windows client
> 192.168.76.226:
> # lsof -nPi4
> smbd 3627 fstest 22u IPv4 7806 TCP
> 192.168.76.200:445->192.168.76.226:1369 (ESTABLISHED)
>
> 2. list all opening files of thread 3627, finding that /mnt/gfs2/x.dat
> is currently accessed by Windows client testing utilities through
> samba:
> # lsof -nP -a -p 3627
> smbd 3627 fstest 18uR REG 253,0 328704000 66392 /mnt/gfs2/x.dat
> smbd 3627 fstest 22u IPv4 7806 TCP
> 192.168.76.200:445->192.168.76.226:1369 (ESTABLISHED)
>
> 3. strace it and record every system call time, output the results to a file:
> # strace -T -p 3627 -o fcntl64.3267
>
> 4. sometime later interrupt it, and analysis the result.
>
> 5. from the result, I found some fcntl64 system call consumes 0.9 second:
> fcntl64(18, F_GETLK64, {type=F_UNLCK, whence=SEEK_SET,
> start=325263360, len=65536, pid=0}) = 0 <0.915883>
> fcntl64(18, F_GETLK64, {type=F_UNLCK, whence=SEEK_SET, start=22093824,
> len=65536, pid=0}) = 0 <0.916389>
> ...
>
> 6. if straced thread 3627 without output redirected, the fcntl64 will
> appear as an apparent pause.
>
> 7. Since the default-kernel+gfs2+samba serves high efficiently, and
> the latest gfs2-nmw does not, this should be some problem in gfs2-nmw?
>
> 8. I noted sometimes the thread 3627 will becomes D state
> (uninterruptible), I use /proc/sysrq-trigger to record the call trace
> in the kernel space:
> smbd D 00000000 2080 3627 3620
> 00000000 00200082 00000001 00000000 c042148e 00000000 19b1e094 000002b6
> f68f92c0 f68f94f0 c3616d80 00000000 f6dda040 000004ab 00000000 00000003
> f605ae9c f8c54340 c043bc87 f34fa0c0 f605af14 f605ae9c f488fe00 f8c4fbea
> Call Trace:
> [<c042148e>] __wake_up_common+0x32/0x5c
> [<c043bc87>] prepare_to_wait+0x24/0x3f
> [<f8c4fbea>] gdlm_plock_get+0xeb/0x16b [lock_dlm]
> [<c043bb3d>] autoremove_wake_function+0x0/0x35
> [<f8c4faff>] gdlm_plock_get+0x0/0x16b [lock_dlm]
> [<f8b7b5d5>] gfs2_lm_plock_get+0x30/0x39 [gfs2]
> [<f8b81003>] gfs2_lock+0x78/0xb5 [gfs2]
> [<f8b80f8b>] gfs2_lock+0x0/0xb5 [gfs2]
> [<c0480a51>] vfs_test_lock+0x18/0x23
> [<c0480d95>] fcntl_getlk64+0x5f/0x108
> [<c047e54b>] sys_fcntl64+0x38/0x6d
> [<c0404dde>] sysenter_past_esp+0x5f/0x85
>
> 9. Someone could tell me if there is a problem of gfs2-nmw or not? Or
> the configuration of mine has problems?
>
> the two attachments are the results of strace and dmesg, fcntl64.3267
> and dmesg.1
>
Are you running single node? if so then use lock_nolock rather than
lock_dlm as it will be much faster for fcntl locks. Even if you intend
to run as a cluster eventually, a single node comparison against
lock_nolock would be useful to try and eliminate some possibilities,
Steve.
More information about the Cluster-devel
mailing list