<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3059" name=GENERATOR></HEAD>
<BODY>
<DIV><SPAN class=547161417-30032007>I'm trying to get some performance numbers
out of GFS2. Before describing the problem, I got to mention the previous
dlm_sendd/recvd spinning issue is no longer seen after moving to 2.6.21-rc4
kernel on FC6.</SPAN></DIV>
<DIV><SPAN class=547161417-30032007></SPAN> </DIV>
<DIV><SPAN class=547161417-30032007>I've a two node GFS2 setup sharing a disk
off node1 using GNBD. I'm running meta-data heavy tests (i.e create/read/delete
tons of small 8k files) from node2. The test kind of hangs in the middle. I
see the following log mesgs in node1,</SPAN></DIV>
<DIV><SPAN class=547161417-30032007></SPAN> </DIV>
<DIV><SPAN class=547161417-30032007>Mar 29 15:16:23 cfs1 gnbd_serv[2723]:
startup succeeded<BR>Mar 29 15:16:37 cfs1 gnbd_clusterd[2729]: connected<BR>Mar
29 15:17:15 cfs1 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm",
"ciscogfs2:sridhar"<BR>Mar 29 15:17:15 cfs1 kernel: dlm: connecting to 2<BR>Mar
29 15:17:15 cfs1 kernel: dlm: got connection from 2<BR>Mar 29 15:17:15 cfs1
kernel: GFS2: fsid=ciscogfs2:sridhar.1: Joined cluster. Now mounting
FS...<BR>Mar 29 15:17:16 cfs1 kernel: GFS2: fsid=ciscogfs2:sridhar.1: jid=1,
already locked for use<BR>Mar 29 15:17:16 cfs1 kernel: GFS2:
fsid=ciscogfs2:sridhar.1: jid=1: Looking at journal...<BR>Mar 29 15:17:16 cfs1
kernel: GFS2: fsid=ciscogfs2:sridhar.1: jid=1: Done<BR>Mar 30 09:58:34 cfs1
kernel: dlm: sridhar: remove fr 2 none<BR>Mar 30 09:58:34 cfs1 kernel: dlm:
message size 5457 from 2 too big, buf len 4632<BR>Mar 30 09:58:34 cfs1 kernel:
dlm: sridhar: remove fr 2 none<BR>Mar 30 09:58:35 cfs1 last message repeated 51
times<BR>Mar 30 09:58:35 cfs1 kernel: dlm: message size 13880 from 2 too big,
buf len 85072<BR>Mar 30 09:58:35 cfs1 kernel: dlm: sridhar: remove fr 2
none<BR>Mar 30 09:58:35 cfs1 last message repeated 3 times<BR>Mar 30 09:58:35
cfs1 kernel: dlm: message size 13880 from 2 too big, buf len 93760<BR>Mar 30
09:58:37 cfs1 kernel: dlm: message size 8224 from 2 too big, buf len
101136<BR>Mar 30 09:58:37 cfs1 kernel: dlm: message size 8224 from 2 too big,
buf len 101248<BR>Mar 30 09:58:39 cfs1 kernel: dlm: message size 8224 from 2 too
big, buf len 101472<BR></SPAN></DIV>
<DIV><SPAN class=547161417-30032007>sar/iostat shows there is no major
network/disk-io traffic going on after this problem. strace'ing any of the GFS
process hangs. Except I see tons of activity in 'aisexec' process with lots of
sendmsg/recvmsg going on. It seems some cluster level component - cman or dlm
error - causes GFS2 to lock up.</SPAN></DIV>
<DIV><SPAN class=547161417-30032007>
<DIV><SPAN class=547161417-30032007></SPAN> </DIV></SPAN></DIV>
<DIV><SPAN class=547161417-30032007>Previous tests with random file size (range
0 to 1MB) went thru' fine. But I also remember one of previous block test
(create/read/rw a1GB file) had similar problem.</SPAN></DIV>
<DIV><SPAN class=547161417-30032007>
<DIV><SPAN class=547161417-30032007></SPAN> </DIV>
<DIV><SPAN class=547161417-30032007>Anyone seen such a problem? Any clues to
resolve?</SPAN></DIV>
<DIV><SPAN class=547161417-30032007></SPAN> </DIV></SPAN></DIV>
<DIV><SPAN class=547161417-30032007>thanks,</SPAN></DIV>
<DIV><SPAN class=547161417-30032007>Sridhar</SPAN></DIV></BODY></HTML>