R: [Linux-cluster] Why GFS is so slow? What it is waiting for?

Fri May 9 07:15:54 UTC 2008

Just remember to disable atime on the GFS volume. If atime is enabled maybe there is the lock contention for the writing of this info if multiple clients "read" the directory.

Leandro

> -----Messaggio originale-----
> Da: linux-cluster-bounces at redhat.com 
> [mailto:linux-cluster-bounces at redhat.com] Per conto di Ja S
> Inviato: venerdì 9 maggio 2008 8.38
> A: linux clustering
> Oggetto: Re: [Linux-cluster] Why GFS is so slow? What it is 
> waiting for?
> 
> Hi, Andrew:
> 
> Thank you very much for the help. Yes, your explanation 
> really makes sense. I buy it.
> 
> But I would like to discuss it a little bit further.
> The following message was part of my previous reply to Wendy. 
> Just paste it here for your convenience. 
> 
> 
> # stat abc/
>   File: `abc/'
>   Size: 8192            Blocks: 6024       IO Block:
> 4096   directory
> Device: fc00h/64512d    Inode: 1065226     Links: 2
> Access: (0770/drwxrwx---)  Uid: (    0/    root)  
> Gid: (    0/    root)
> Access: 2008-05-08 06:18:58.000000000 +0000
> Modify: 2008-04-15 03:02:24.000000000 +0000
> Change: 2008-04-15 07:11:52.000000000 +0000
> 
> # cd abc/
> # time ls | wc -l
> 31764
> 
> real    0m44.797s
> user    0m0.189s
> sys     0m2.276s
> 
> 
> >From the test results, it seems that the system really
> only used 2.276 seconds to perform the disk IO, read the 
> directory and count the number of files.
> 
> I am not sure whether I missed anything or not. I really 
> cannot understand how the system took about 42 seconds to 
> process the lock on the single directory.
> 
> Any further comments? 
> 
> Thanks again in advance,
> 
> Jas
> 
> 
> --- "Andrew A. Neuschwander" <andrew at ntsg.umt.edu>
> wrote:
> 
> > I've looked at this problem a bit as well. My system is a 
> 4Gb FC SAN 
> > with a bonded GigE DLM dedicated network. Stat'ing 30,000 
> files in 3 
> > minutes on GFS isn't unreasonable considering that it must get and 
> > release the gfs locks. In this scenario, you are averaging 
> about 6ms 
> > per file stat. When we did our tests, all of our subsystems 
> (FC, Net, 
> > CPU, Memory, Disk) were near idle. I think the 6ms is simply the 
> > accumulated latency of all the subsystems involved. There 
> is a lot of 
> > work happening in that short period of time.
> > 
> > -A
> > --
> > Andrew A. Neuschwander, RHCE
> > Linux Systems/Software Engineer
> > College of Forestry and Conservation
> > The University of Montana
> > http://www.ntsg.umt.edu
> > andrew at ntsg.umt.edu - 406.243.6310
> > 
> > 
> > On Thu, May 8, 2008 4:29 pm, Bob Peterson wrote:
> > > On Thu, 2008-05-08 at 14:27 -0700, Ja S wrote:
> > >> Hi, All:
> > >>
> > >> I used to post this question before, but have not received any 
> > >> comments yet. Please allow me post
> > it
> > >> again.
> > >>
> > >> I have a subdirectory containing more than 30,000 small 
> files on a 
> > >> SAN storage (GFS1+DLM, RAID10).
> > No
> > >> user application knows the existence of the 
> subdirectory. In other 
> > >> words, the subdirectory is
> > free
> > >> of accessing.
> > >>
> > >> However, it took ages to list the subdirectory on
> > an
> > >> absolute idle cluster node. See below:
> > >>
> > >> # time ls -la | wc -l
> > >> 31767
> > >>
> > >> real    3m5.249s
> > >> user    0m0.628s
> > >> sys     0m5.137s
> > >>
> > >> There are about 3 minutes spent on somewhere.
> > Does
> > >> anyone have any clue what the system was waiting
> > for?
> > >>
> > >>
> > >> Thanks for your time and wish to see your
> > valuable
> > >> comments soon.
> > >>
> > >> Jas
> > >
> > > Hi Jas,
> > >
> > > I believe the answer to your question is in the
> > FAQ:
> > >
> > >
> >
> http://sources.redhat.com/cluster/wiki/FAQ/GFS#gfs_slow
> > >
> > > Regards,
> > >
> > > Bob Peterson
> > > Red Hat Clustering & GFS
> > >
> > >
> > > --
> > > Linux-cluster mailing list
> > > Linux-cluster at redhat.com
> > >
> >
> https://www.redhat.com/mailman/listinfo/linux-cluster
> > >
> > >
> > 
> > --
> > Linux-cluster mailing list
> > Linux-cluster at redhat.com
> >
> https://www.redhat.com/mailman/listinfo/linux-cluster
> > 
> 
> 
> 
>       
> ______________________________________________________________
> ______________________
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile.  Try it now.  
> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
> 