<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 12 (filtered medium)"> <style>  </style>  </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> We are using GFS to store session files for our web application. I've spent some time exploring GFS performance and tuning the software for optimal latency on system calls—we control the software, and the core libraries are written in C. So I've been following related discussions of e.g. stat() performance with a great deal of interest.<o:p></o:p> <o:p> </o:p> I hit a wall reducing latency of new file creation. Average create times are around 10ms and fluctuate from about 1ms up to 100ms or so. Here's an example:<o:p></o:p> <o:p> </o:p> open("/tb2/session/localhost/1800/ac18c/379/905bbc40.ts", O_WRONLY|O_CREAT|O_EXCL, 0660) = 4 <0.015415><o:p></o:p> <o:p> </o:p> The parent directory of this file (379) was created on this node. Our session storage ensures that no two nodes will attempt to create files in the same directory. I'm also limiting the number of directories we have to create so there is about a 50:1 ratio of files to directories (mkdir performance on GFS is generally awful).<o:p></o:p> <o:p> </o:p> Here's a breakdown of the most common system calls made from my test harness:<o:p></o:p> <o:p> </o:p> % time seconds usecs/call calls errors syscall<o:p></o:p> ------ ----------- ----------- --------- --------- ----------------<o:p></o:p> 91.26 0.046362 228 203 open<o:p></o:p> 7.87 0.003998 1999 2 mkdir<o:p></o:p> 0.59 0.000298 0 600 2 stat<o:p></o:p> 0.19 0.000098 0 200 write<o:p></o:p> 0.09 0.000045 0 302 read<o:p></o:p> 0.00 0.000000 0 202 close<o:p></o:p> <o:p> </o:p> Note this report doesn't show wall-clock time (I obtained it with strace –c). Roughly half the calls to open() are creating files, the rest open existing files.<o:p></o:p> <o:p> </o:p> My questions:<o:p></o:p> <o:p> </o:p> <![if !supportLists]>- <![endif]>What exactly happens during open()? I'm guessing that at least the journal is flushed to disk. Timings for open() are long and highly variable compared to other filesystems (e.g. ext3). The strace utility is limited to showing system calls from user space—it'd be interesting to see what I/O takes place in kernel space, but I don't have any way to do that (do I?). Am I network bound or I/O bound here? The latency looks suspiciously like disk seek times to me.<o:p></o:p> <![if !supportLists]>- <![endif]>Is there a strategy I can use to return more quickly from open() on a GFS filesystem?<o:p></o:p> <![if !supportLists]>- <![endif]>Before I spend time migrating to GFS2, is there any reason to believe GFS2 would perform significantly better here?<o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> <o:p> </o:p> Jeff<o:p></o:p> <o:p> </o:p> </div> </body> </html>