<div dir="ltr">Hi,<div style><br>I figured I'd try to solicit the kind help of the mailing list again on this as I continue to have issues with XFS and RHEL6. For example, I have a 12 TB software RAID5 filesystem on a LSI <span style="font-family:arial,sans-serif;font-size:13px">92118 and the drives are 3 TB Seagate Barracuda ST3000DM001. This filesystem currently has around 140 million files with many of them smaller than 50 KB. This system is running a fully patched RHEL6.4</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style><span style="font-family:arial,sans-serif;font-size:13px">Within this filesystem, I have a one particular tree of files I need to remove. There are ~170 folders with around 4-10 sub-folders each and about 1,000 files in each of those sub-folders. Most files are less than 40KB. Attempting to list out one of those top level folders like so:</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div style><span style="font-family:arial,sans-serif">ls -R * | wc -l</span><br></div><div style><font face="arial, sans-serif"><br></font></div>
<div style><font face="arial, sans-serif">takes 50 seconds and wc reports 3825 lines (~files). Watching iostat during this operation, the tps value pokes along around 100 to 150 tps. This filesystem is doing other things at the time as well. Just running iostat without args currently reports:</font></div>
<div style><font face="arial, sans-serif"><br></font></div><div style><font face="arial, sans-serif"><div>avg-cpu: %user %nice %system %iowait %steal %idle</div><div> 11.12 0.03 2.70 3.60 0.00 82.56</div>
<div><br></div><div>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn</div><div>md127 134.36 10336.87 11381.45 19674692141 21662893316<br></div><div><br></div><div style>If I go into one of these folders with 1,000 files or so in it and attempt to list out the directory cold, it takes 10-15 seconds. Attempting to remove one of the top level folders takes a long time and the other filesystem operations at the time feel very sluggish as well.</div>
<div style><br></div><div style>$ time rm -rf myfolder (this is around 4,000 files total within 6 subfolders of myfolder)<br></div></font></div><div style><font face="arial, sans-serif"><div><br></div><div>real<span class="" style="white-space:pre"> </span>2m36.925s</div>
<div>user<span class="" style="white-space:pre"> </span>0m0.018s</div><div>sys<span class="" style="white-space:pre"> </span>0m0.657s</div><div><br></div><div style>Running hdparm on one of the software raid5 drives reports decent numbers.</div>
<div style><br></div><div style><div>/dev/sdb:</div><div> Timing cached reads: 12882 MB in 2.00 seconds = 6448.13 MB/sec</div><div> Timing buffered disk reads: 396 MB in 3.06 seconds = 129.39 MB/sec</div><div><br></div>
<div style>running some crude dd tests shows reasonable numbers, I think.</div><div style><br></div><div style><div># dd bs=1M count=1280 if=/dev/zero of=test conv=fdatasync</div><div>1342177280 bytes (1.3 GB) copied, 29.389 s, 45.7 MB/s<br>
</div></div></div><div><br></div><div style>I have other similiar filesystems on ext4 with similiar hardware and millions of small files as well. I don't see such sluggishness with small files and directories there. I guess I picked XFS for this filesystem initially because of its fast fsck times.</div>
<div><br></div><div style>Here are some more details on the filesystem</div><div style><br></div><div style><div style="font-size:13px"><div># xfs_info /dev/md127</div><div>meta-data=/dev/md127 isize=256 agcount=32, agsize=91570816 blks</div>
<div> = sectsz=512 attr=2, projid32bit=0</div><div>data = bsize=4096 blocks=2930265088, imaxpct=5</div><div> = sunit=128 swidth=512 blks</div>
<div>naming =version 2 bsize=4096 ascii-ci=0</div><div>log =internal bsize=4096 blocks=521728, version=2</div><div> = sectsz=512 sunit=8 blks, lazy-count=1</div>
<div>realtime =none extsz=4096 blocks=0, rtextents=0</div></div><div style="font-size:13px"><br></div><div style="font-size:13px"><div># grep md127 /proc/mounts </div><div>/dev/md127 /mesonet xfs rw,noatime,attr2,delaylog,sunit=1024,swidth=4096,noquota 0 0</div>
</div><div style="font-size:13px"><br></div><div style="font-size:13px"><div># cat /proc/mdstat </div><div>Personalities : [raid6] [raid5] [raid4] </div><div>md127 : active raid5 sdf[3] sde[0] sdd[5] sdc[1] sdb[2]</div><div>
11721060352 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]</div></div></div><div><br></div><div style>Anybody have ideas? or are these still known issues with XFS on RHEL as noted here:</div><div style>
<br></div><div style><a href="http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf">http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf</a></div>
<div><br></div><div style>thanks</div><div style>daryl</div><div><br></div></font></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 22, 2012 at 7:45 PM, Daryl Herzmann <span dir="ltr"><<a href="mailto:akrherz@iastate.edu" target="_blank">akrherz@iastate.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Tue, Jun 5, 2012 at 3:10 PM, Jussi Silvennoinen<br>
<<a href="mailto:jussi_rhel6@silvennoinen.net">jussi_rhel6@silvennoinen.net</a>> wrote:<br>
>> I've been noticing lots of annoying problems with XFS performance with<br>
>> RHEL6.2 on 64bit. I typically have 20-30 TB file systems with data<br>
>> structured in directories based on day of year, product type, for example,<br>
>><br>
>> /data/2012/06/05/product/blah.gif<br>
>><br>
>> Doing operations like tar or rm over these directories bring the system to<br>
>> a grinding halt. Load average goes vertical and eventually the power button<br>
>> needs to be pressed in many cases :( A hack workaround is to break apart the<br>
>> task into smaller chunks and let the system breath in between operations...<br>
>><br>
>> Anyway, I read Ric Wheeler's "Billion Files" with great interest<br>
>><br>
>><br>
>> <a href="http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf" target="_blank">http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf</a><br>
>><br>
>> It appears there are 'known issues' with XFS and RHEL6.1. It does not<br>
>> appear these issues were addressed in RHEL 6.2?<br>
>><br>
>> Does anybody know if these issues were addressed in the upcoming RHEL 6.3?<br>
>> My impression is that upstream fixes for this only recently (last 6 months?)<br>
>> appeared in the mainline kernel.<br>
>><br>
>> Perhaps I am missing some tuning that could be done to help with this?<br>
><br>
><br>
> Enabling lazy-count does wonders for workloads that involve massive amounts<br>
> of metadata. Unfortunately it's a mkfs-time option only AFAIK.<br>
<br>
</div>Thanks, but it was already enabled...<br>
<span class="HOEnZb"><font color="#888888"><br>
daryl<br>
</font></span></blockquote></div><br></div>