From lakshmipathi.g at gmail.com Sat Nov 21 05:12:34 2009 From: lakshmipathi.g at gmail.com (lakshmi pathi) Date: Sat, 21 Nov 2009 10:42:34 +0530 Subject: best file system performance analysis tool ? Message-ID: Hi , Could please redirect me to few of the best file system analysis tool available in open source community. I have googled out following url ,it has few tools , but not sure ,which tool to begin with? http://www.cse.wustl.edu/~jain/cse567-08/ftp/hw/index.html Thanks for your help in advance. -- Cheers, Lakshmipathi.G www.giis.co.in From pg_ext3 at ext3.for.sabi.co.UK Sat Nov 21 22:20:05 2009 From: pg_ext3 at ext3.for.sabi.co.UK (Peter Grandi) Date: Sat, 21 Nov 2009 22:20:05 +0000 Subject: best file system performance analysis tool ? In-Reply-To: References: Message-ID: <19208.26517.831309.239743@tree.ty.sabi.co.uk> > Hi , Could please redirect me to few of the best file system > analysis tool available in open source community. [ ... ] Most such tools are not very good, and anyhow it seems really difficult for most people who do storage performance testing to use them properly, as many "benchmarks" I see are mostly meaningless. However, used properly, FIO and Bonnie 1.4 can return useful results. For various reasons tools like Bonnie++ and IOzone are of dubious value. BTW, in general the first place where to look for GNU/Linux tools is the freshmeat.net database... From lm at bitmover.com Sun Nov 22 02:40:38 2009 From: lm at bitmover.com (Larry McVoy) Date: Sat, 21 Nov 2009 18:40:38 -0800 Subject: best file system performance analysis tool ? In-Reply-To: <19208.26517.831309.239743@tree.ty.sabi.co.uk> References: <19208.26517.831309.239743@tree.ty.sabi.co.uk> Message-ID: <20091122024038.GB3445@bitmover.com> On Sat, Nov 21, 2009 at 10:20:05PM +0000, Peter Grandi wrote: > > Hi , Could please redirect me to few of the best file system > > analysis tool available in open source community. [ ... ] > > Most such tools are not very good, and anyhow it seems really > difficult for most people who do storage performance testing to > use them properly, as many "benchmarks" I see are mostly > meaningless. Yup, I couldn't agree more. I dunno if this is helpful or not, but I used to do FS performance work and wrote a little tool called lmdd (I'm lm :). It's a lot like dd except it spits out perf numbers, does randoms, all sorts of stuff. At some point in the past most of the disk drive people were using it for testing. Part of lmbench, get that, compile it, then find bin/lmdd and play around. There is a man page in the test suite. Hang on, here ya go. Read the source, there are a few more interesting options. LMDD(8) LMBENCH LMDD(8) NAME lmdd - move io for performance and debugging tests SYNOPSIS lmdd [ option=value ] ... DESCRIPTION lmdd copies a specified input file to a specified output with possible conversions. This program is primarily useful for timing I/O since it prints out the timing statistics after completing. OPTIONS if=name Input file is taken from name; internal is the default. internal is a special file that acts like Sun's /dev/zero, i.e., it provides a buffer of zeros without doing a system call to get them. The following file names are taken to mean the standard input: -, 0, or stdin. of=name Output file is taken from name; internal is the default. internal is a special file that acts like /dev/null, without doing a system call to get rid of the data. The following file names are taken to mean the standard output: -, 1, or stdout. The following file names are taken to mean the standard error: 2, or stderr. bs=n Input and output block size n bytes (default 8192). Note that this is different from dd(1), it has a 512 byte default. Also note that the block size can be followed by 'k' or 'm' to indicate kilo bytes (*1024) or megabytes (*1024*1024), respectively. ipat=n If n is non zero, expect a known pattern in the file (see opat). Mismatches will be displayed as "ERROR: off=%d want=%x got=%x". The pattern is a sequence of 4 byte integers with the first 0, second 1, and so on. The default is not to check for the pattern. opat=n If n is non zero, generate a known pattern on the output stream. Used for debugging file system correctness. The default is not to generate the pattern. mismatch=n If n is non zero, stop at the first mismatched value. Used with ipat. skip=n Skip n input blocks before starting copy. fsync=n If n is non-zero, call fsync(2) on the output file before exiting or printing timing statistics. sync=n If n is non-zero, call sync(2) before exiting or print- ing timing statistics. rand=n This argument, by default off, turns on random behavior. The argument is not a flag, it is a size, that size is used as the upper bound for the seeks. Also note that the block size can be followed by 'k' or 'm' to indicate kilo bytes (*1024) or megabytes (*1024*1024), flush=n If n is non-zero and mmap(2) is available, call msync(2) to invalidate the output file. This flushes the file to disk so that you don't have unmount/mount. It is not as good as mount/unmount because it just flushes file pages - it misses the indirect blocks which are still cached. Not supported on all systems, compile time option. rusage=n If n is non-zero, print rusage statistics as well as timing statistics. Not supported on all systems, com- pile time option. count=n Copy only n input records. EXAMPLES This is the most common usage, the intent is to measure disk perfor- mance. The disk is a spare partition mounted on /spare. # mount /spare # lmdd if=internal of=/spare/XXX count=1000 fsync=1 7.81 MB in 3.78 seconds (2.0676 MB/sec) : Flush cache # umount /spare # mount /spare # lmdd if=/spare/XXX of=internal 7.81 MB in 2.83 seconds (2.7611 MB/sec) AUTHOR Larry McVoy, lm at sun.com (c)1994 Larry McVoy $Date$ LMDD(8) -- --- Larry McVoy lm at bitmover.com http://www.bitkeeper.com From lakshmipathi.g at gmail.com Thu Nov 26 10:06:53 2009 From: lakshmipathi.g at gmail.com (lakshmi pathi) Date: Thu, 26 Nov 2009 15:36:53 +0530 Subject: is it 4kb or 4.1kb - dd command? Message-ID: When I ran dd command , #dd if=/dev/zero of=dd.txt bs=4096 count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000445 seconds, 9.2 MB/s why it's shown as 4.1kB where as 4096 bytes is equal to 4KB? Thanks for your replies in advance. -- Cheers, Lakshmipathi.G www.giis.co.in From Ralf.Hildebrandt at charite.de Thu Nov 26 10:23:00 2009 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Thu, 26 Nov 2009 11:23:00 +0100 Subject: is it 4kb or 4.1kb - dd command? In-Reply-To: References: Message-ID: <20091126102300.GV12573@charite.de> * lakshmi pathi : > When I ran dd command , > #dd if=/dev/zero of=dd.txt bs=4096 count=1 > 1+0 records in > 1+0 records out > 4096 bytes (4.1 kB) copied, 0.000445 seconds, 9.2 MB/s > > why it's shown as 4.1kB where as 4096 bytes is equal to 4KB? 1) What does this have to do with ext3? 2) man dd says: BLOCKS and BYTES may be followed by the following multiplicative suffixes: c =1, w =2, b =512, kB =1000, K =1024, MB=1000*1000, M =1024*1024, xM =M GB =1000*1000*1000, G=1024*1024*1024, and so on for T, P, E, Z, Y. Thus: 1kb = 1000bytes -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From jcubedla at gmail.com Sun Nov 29 17:14:49 2009 From: jcubedla at gmail.com (John) Date: Sun, 29 Nov 2009 09:14:49 -0800 Subject: Effects of Missing ext3 Parameters Message-ID: Hi, I have two 3TB (hardware RAID-5) ext3 filesystems and recently added 1TB to each. I resized each filesystem and e2fsck -f reports that both are fine. However, when I look at the ext3 parameters with tune2fs -l one seems to have some parameters that the other one doesn't. In particular, one has: "Reserved GDT blocks", "Filesystem created", "Default directory hash" and "Directory Hash Seed" while the other one doesn't. Doing some Googling, I found out that "Reserved GDT blocks" is only important for online resizing. Since I don't do that, that's not an issue for me. I figure that "Filesystem created" is just a timestamp and is purely informational, so I think that's harmless if it's missing. But what about "Default directory hash" and "Directory Hash Seed"? What are the effects of either/both of those parameters missing? How can I set/change any/all of the missing parameters? Thanks for any help you can provide. John From tytso at mit.edu Sun Nov 29 18:32:31 2009 From: tytso at mit.edu (tytso at mit.edu) Date: Sun, 29 Nov 2009 13:32:31 -0500 Subject: Effects of Missing ext3 Parameters In-Reply-To: References: Message-ID: <20091129183231.GE6763@thunk.org> On Sun, Nov 29, 2009 at 09:14:49AM -0800, John wrote: > Hi, > > I have two 3TB (hardware RAID-5) ext3 filesystems and recently added > 1TB to each. I resized each filesystem and e2fsck -f reports that both > are fine. However, when I look at the ext3 parameters with tune2fs -l > one seems to have some parameters that the other one doesn't. In > particular, one has: "Reserved GDT blocks", "Filesystem created", > "Default directory hash" and "Directory Hash Seed" while the other one > doesn't. The difference is caused on the version of e2fsprogs used to create the file systems, and to a lesser extent perhaps on the version of e2fscks you currently have installed on the systems and which you used to resize the file systems. You can check out the differences by looking at the "Filesystem features" line in dumpe2fs. It might look like this for a newer ext3 file system: Filesystem features: ext_attr resize_inode dir_index filetype sparse_super Or for a brand-new ext4 file system on a Fedora 12 or Ubuntu 9.10 system it might look like this: Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize For some features, such as dir_index, kernels have supported it for a long time, and we were late in turning it on by default. In other cases, enabling a feature may require a specific kernel version. This might not be a problem if the file system was originally created on (for example) a RHEL 3 system, and you've since upgraded to RHEL 5. So the specifics very much depend on the version of the system that you are running on, and it's hard to give general advice that is broadly applicable. > Doing some Googling, I found out that "Reserved GDT blocks" is only > important for online resizing. Since I don't do that, that's not an > issue for me. I figure that "Filesystem created" is just a timestamp > and is purely informational, so I think that's harmless if it's > missing. Yes, all of the above is accurate. > But what about "Default directory hash" and "Directory Hash Seed"? > What are the effects of either/both of those parameters missing? How > can I set/change any/all of the missing parameters? The "default directory hash" and "directory hash seed" are associated with the dir_index feature. This feature gives faster random access to a very large directories in cold cache situations. It can slow down some workloads that do readdir/stat, however, unless the application is smart enough to sort the returned results from readdir by inode number before accessing the files. Regards, - Ted