From lakshmipathi.g at gmail.com Mon Jun 13 06:56:39 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Mon, 13 Jun 2011 12:26:39 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool Message-ID: Hi - Updated my old project named "ext3carve" and renamed it as "extcarve" . It uses libext2fs. (To be precise,re-uses on 'debugfs' command's "dump_unused" feature) In summary,the tool will do the following - It will scan the linux machine ,for unused/deleted blocks and search for magic signatures. If it finds valid signature (both header and footer) It saves the file at given external drive. Now it can recover- non-fragmented (like png,jpg,gif,html,c/cpp/php,pdf files) deleted files.One main advantage is that it opens the affected partitions on read-only mode,thus no changes made to affected disk.Disk remains the same - regardless of whether extcarve recovers them or not. Simply usage would be - 1. Copy extcarve binary to Pen drive. 2. Plug-in the pen-drive to affected system.(the system from where you want to recover files) 3.Attach an external harddrive to affected system so that recovered files will be stored on external hdd. 3. Run the extcarve from within pendrive - Provide affected drive as input and external drive as output directory. Checkout recovered files at external hdd. Download url - www.giis.co.in/giis Any feedbacks/comments are welcome. -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmaiolino at maiolino.org Mon Jun 13 15:52:48 2011 From: cmaiolino at maiolino.org (Carlos Maiolino) Date: Mon, 13 Jun 2011 12:52:48 -0300 Subject: Allocation of Indirect Blocks In-Reply-To: <4DB5F29D.7000506@nasa.gov> References: <4DB5F099.8010206@nasa.gov> <4DB5F1F7.2010808@redhat.com> <4DB5F29D.7000506@nasa.gov> Message-ID: <20110613155248.GA2308@pegasus.maiolino.org> Hi Sean, > Eric Sandeen wrote: > >On 4/25/11 5:07 PM, Sean McCauliff wrote: > >>Does ext3 allocate indirect blocks as needed or is there some fixed number of these like inodes? Should I be concerned with running out of indirect blocks? > >ext3 allocates them as needed. > > > >In fact you will often see them allocated consecutively with the data blocks they refer to: > > Not sure if it's an useful comment, but even that ext2/3 uses indirect blocks as needed and you need not care about it while writing to the FS, you'll still need to be careful about the maximum file size, which can be up to 2TiB using default 4k blocks iirc. Cheers -- -Carlos From jidong.xiao at gmail.com Tue Jun 14 05:19:29 2011 From: jidong.xiao at gmail.com (Jidong Xiao) Date: Tue, 14 Jun 2011 01:19:29 -0400 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Mon, Jun 13, 2011 at 2:56 AM, Lakshmipathi.G wrote: > Hi - > Updated my old project named "ext3carve" and renamed it as "extcarve" . It > uses libext2fs. (To be precise,re-uses on 'debugfs' command's "dump_unused" > feature) > > In summary,the tool will do the following -?? It will scan the linux machine > ,for unused/deleted blocks and search for magic signatures. If it finds > valid signature (both header and footer) It saves the file at given external > drive. > > Now it can recover- non-fragmented (like png,jpg,gif,html,c/cpp/php,pdf > files) deleted files.One main advantage is that it opens the affected > partitions on read-only mode,thus no changes made to affected disk.Disk > remains the same - regardless of whether extcarve recovers them or not. > > Simply usage would be - > 1. Copy extcarve binary to Pen drive. > 2. Plug-in the pen-drive to affected system.(the system from where you want > to recover files) > 3.Attach an external harddrive to affected system so that recovered files > will be stored on external hdd. > 3. Run the extcarve from within pendrive - Provide affected drive as input > and external drive as output directory. > Checkout recovered files at external hdd. > > Download url - www.giis.co.in/giis > Any feedbacks/comments are welcome. > What are the pros and cons when compared to ext3grep and extundelete? In addition, what is the Pen drive? I mean, since we need to attach an external harddrive, why don't we run the command within the harddrive? Regards Jidong From lakshmipathi.g at gmail.com Tue Jun 14 06:11:27 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Tue, 14 Jun 2011 11:41:27 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: What are the pros and cons when compared to ext3grep and extundelete? > > Carlo wood's ext3grep uses Journal entries to recover the files. If I'm not wrong extundelete ,is an extension of ext3grep which supports ext4. If Journal entries are lost or overwritten,It would be difficult for them to recover. (Please correct me,If I'm wrong :D ) extcarve doesn't depend on journal entries - It scans the disk for valid magic signature of a file and tries to recover them. I think extcarve similar to tool like foremost http://foremost.sourceforge.net/ > In addition, what is the Pen drive? I mean, since we need to attach an > external harddrive, why don't we run the command within the harddrive? > > I was just extra-careful, I have seen some new users installs the recover software on the affected partition itself,which is not a good thing (The recover software may overwrite the file,user wants to recover) .Yes,you can install the command on external hard-drive and use it. > Regards > Jidong > -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From jidong.xiao at gmail.com Tue Jun 14 06:41:05 2011 From: jidong.xiao at gmail.com (Jidong Xiao) Date: Tue, 14 Jun 2011 02:41:05 -0400 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Tue, Jun 14, 2011 at 2:11 AM, Lakshmipathi.G wrote: > >> >> What are the pros and cons when compared to ext3grep and extundelete? >> > Carlo wood's ext3grep uses Journal entries to recover the files. If I'm not > wrong extundelete ,is an extension of ext3grep which supports ext4. > If Journal entries are lost or overwritten,It would be difficult for them to > recover. (Please correct me,If I'm wrong :D ) > > extcarve doesn't depend on journal entries - It scans the disk for valid > magic signature of a file and tries to recover them. > I think extcarve? similar to tool like foremost > http://foremost.sourceforge.net/ > > Good, I tested these three tools on one of my disk, which I deleted many files last week, the results shows: ext3grep can list what I deleted, but cannot recover extundelete can recover part of my files extcarve, fails to recover any of my files, it is keeping print messages like this: no header found.at all Searching Unused block 415494 which contains non-zero data: no header found.at all Searching Unused block 415495 which contains non-zero data: no header found.at all Searching Unused block 415496 which contains non-zero data: no header found.at all Searching Unused block 415497 which contains non-zero data: no header found.at all Searching Unused block 415498 which contains non-zero data: no header found.at all Searching Unused block 415499 which contains non-zero data: no header found.at all Searching Unused block 415500 which contains non-zero data: no header found.at all Searching Unused block 415501 which contains non-zero data: no header found.at all Searching Unused block 415502 which contains non-zero data: no header found.at all Searching Unused block 415503 which contains non-zero data: no header found.at all Searching Unused block 415504 which contains non-zero data: no header found.at all Searching Unused block 415505 which contains non-zero data: no header found.at all Searching Unused block 415506 which contains non-zero data: no header found.at all Searching Unused block 415507 which contains non-zero data: no header found.at all Searching Unused block 415508 which contains non-zero data: no header found.at all Searching Unused block 415509 which contains non-zero data: I saw almost a million of such lines. >> >> In addition, what is the Pen drive? I mean, since we need to attach an >> external harddrive, why don't we run the command within the harddrive? >> > > I was just extra-careful, I have seen some new users installs the recover > software on the affected partition itself,which is not a good thing (The > recover software may overwrite the file,user wants to recover) .Yes,you can > install the command on external hard-drive and use it. > > Okay, I see, thank you. Jidong From lakshmipathi.g at gmail.com Tue Jun 14 07:18:13 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Tue, 14 Jun 2011 12:48:13 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: > extcarve, fails to recover any of my files, it is keeping print > messages like this: > > no header found.at all > Searching Unused block 415494 which contains non-zero data: > > no header found.at all > Searching Unused block 415495 which contains non-zero data: > > I saw almost a million of such lines. > It will scan for unused/deleted blocks (415494, 415495 etc). While scanning It throws out such output messages. When I finds valid header+footer it saves them on external drive. As of now, extcarve can recover only limited non-fragmented files of type png,jpg,gif,html,c/cpp/php,pdf .(recoverd filename will begin with extcarve.extenstion) current version of tool ,will not recover a partial files - It needs to be find both header and footer. extcarve it still running or completed ? If its completed and didn't recover any files ,then possible reason would be - 1).Deleted files are fragmented,so extcarve can't find footer. 2).Deleted files doesn't belong to currently supported file types. 3).Or simply extcarve has a bug,that needs to fixed. :P -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From jidong.xiao at gmail.com Tue Jun 14 15:10:45 2011 From: jidong.xiao at gmail.com (Jidong Xiao) Date: Tue, 14 Jun 2011 11:10:45 -0400 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Tue, Jun 14, 2011 at 3:18 AM, Lakshmipathi.G wrote: > >> extcarve, fails to recover any of my files, it is keeping print >> messages like this: >> >> no header found.at all >> Searching Unused block 415494 which contains non-zero data: >> >> no header found.at all >> Searching Unused block 415495 which contains non-zero data: >> >> I saw almost a million of such lines. > > It will scan for unused/deleted blocks (415494, 415495 etc). While scanning > It throws out such? output? messages. > When I finds valid header+footer it saves them on external drive. As of now, > extcarve can recover? only limited non-fragmented files of type > png,jpg,gif,html,c/cpp/php,pdf .(recoverd filename will begin with > extcarve.extenstion) > > current version of tool ,will not recover a partial files - It needs to be > find both header and footer. > > extcarve it still running or completed ??? If its completed and didn't > recover any files ,then possible reason would be - > 1).Deleted files are fragmented,so extcarve can't find footer. > 2).Deleted files doesn't belong to currently supported file types. > 3).Or simply extcarve has a bug,that needs to fixed. :P > > > It looks like extcarve is completed. Following is the last lines of output I can see on the screen, and the program stalled there and not generate any more outputs. no header found.at all Searching Unused block 20353513 which contains non-zero data: no header found.at all Searching Unused block 20353514 which contains non-zero data: no header found.at all Searching Unused block 20353515 which contains non-zero data: no header found.at all Searching Unused block 20353516 which contains non-zero data: n ================================================ It did generate some output file under the output directory. However, none of these files are the original files. About 196M bytes files are generated, however, every single file is of the same size, 4KB. There were four types of files, cpp/png/gif/pdf. I tried to open the pdf files, it could be open, but the contents are not what I expected, there are just some weird contents in the pdf file. Actually how can a normal pdf be only 4KB? Regards Jidong From lakshmipathi.g at gmail.com Tue Jun 14 15:38:24 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Tue, 14 Jun 2011 21:08:24 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: It did generate some output file under the output directory. However, none of these files are the original files. About 196M bytes files are generated, however, every single file is of the same size, 4KB. That's strange,since while testing it i have seen files upto 274KB. were four types of files, cpp/png/gif/pdf. I tried to open the pdf files, it could be open, but the contents are not what I expected, there are just some weird contents in the pdf file. Actually how can a normal pdf be only 4KB? I find it difficult to understand why all files are 4KB? What was the command that you have used to invoke extcarve? What's your file system's default block size? (Use command : tune2fs -l /dev/ | grep "Block size") I have tested with ext3,ext4 with 4KB as block size. > Regards > Jidong > -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From jidong.xiao at gmail.com Tue Jun 14 17:06:15 2011 From: jidong.xiao at gmail.com (Jidong Xiao) Date: Tue, 14 Jun 2011 13:06:15 -0400 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Tue, Jun 14, 2011 at 11:38 AM, Lakshmipathi.G wrote: > It did generate some output file under the output directory. However, > none of these files are the original files. About 196M bytes files are > generated, however, every single file is of the same size, 4KB. > That's strange,since ?while testing it i have seen files upto 274KB. > > were four types of files, cpp/png/gif/pdf. I tried to open the pdf > files, it could be open, but the contents are not what I expected, > there are just some weird contents in the pdf file. Actually how can a > normal pdf be only 4KB? > I find it difficult to understand why all files are 4KB? What ?was the > command that you have used to invoke extcarve? > What's your file system's default block size? (Use command : tune2fs -l > /dev/ ?| grep "Block size") I have tested > with ext3,ext4 with 4KB as block size. >> I was using "extcarve -g". Here are the information you asked: test # tune2fs -l /dev/sda3 tune2fs 1.41.14 (22-Dec-2010) Filesystem volume name: SCRATCH Last mounted on: Filesystem UUID: fe1d144c-7dac-47bf-b28d-c16aa591a1b5 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 28229632 Block count: 112916868 Reserved block count: 5645843 Free blocks: 111069127 Free inodes: 28205356 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 997 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Filesystem created: Thu Sep 24 07:08:16 2009 Last mount time: Mon Jun 13 14:59:06 2011 Last write time: Mon Jun 13 18:06:10 2011 Mount count: 4 Maximum mount count: 28 Last checked: Thu Jun 2 19:32:20 2011 Check interval: 15552000 (6 months) Next check after: Tue Nov 29 18:32:20 2011 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 254f5863-3bf9-4ae9-ab99-9c433bce0aa8 Journal backup: inode blocks Jidong From lakshmipathi.g at gmail.com Tue Jun 14 18:05:27 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Tue, 14 Jun 2011 23:35:27 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: > Block size: 4096 Yes,your block size same as my test machine -thats fine (4KB should work fine).I can recommend you to use option "-i" but the option "-g" should recover files upto 48KB(but we are getting only 4KB) . So using option "-i" won't make any difference here. Files with 4KB size are possible only when extcarve finds header and footer on a single block.Other this, I'm running out of thoughts. I'll try with different data set (test in different environments) and see whether I can reproduce this issue. > > Jidong > -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at dilger.ca Tue Jun 14 20:34:27 2011 From: adilger at dilger.ca (Andreas Dilger) Date: Tue, 14 Jun 2011 14:34:27 -0600 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On 2011-06-14, at 12:05 PM, Lakshmipathi.G wrote: >> Block size: 4096 > > Yes,your block size same as my test machine -thats fine (4KB should work fine).I can recommend you to use option "-i" but the option "-g" should recover files upto 48KB(but we are getting only 4KB) . So using option "-i" won't make any difference here. > > Files with 4KB size are possible only when extcarve finds header and footer on a single block.Other this, I'm running out of thoughts. I'll try with different data set (test in different environments) and see whether I can reproduce this issue. Since ext3/4 work hard to allocate file blocks contiguously, it makes sense to assume that after the header blocks the file blocks will be present. While this is less likely to be true as the file size increases, for ext4 it is usually true for files smaller than a few MB, which is many of them. Cheers, Andreas From lakshmipathi.g at gmail.com Wed Jun 15 04:20:59 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Wed, 15 Jun 2011 09:50:59 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Wed, Jun 15, 2011 at 2:04 AM, Andreas Dilger wrote: > > Since ext3/4 work hard to allocate file blocks contiguously, it makes sense > to > assume that after the header blocks the file blocks will be present. While > this is less likely to be true as the file size increases, for ext4 it is > usually true for files smaller than a few MB, which is many of them. > Thanks for the clarification. On ext4,I believe an extent can address upto 128MB. So I assume, for a file of size 50MB ,there is possibility this will be stored contiguously. Is that correct assumption? and If we mounted ext3 as ext4, will the inode become extent based rather than the old direct/indirect block accessing method? > > Cheers, Andreas > > -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From jidong.xiao at gmail.com Wed Jun 15 13:19:34 2011 From: jidong.xiao at gmail.com (Jidong Xiao) Date: Wed, 15 Jun 2011 09:19:34 -0400 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Wed, Jun 15, 2011 at 12:20 AM, Lakshmipathi.G wrote: > > On Wed, Jun 15, 2011 at 2:04 AM, Andreas Dilger wrote: >> >> Since ext3/4 work hard to allocate file blocks contiguously, it makes >> sense to >> assume that after the header blocks the file blocks will be present. >> ?While >> this is less likely to be true as the file size increases, for ext4 it is >> usually true for files smaller than a few MB, which is many of them. > > Thanks for the clarification. On ext4,I believe an extent can address upto > 128MB. ?So I assume, for a file of size 50MB ,there is possibility this will > be stored contiguously. Is that correct assumption? and ?If we ?mounted ext3 > as ext4, will the inode become extent based rather than the old > direct/indirect block accessing method? > My file system is ext3. In addition, my partition size is very large, it is 400GB. Is that a problem? I tried extcarve at two machines, both got the same results, i.e., only generated some 4KB non-sense files there. Jidong From lakshmipathi.g at gmail.com Wed Jun 15 13:56:59 2011 From: lakshmipathi.g at gmail.com (Lakshmipathi.G) Date: Wed, 15 Jun 2011 19:26:59 +0530 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Wed, Jun 15, 2011 at 6:49 PM, Jidong Xiao wrote: > My file system is ext3. > > In addition, my partition size is very large, it is 400GB. Is that a > problem? > > I tried extcarve at two machines, both got the same results, i.e., > only generated some 4KB non-sense files there. > > ext3 shouldn't be a problem. Give a try with "-i" option, for the third input provide 256 as input. So that , It'll try upto 1MB files. ---- Please enter increased block limit:(default is 12) :256 ---- Does that produce files greater than 4KB size ? > Jidong > -- ---- Cheers, Lakshmipathi.G FOSS Programmer. www.giis.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From jidong.xiao at gmail.com Wed Jun 15 15:42:33 2011 From: jidong.xiao at gmail.com (Jidong Xiao) Date: Wed, 15 Jun 2011 11:42:33 -0400 Subject: extcarve - ext2,ext3,ext4 file carving tool In-Reply-To: References: Message-ID: On Wed, Jun 15, 2011 at 9:56 AM, Lakshmipathi.G wrote: > > > On Wed, Jun 15, 2011 at 6:49 PM, Jidong Xiao wrote: >> >> My file system is ext3. >> >> In addition, my partition size is very large, it is 400GB. Is that a >> problem? >> >> I tried extcarve at two machines, both got the same results, i.e., >> only generated some 4KB non-sense files there. >> > > ext3 shouldn't be a problem. Give a try with "-i" option,? for the third > input provide 256 as input. So that , It'll try upto 1MB files. > ---- > Please enter increased block limit:(default is 12) :256 > ---- > Does that produce files greater than 4KB size ? > > Yes by using -i and 256 as the third input, I can generate a few files whose size is large than 4KB. Here are the results: Most of the files are still 4KB and not meaningful, some files whose size are 8KB, but still not meaningful, i.e., only some unknown and strange contents in those files, only those pdf files whose size are larger than 8KB are real files, they are useful files. Since my partition size is 400GB, I should have far more files than here, but at two of my machines, extcarve stalled at some step, it seems the process is still there but it stops generating more files. jxiao at th107c-6:~> ls -l /mnt/usbstick/RECARVED_FILES/ total 384 -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarve1RITdT.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarve1s8oaZ.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarve2lIpqA.cpp -rwxrwxrwx 1 root root 28672 Jun 15 11:13 extcarve2vm7gW.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarve3JXkZD.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarve7FI0YM.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarve7kADgL.cpp -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarve8wouIy.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvebqkCDK.cpp -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarveCbKucU.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:07 extcarveCGxSgK.png -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarvecmUimP.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:08 extcarved8HC0z.png -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveD9ZSjI.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveDO7CIl.cpp -rwxrwxrwx 1 root root 36864 Jun 15 11:13 extcarvedqoMd4.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvedWuCvQ.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:08 extcarveeCrRf1.png -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveEfGMQZ.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveGdJ8hw.cpp -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarvegf35Oa.pdf -rwxrwxrwx 1 root root 24576 Jun 15 11:13 extcarvegXLHGf.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveGZenG1.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvehCFJFW.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveiYO3Jp.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveJkY8r1.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveKyRrks.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:08 extcarvemPFqvs.png -rwxrwxrwx 1 root root 32768 Jun 15 11:13 extcarvenAtFWu.pdf -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarveNiJHTt.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveoKb9bp.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveotikLR.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveP194o9.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvep9q2Oe.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveqiQFRN.cpp -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarveqmfEed.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveRF2d55.cpp -rwxrwxrwx 1 root root 28672 Jun 15 11:13 extcarvesKXZwr.pdf -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvesuUtai.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveTDXWSm.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveTQV47h.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveuUmoN6.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveVLW6aB.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvew73gEx.cpp -rwxrwxrwx 1 root root 12288 Jun 15 11:13 extcarvew9bYbn.jpg -rwxrwxrwx 1 root root 8192 Jun 15 11:08 extcarvewEpAGI.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveweQmFD.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvexamR4b.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveY4OWK3.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvey5zhr8.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:08 extcarveYiwhLT.cpp -rwxrwxrwx 1 root root 8192 Jun 15 11:13 extcarvezC2MHI.jpg -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarvezGrAFk.cpp -rwxrwxrwx 1 root root 4096 Jun 15 11:13 extcarveZppUaG.cpp -Jidong From alan at popey.com Fri Jun 24 09:48:51 2011 From: alan at popey.com (Alan Pope) Date: Fri, 24 Jun 2011 10:48:51 +0100 Subject: How long should resize2fs take? Message-ID: Hullo! First mail, sorry if this is the wrong place for this kind of question. I realise this is a "piece of string" type question. tl;dr version: I have a resizefs shrinking an ext4 filesystem from ~4TB to ~3TB and it's been running for ~2 days. Is this normal? Strace shows lots of:- lseek(3, 42978250752, SEEK_SET) = 42978250752 read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 lseek(3, 4675072176128, SEEK_SET) = 4675072176128 read(3, "\355A\350\3\0\20\0\0009\271\371M\206\177\0N~\206\33M\0\0\0\0\350\3\2\0\10\0\0\0"..., 4096) = 4096 lseek(3, 42978254848, SEEK_SET) = 42978254848 read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096 lseek(3, 42978238464, SEEK_SET) = 42978238464 write(3, "\355A\350\3\0\20\0\0009\271\371M\364Y\4N\200\206\33M\0\0\0\0\350\3\2\0\10\0\0\0"..., 4096) = 4096 lseek(3, 4675072180224, SEEK_SET) = 4675072180224 read(3, "\355A\350\3\0\20\0\0009\271\371M\206\177\0N~\206\33M\0\0\0\0\350\3\3\0\10\0\0\0"..., 4096) = 4096 lseek(3, 4675072163840, SEEK_SET) = 4675072163840 write(3, "\355A\350\3\0\20\0\0009\271\371M\206\177\0N\200\206\33M\0\0\0\0\350\3\2\0\10\0\0\0"..., 4096) = 4096 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32719 root 20 0 785m 768m 792 R 98 20.0 2443:18 resize2fs $ sudo resize2fs /dev/mapper/data-data 3000G resize2fs 1.41.11 (14-Mar-2010) Resizing the filesystem on /dev/mapper/data-data to 786432000 (4k) blocks. Time passes. :D It's an LVM comprising 4x2TB disks in RAID10 and 4x500GB in RAID10. $ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md1 : active raid10 sdi1[0] sdg1[1] sdf1[3] sdh1[2] 976767872 blocks 64K chunks 2 near-copies [4/4] [UUUU] md0 : active raid10 sda1[2] sdc1[3] sdb1[1] sdd1[0] 3907023872 blocks 64K chunks 2 near-copies [4/4] [UUUU] Disks are 7200RPM SATA disks. It's ~2TB full of data which is mostly rsnapshots of lots of remote hosts, so lots of little files. Cheers, Al. From tytso at mit.edu Fri Jun 24 14:27:49 2011 From: tytso at mit.edu (Ted Ts'o) Date: Fri, 24 Jun 2011 10:27:49 -0400 Subject: How long should resize2fs take? In-Reply-To: References: Message-ID: <20110624142749.GE3064@thunk.org> On Fri, Jun 24, 2011 at 10:48:51AM +0100, Alan Pope wrote: > Hullo! > > First mail, sorry if this is the wrong place for this kind of > question. I realise this is a "piece of string" type question. > > tl;dr version: I have a resizefs shrinking an ext4 filesystem from > ~4TB to ~3TB and it's been running for ~2 days. Is this normal? Shrinking a file system can take a long time; it depends on how many files are using space in the part of the file system that needs to be evacuated for the shrink to take place. > It's ~2TB full of data which is mostly rsnapshots of lots of remote > hosts, so lots of little files. Yes, that will take longer. Resize2fs is engineered for safety, which means it copies a lot of blocks, and then it updates the inodes, and then copies more blocks, and then updates the inode involved, etc. So it's a fairly seeky operation that can take a while. Most of the time people are growing their file systems, not shrinking them, so we haven't done a huge amount of optimization for speed in the shrink case. Regards, - Ted From alan at popey.com Fri Jun 24 14:56:02 2011 From: alan at popey.com (Alan Pope) Date: Fri, 24 Jun 2011 15:56:02 +0100 Subject: How long should resize2fs take? In-Reply-To: <20110624142749.GE3064@thunk.org> References: <20110624142749.GE3064@thunk.org> Message-ID: Hi Ted, Thanks for the prompt reply. On 24 June 2011 15:27, Ted Ts'o wrote: > Yes, that will take longer. ?Resize2fs is engineered for safety, which > means it copies a lot of blocks, and then it updates the inodes, and > then copies more blocks, and then updates the inode involved, etc. ?So > it's a fairly seeky operation that can take a while. > Is it a process that could be terminated relatively safely? Or is it just a case of "sit tight"? Al. From alan at popey.com Sat Jun 25 08:29:13 2011 From: alan at popey.com (Alan Pope) Date: Sat, 25 Jun 2011 09:29:13 +0100 Subject: How long should resize2fs take? In-Reply-To: References: <20110624142749.GE3064@thunk.org> Message-ID: On 24 June 2011 15:56, Alan Pope wrote: > Is it a process that could be terminated relatively safely? Or is it > just a case of "sit tight"? > I sat tight and it finished okay. alan at ubuntuserver:~$ sudo resize2fs /dev/mapper/data-data 3000G [sudo] password for alan: resize2fs 1.41.11 (14-Mar-2010) Resizing the filesystem on /dev/mapper/data-data to 786432000 (4k) blocks. The filesystem on /dev/mapper/data-data is now 786432000 blocks long. Thanks, Al. From sean.d.mccauliff at nasa.gov Sun Jun 26 17:33:16 2011 From: sean.d.mccauliff at nasa.gov (Mccauliff, Sean D. (ARC-PX)[Lockheed Martin Space OPNS]) Date: Sun, 26 Jun 2011 12:33:16 -0500 Subject: High CPU Utilization When Copying to Ext4 Message-ID: <341DAA96EE3A8444B6E4657BE8A846EA4B3DA126FE@NDJSSCC06.ndc.nasa.gov> Sorry if this is not the correct mailing list for ext4 questions. I'm copying terabytes of data from an ext3 file system to a new ext4 file system. I'm seeing high CPU usage from the processes flush-253:2, kworker-3:0, kworker-2:2, kworker-1:1, and kworker-0:0. Does anyone on the list have any idea what these processes do, why they are consuming so much cpu time and if there is something that can be done about it? This is using Fedora 15. Thanks! Sean