From my_qa2004 at yahoo.com Tue Oct 5 14:21:53 2004 From: my_qa2004 at yahoo.com (Ash) Date: Tue, 5 Oct 2004 07:21:53 -0700 (PDT) Subject: Corrupted journal In-Reply-To: <1096573477.1977.425.camel@sisko.scot.redhat.com> Message-ID: <20041005142153.55292.qmail@web53207.mail.yahoo.com> > >From "man tune2fs": > > -f Force the tune2fs operation to > complete even in the face of > errors. This option is useful when > removing the has_journal > filesystem feature from a filesystem > which has an external jour- > nal (or is corrupted such that it > appears to have an external > journal), but that external journal is > not available. > > Does this help you to get further? > I don't think so. It gave me some error about the filesystem having the "needs_recovery" attribute set. I don't remember exactly but in the end, I landed up using the 'features' command of debugfs to clear these attributes and then attach a fresh journal back. Was there any other way ? Thanks Ash __________________________________ Do you Yahoo!? Yahoo! Mail - 50x more storage than other providers! http://promotions.yahoo.com/new_mail From my_qa2004 at yahoo.com Tue Oct 5 14:26:59 2004 From: my_qa2004 at yahoo.com (Ash) Date: Tue, 5 Oct 2004 07:26:59 -0700 (PDT) Subject: information about known issues/bugs Message-ID: <20041005142659.47795.qmail@web53204.mail.yahoo.com> Hi Where can I find more information about open bugs/ known limitations against Ext3. Also, is there any doc giving planned features/tasks ? It will be a big help if someone could point me to this information. Thanks, Ash __________________________________ Do you Yahoo!? New and Improved Yahoo! Mail - Send 10MB messages! http://promotions.yahoo.com/new_mail From ogi at fmi.uni-sofia.bg Wed Oct 6 04:33:30 2004 From: ogi at fmi.uni-sofia.bg (Ognyan Kulev) Date: Wed, 06 Oct 2004 07:33:30 +0300 Subject: information about known issues/bugs In-Reply-To: <20041005142659.47795.qmail@web53204.mail.yahoo.com> References: <20041005142659.47795.qmail@web53204.mail.yahoo.com> Message-ID: <4163759A.8070703@fmi.uni-sofia.bg> Ash wrote: > Where can I find more information about open bugs/ > known limitations against Ext3. Also, is there any doc > giving planned features/tasks ? Somewhat old, but you'll get an idea: http://www.usenix.org/publications/library/proceedings/usenix02/tech/freenix/tso.html Regards, ogi From my_qa2004 at yahoo.com Wed Oct 6 09:05:00 2004 From: my_qa2004 at yahoo.com (Ash) Date: Wed, 6 Oct 2004 02:05:00 -0700 (PDT) Subject: information about known issues/bugs In-Reply-To: <4163759A.8070703@fmi.uni-sofia.bg> Message-ID: <20041006090500.29677.qmail@web53206.mail.yahoo.com> Thanks. Is there any kind of bug tracking system which allows to query and view bug reports against Ext3 ? I checked bugzilla on Redhat, but I could not find any component as "Ext3" in there. How do I search for any bugs specifcally against Ext3 in that system ? Any help will be appreicated. Thanks Ash --- Ognyan Kulev wrote: > Ash wrote: > > Where can I find more information about open bugs/ > > known limitations against Ext3. Also, is there any > doc > > giving planned features/tasks ? > > Somewhat old, but you'll get an idea: > http://www.usenix.org/publications/library/proceedings/usenix02/tech/freenix/tso.html > > Regards, > ogi > > _______________________________ Do you Yahoo!? Express yourself with Y! Messenger! Free. Download now. http://messenger.yahoo.com From ncunningham at linuxmail.org Thu Oct 7 00:47:30 2004 From: ncunningham at linuxmail.org (Nigel Cunningham) Date: Thu, 07 Oct 2004 10:47:30 +1000 Subject: cp -arl stuck in __wait_on_freeing_inode Message-ID: <1097110050.9693.61.camel@desktop.cunninghams> Hi all. I posted this to LKML, but might get a faster reply if I go direct to you. I haven't subscribed, so please reply to me too... I began a bk export -du patch and shortly afterwards a cp -arl on the kernel source tree. The latter got stuck in ext3_lookup iget_locked find_inode_fast __wait_on_freeing_inode schedule The box is running 2.6.8.1 with SMP, HT scheduler and preempt (can send a full .config if someone wants it). No other processes are stuck; it looks to me like it lost a race. Is this a known/fixed issue, or is there something I can do to diagnose further. I have KDB compiled in and haven't rebooted yet. Regards, Nigel -- Nigel Cunningham Pastoral Worker Christian Reformed Church of Tuggeranong PO Box 1004, Tuggeranong, ACT 2901 Many today claim to be tolerant. True tolerance, however, can cope with others being intolerant. From zhuye at pronetway.com Fri Oct 8 01:35:19 2004 From: zhuye at pronetway.com (Zhu Ye) Date: Fri, 8 Oct 2004 09:35:19 +0800 Subject: Recover partition Message-ID: <001301c4acd7$1591cf80$eb00a8c0@zhuye> Hi All, The partition of Red Hat 7.3 was destroied by hacker. I couldn't see any partitin use fdisk. Is there any tools can help me recover the partitions? Help me! Best Regards, Zhu Ye -------------- next part -------------- An HTML attachment was scrubbed... URL: From jrumpf at heavyload.net Fri Oct 8 02:46:59 2004 From: jrumpf at heavyload.net (Jeremy Rumpf) Date: Thu, 7 Oct 2004 22:46:59 -0400 Subject: Recover partition In-Reply-To: <001301c4acd7$1591cf80$eb00a8c0@zhuye> References: <001301c4acd7$1591cf80$eb00a8c0@zhuye> Message-ID: <200410072246.59316.jrumpf@heavyload.net> On Thursday 07 October 2004 09:35 pm, Zhu Ye wrote: > Hi All, > > The partition of Red Hat 7.3 was destroied by hacker. I couldn't see any > partitin use fdisk. Is there any tools can help me recover the partitions? > > Help me! > > Best Regards, > > Zhu Ye http://www.linuxforum.com/linux-partition/recovering.html http://www.stud.uni-hannover.de/user/76201/gpart/ From linuxtard at yahoo.com Fri Oct 8 04:09:58 2004 From: linuxtard at yahoo.com (Linux Tard) Date: Thu, 7 Oct 2004 21:09:58 -0700 (PDT) Subject: Recover partition In-Reply-To: <200410072246.59316.jrumpf@heavyload.net> Message-ID: <20041008040958.63190.qmail@web41004.mail.yahoo.com> rescuept as well. --- Jeremy Rumpf wrote: > On Thursday 07 October 2004 09:35 pm, Zhu Ye wrote: > > Hi All, > > > > The partition of Red Hat 7.3 was destroied by > hacker. I couldn't see any > > partitin use fdisk. Is there any tools can help me > recover the partitions? > > > > Help me! > > > > Best Regards, > > > > Zhu Ye > > http://www.linuxforum.com/linux-partition/recovering.html > > http://www.stud.uni-hannover.de/user/76201/gpart/ > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users > __________________________________ Do you Yahoo!? Yahoo! Mail Address AutoComplete - You start. We finish. http://promotions.yahoo.com/new_mail From urban.purkat at hermes.si Fri Oct 8 05:56:05 2004 From: urban.purkat at hermes.si (Urban Purkat) Date: Fri, 8 Oct 2004 07:56:05 +0200 Subject: kmem_cache_destroy: Can't free all objects Message-ID: Hello! I am writing a FS filter that will be above the ext3 filesystem. For my own purposes I need severl bytes in inode. There are not enough space current inode so I need to create my own inode functions alloc_inode() & destroy_inode(). The problem causes when destroying slab cache at removing my module (rmmod). kmem_cache_destroy: Can't free all objects What I do: - install my module (insmod) - Mount filesystem - list the mountpoint with ls -i - cat /proc/slabinfo shows: my_ii_cache 1 7 512 1 1 1 - I make umnount - cat /proc/slabinfo shows: my_ii_cache 4 14 512 1 2 1 Where did 4 came from? I believe it should be 0. - Then I remove module (rmmod): when callong function kmem_cache_destroy() I receive an error: kmem_cache_destroy: Can't free all objects de1b42b0 Can anyone help me? Is is possible that this is a kernel bug? BTW: I use RHEL3UP3 (kernel 2.4.21-20.EL) Regards, Urban Purkat From urban.purkat at hermes.si Thu Oct 7 14:52:39 2004 From: urban.purkat at hermes.si (Urban Purkat) Date: Thu, 7 Oct 2004 16:52:39 +0200 Subject: kmem_cache_destroy: Can't free all objects Message-ID: Hello! I am writing a FS filter that will be above the ext3 filesystem. For my own purposes I need severl bytes in inode. There are not enough space current inode so I need to create my own inode functions alloc_inode() & destroy_inode(). The problem causes when destroying slab cache at removing my module (rmmod). kmem_cache_destroy: Can't free all objects What I do: - install my module (insmod) - Mount filesystem - list the mountpoint with ls -i - cat /proc/slabinfo shows: my_ii_cache 1 7 512 1 1 1 - I make umnount - cat /proc/slabinfo shows: my_ii_cache 4 14 512 1 2 1 Where did 4 came from? I believe it should be 0. - Then I remove module (rmmod): when callong function kmem_cache_destroy() I receive an error: kmem_cache_destroy: Can't free all objects de1b42b0 Can anyone help me? Is is possible that this is a kernel bug? BTW: I use RHEL3UP3 (kernel 2.4.21-20.EL) Can you CC my e-mail address? I am not on the list. Regards, Urban Purkat From sp0ck1701 at yahoo.com Fri Oct 8 01:33:55 2004 From: sp0ck1701 at yahoo.com (hell know) Date: Thu, 7 Oct 2004 18:33:55 -0700 (PDT) Subject: Ext 2/3 overwriting remnant data & use of data blocks - security Message-ID: <20041008013355.47483.qmail@web53305.mail.yahoo.com> Greetings all- I am conducting security testing on a device that uses Linux 2.4 with ext3. I am testing secure overwrite of remnant data in temporary files, but have run into a real good stumpper in the way Ext allocates data blocks. I've got 10 yrs of *NIX behind me, several with Linux, and this has really got me perplexed as I can't find any documentation explaining the subject enough for me to figure out what's going on so I'm hoping someone can help shed some light on this... any help would be appreciated! BACKGROUND: Device under test uses temporary spool files. When those files are no longer needed, they are to be overwritten by the three-pass DOD overwrite (pattern '35', 'ca', '97'), then deleted. (Incase anyone out there asks the obvious question, I am aware that Ext supports a "secure" attribute but unfortunately that isn't enough for our purposes. It HAS to be a 3-pass overwrite... afterall that answer would be TOO EASY ;-). Also, the file is written and overwritten sequentially- that may be important to know when I get to the problem. What is supposed to happen is this: 1) File is created. Inode allocated. Data blocks allocated, etc. Initial data is put into the file. {For example, lets say the file occupies data blocks 100 - 200 } 2) File is read/processed. (And is then no longer needed) 3) File is overwritten three times from beginning to end to overwrite the remnant data: 3a) with pattern '35' { If I go in with dd and check data blocks 100-200 I should see all '35' } 3b) with pattern 'ca' { data blocks 100-200 should be all 'ca' } { If I go in with dd and check data blocks 100-200 I should see all 'ca' } 3c) with pattern '97' { data blocks 100-200 should be all '97 } { If I go in with dd and check data blocks 100-200 I should see all '97' } 4) File is deleted (inode/dir entry zapped per typical *NIX behavior) {data blocks 100-200 deallocated } { If I go in with dd and check data blocks 100-200 I should see all '97' still, until the blocks are reused by another file } Makes good sense, right? ;-) PROBLEM: I'm not seeing the behavior described above. Linux keeps "shifting" the data blocks around each time the file is written to. So using our hypothetical data block numbering, I see something like this occurring: 1) File is created. Inode allocated. Data blocks allocated, etc. Initial data is put into the file. {Data blocks 100 - 200 } 2) File is read/processed. (And is then no longer needed) 3) File is overwritten three times from beginning to end to overwrite the remnant data: 3a) with pattern '35' { data blocks 300-400 get written with all '35', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 3b) with pattern 'ca' { data blocks 500-600 get written with all 'ca', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 3c) with pattern '97' { data blocks 100-200 should be all '97 } { data blocks 700-800 get written with all '97', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 4) File is deleted (inode/dir entry zapped per typical *NIX behavior) {data blocks 100-200 deallocated } { If I go in with dd and check data blocks 100-200 I still see the original data until the data blocks are overwritten by another file } So you can see my problem. I suspect it has something to do with the way Ext 2/3 preallocates blocks, and it's use of "block groups". But I have been unable to locate any good docs which clearly explain exactly what algorithm logic controls this, and how to either change it or work with it somehow for the overwrite process. Can anyone shed any light into this? Am I heading in the right direction here? I'm not afraid to read any docs if you can point me to them. (Or if it can be explained simply and you're willing to type it out, that's great too!) MORE INFO As proof of concept, I wrote the simple script below. As you can see all it does is open a file, write a chunk of data to it (simulating the initial data write), and then repeats that process 25 times (simulating cycles of overwrites); each time it grabs the list of data blocks using debugfs and its "stat" command. And for my immediate purposes, the first 12 direct blocks are sufficient-- don't want to even think about indirect blocks until I get this behavior figured out. Before I get to the tty dumps, I want to say THANK YOU to anyone who's read this post this far... and of course, thanks in advance to any help!!! The pertinent info: # uname -a Linux vrh90 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 i686 athlon i386 GNU/Linux # mount /dev/hda2 on / type ext3 (rw) none on /proc type proc (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) //192.168.0.1/data1 on /mnt type smbfs (0) ---- for (( i=0;i<20;i=i+1 )); do echo Pass $i dd if=/dev/urandom of=/tmp/target1 bs=1 count=102400 debugfs /dev/hda2 -R "stat /tmp/target1" 2> /dev/null | tee -a /tmp/log done ---- Here is the "log" file containing debugfs "stat /tmp/target1" for each cycle. (I can see a definate pattern on some of these in which blocks are used, but again I'm trying to figure out what the algorithm is that causes it): ---- Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae1 -- Thu Oct 7 17:53:37 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae1 -- Thu Oct 7 17:53:37 2004 BLOCKS: (0-11):453994-454005, (IND):454006, (12-13):454007-454008, (14-24):455201-455211 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae5 -- Thu Oct 7 17:53:41 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae5 -- Thu Oct 7 17:53:41 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae8 -- Thu Oct 7 17:53:44 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae8 -- Thu Oct 7 17:53:44 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baec -- Thu Oct 7 17:53:48 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baec -- Thu Oct 7 17:53:48 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baef -- Thu Oct 7 17:53:51 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baef -- Thu Oct 7 17:53:51 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf2 -- Thu Oct 7 17:53:54 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf2 -- Thu Oct 7 17:53:54 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf6 -- Thu Oct 7 17:53:58 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf6 -- Thu Oct 7 17:53:58 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf9 -- Thu Oct 7 17:54:01 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf9 -- Thu Oct 7 17:54:01 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bafc -- Thu Oct 7 17:54:04 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bafc -- Thu Oct 7 17:54:04 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb00 -- Thu Oct 7 17:54:08 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb00 -- Thu Oct 7 17:54:08 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb03 -- Thu Oct 7 17:54:11 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb03 -- Thu Oct 7 17:54:11 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb07 -- Thu Oct 7 17:54:15 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb07 -- Thu Oct 7 17:54:15 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb0a -- Thu Oct 7 17:54:18 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb0a -- Thu Oct 7 17:54:18 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb0d -- Thu Oct 7 17:54:21 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb0d -- Thu Oct 7 17:54:21 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb11 -- Thu Oct 7 17:54:25 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb11 -- Thu Oct 7 17:54:25 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb14 -- Thu Oct 7 17:54:28 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb14 -- Thu Oct 7 17:54:28 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb17 -- Thu Oct 7 17:54:31 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb17 -- Thu Oct 7 17:54:31 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb1a -- Thu Oct 7 17:54:34 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb1a -- Thu Oct 7 17:54:34 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb1e -- Thu Oct 7 17:54:38 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb1e -- Thu Oct 7 17:54:38 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb21 -- Thu Oct 7 17:54:41 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb21 -- Thu Oct 7 17:54:41 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sp0ck1701 at yahoo.com Fri Oct 8 01:34:06 2004 From: sp0ck1701 at yahoo.com (hell know) Date: Thu, 7 Oct 2004 18:34:06 -0700 (PDT) Subject: Ext 2/3 overwriting remnant data & use of data blocks - security Message-ID: <20041008013406.85099.qmail@web53308.mail.yahoo.com> Greetings all- I am conducting security testing on a device that uses Linux 2.4 with ext3. I am testing secure overwrite of remnant data in temporary files, but have run into a real good stumpper in the way Ext allocates data blocks. I've got 10 yrs of *NIX behind me, several with Linux, and this has really got me perplexed as I can't find any documentation explaining the subject enough for me to figure out what's going on so I'm hoping someone can help shed some light on this... any help would be appreciated! BACKGROUND: Device under test uses temporary spool files. When those files are no longer needed, they are to be overwritten by the three-pass DOD overwrite (pattern '35', 'ca', '97'), then deleted. (Incase anyone out there asks the obvious question, I am aware that Ext supports a "secure" attribute but unfortunately that isn't enough for our purposes. It HAS to be a 3-pass overwrite... afterall that answer would be TOO EASY ;-). Also, the file is written and overwritten sequentially- that may be important to know when I get to the problem. What is supposed to happen is this: 1) File is created. Inode allocated. Data blocks allocated, etc. Initial data is put into the file. {For example, lets say the file occupies data blocks 100 - 200 } 2) File is read/processed. (And is then no longer needed) 3) File is overwritten three times from beginning to end to overwrite the remnant data: 3a) with pattern '35' { If I go in with dd and check data blocks 100-200 I should see all '35' } 3b) with pattern 'ca' { data blocks 100-200 should be all 'ca' } { If I go in with dd and check data blocks 100-200 I should see all 'ca' } 3c) with pattern '97' { data blocks 100-200 should be all '97 } { If I go in with dd and check data blocks 100-200 I should see all '97' } 4) File is deleted (inode/dir entry zapped per typical *NIX behavior) {data blocks 100-200 deallocated } { If I go in with dd and check data blocks 100-200 I should see all '97' still, until the blocks are reused by another file } Makes good sense, right? ;-) PROBLEM: I'm not seeing the behavior described above. Linux keeps "shifting" the data blocks around each time the file is written to. So using our hypothetical data block numbering, I see something like this occurring: 1) File is created. Inode allocated. Data blocks allocated, etc. Initial data is put into the file. {Data blocks 100 - 200 } 2) File is read/processed. (And is then no longer needed) 3) File is overwritten three times from beginning to end to overwrite the remnant data: 3a) with pattern '35' { data blocks 300-400 get written with all '35', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 3b) with pattern 'ca' { data blocks 500-600 get written with all 'ca', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 3c) with pattern '97' { data blocks 100-200 should be all '97 } { data blocks 700-800 get written with all '97', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 4) File is deleted (inode/dir entry zapped per typical *NIX behavior) {data blocks 100-200 deallocated } { If I go in with dd and check data blocks 100-200 I still see the original data until the data blocks are overwritten by another file } So you can see my problem. I suspect it has something to do with the way Ext 2/3 preallocates blocks, and it's use of "block groups". But I have been unable to locate any good docs which clearly explain exactly what algorithm logic controls this, and how to either change it or work with it somehow for the overwrite process. Can anyone shed any light into this? Am I heading in the right direction here? I'm not afraid to read any docs if you can point me to them. (Or if it can be explained simply and you're willing to type it out, that's great too!) MORE INFO As proof of concept, I wrote the simple script below. As you can see all it does is open a file, write a chunk of data to it (simulating the initial data write), and then repeats that process 25 times (simulating cycles of overwrites); each time it grabs the list of data blocks using debugfs and its "stat" command. And for my immediate purposes, the first 12 direct blocks are sufficient-- don't want to even think about indirect blocks until I get this behavior figured out. Before I get to the tty dumps, I want to say THANK YOU to anyone who's read this post this far... and of course, thanks in advance to any help!!! The pertinent info: # uname -a Linux vrh90 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 i686 athlon i386 GNU/Linux # mount /dev/hda2 on / type ext3 (rw) none on /proc type proc (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) //192.168.0.1/data1 on /mnt type smbfs (0) ---- for (( i=0;i<20;i=i+1 )); do echo Pass $i dd if=/dev/urandom of=/tmp/target1 bs=1 count=102400 debugfs /dev/hda2 -R "stat /tmp/target1" 2> /dev/null | tee -a /tmp/log done ---- Here is the "log" file containing debugfs "stat /tmp/target1" for each cycle. (I can see a definate pattern on some of these in which blocks are used, but again I'm trying to figure out what the algorithm is that causes it): ---- Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae1 -- Thu Oct 7 17:53:37 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae1 -- Thu Oct 7 17:53:37 2004 BLOCKS: (0-11):453994-454005, (IND):454006, (12-13):454007-454008, (14-24):455201-455211 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae5 -- Thu Oct 7 17:53:41 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae5 -- Thu Oct 7 17:53:41 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae8 -- Thu Oct 7 17:53:44 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae8 -- Thu Oct 7 17:53:44 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baec -- Thu Oct 7 17:53:48 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baec -- Thu Oct 7 17:53:48 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baef -- Thu Oct 7 17:53:51 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baef -- Thu Oct 7 17:53:51 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf2 -- Thu Oct 7 17:53:54 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf2 -- Thu Oct 7 17:53:54 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf6 -- Thu Oct 7 17:53:58 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf6 -- Thu Oct 7 17:53:58 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf9 -- Thu Oct 7 17:54:01 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf9 -- Thu Oct 7 17:54:01 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bafc -- Thu Oct 7 17:54:04 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bafc -- Thu Oct 7 17:54:04 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb00 -- Thu Oct 7 17:54:08 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb00 -- Thu Oct 7 17:54:08 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb03 -- Thu Oct 7 17:54:11 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb03 -- Thu Oct 7 17:54:11 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb07 -- Thu Oct 7 17:54:15 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb07 -- Thu Oct 7 17:54:15 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb0a -- Thu Oct 7 17:54:18 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb0a -- Thu Oct 7 17:54:18 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb0d -- Thu Oct 7 17:54:21 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb0d -- Thu Oct 7 17:54:21 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb11 -- Thu Oct 7 17:54:25 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb11 -- Thu Oct 7 17:54:25 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb14 -- Thu Oct 7 17:54:28 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb14 -- Thu Oct 7 17:54:28 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb17 -- Thu Oct 7 17:54:31 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb17 -- Thu Oct 7 17:54:31 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb1a -- Thu Oct 7 17:54:34 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb1a -- Thu Oct 7 17:54:34 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb1e -- Thu Oct 7 17:54:38 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb1e -- Thu Oct 7 17:54:38 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb21 -- Thu Oct 7 17:54:41 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb21 -- Thu Oct 7 17:54:41 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 --------------------------------- Do you Yahoo!? Express yourself with Y! Messenger! Free. Download now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mermell at amazon.com Fri Oct 8 19:04:42 2004 From: mermell at amazon.com (Mermell, Todd) Date: Fri, 8 Oct 2004 12:04:42 -0700 Subject: Ext 2/3 overwriting remnant data & use of data blocks - security Message-ID: <08399A9C7DB2294896E5ACBB120B607205E2CAB2@ex-mail-sea-01.ant.amazon.com> Hi there, In your example, you are using dd, which is going to truncate the file each time, resulting in reallocation of the data blocks: open("./", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 1 (where is an existing file) You might want to check what method you are using to overwrite the file, i.e. C, perl, shell, etc. If you're making the read/write or pread/pwrite system calls, then you might want to just check what flags are being passed to "open." -Todd -----Original Message----- From: hell know [mailto:sp0ck1701 at yahoo.com] Sent: Thu 10/7/2004 6:33 PM To: ext3-users at redhat.com Cc: Subject: Ext 2/3 overwriting remnant data & use of data blocks - security Greetings all- I am conducting security testing on a device that uses Linux 2.4 with ext3. I am testing secure overwrite of remnant data in temporary files, but have run into a real good stumpper in the way Ext allocates data blocks. I've got 10 yrs of *NIX behind me, several with Linux, and this has really got me perplexed as I can't find any documentation explaining the subject enough for me to figure out what's going on so I'm hoping someone can help shed some light on this... any help would be appreciated! BACKGROUND: Device under test uses temporary spool files. When those files are no longer needed, they are to be overwritten by the three-pass DOD overwrite (pattern '35', 'ca', '97'), then deleted. (Incase anyone out there asks the obvious question, I am aware that Ext supports a "secure" attribute but unfortunately that isn't enough for our purposes. It HAS to be a 3-pass overwrite... afterall that answer would be TOO EASY ;-). Also, the file is written and overwritten sequentially- that may be important to know when I get to the problem. What is supposed to happen is this: 1) File is created. Inode allocated. Data blocks allocated, etc. Initial data is put into the file. {For example, lets say the file occupies data blocks 100 - 200 } 2) File is read/processed. (And is then no longer needed) 3) File is overwritten three times from beginning to end to overwrite the remnant data: 3a) with pattern '35' { If I go in with dd and check data blocks 100-200 I should see all '35' } 3b) with pattern 'ca' { data blocks 100-200 should be all 'ca' } { If I go in with dd and check data blocks 100-200 I should see all 'ca' } 3c) with pattern '97' { data blocks 100-200 should be all '97 } { If I go in with dd and check data blocks 100-200 I should see all '97' } 4) File is deleted (inode/dir entry zapped per typical *NIX behavior) {data blocks 100-200 deallocated } { If I go in with dd and check data blocks 100-200 I should see all '97' still, until the blocks are reused by another file } Makes good sense, right? ;-) PROBLEM: I'm not seeing the behavior described above. Linux keeps "shifting" the data blocks around each time the file is written to. So using our hypothetical data block numbering, I see something like this occurring: 1) File is created. Inode allocated. Data blocks allocated, etc. Initial data is put into the file. {Data blocks 100 - 200 } 2) File is read/processed. (And is then no longer needed) 3) File is overwritten three times from beginning to end to overwrite the remnant data: 3a) with pattern '35' { data blocks 300-400 get written with all '35', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 3b) with pattern 'ca' { data blocks 500-600 get written with all 'ca', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 3c) with pattern '97' { data blocks 100-200 should be all '97 } { data blocks 700-800 get written with all '97', not 100-200 } { If I go in with dd and check data blocks 100-200 I still see the original data } 4) File is deleted (inode/dir entry zapped per typical *NIX behavior) {data blocks 100-200 deallocated } { If I go in with dd and check data blocks 100-200 I still see the original data until the data blocks are overwritten by another file } So you can see my problem. I suspect it has something to do with the way Ext 2/3 preallocates blocks, and it's use of "block groups". But I have been unable to locate any good docs which clearly explain exactly what algorithm logic controls this, and how to either change it or work with it somehow for the overwrite process. Can anyone shed any light into this? Am I heading in the right direction here? I'm not afraid to read any docs if you can point me to them. (Or if it can be explained simply and you're willing to type it out, that's great too!) MORE INFO As proof of concept, I wrote the simple script below. As you can see all it does is open a file, write a chunk of data to it (simulating the initial data write), and then repeats that process 25 times (simulating cycles of overwrites); each time it grabs the list of data blocks using debugfs and its "stat" command. And for my immediate purposes, the first 12 direct blocks are sufficient-- don't want to even think about indirect blocks until I get this behavior figured out. Before I get to the tty dumps, I want to say THANK YOU to anyone who's read this post this far... and of course, thanks in advance to any help!!! The pertinent info: # uname -a Linux vrh90 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 i686 athlon i386 GNU/Linux # mount /dev/hda2 on / type ext3 (rw) none on /proc type proc (rw) /dev/hda1 on /boot type ext3 (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) //192.168.0.1/data1 on /mnt type smbfs (0) ---- for (( i=0;i<20;i=i+1 )); do echo Pass $i dd if=/dev/urandom of=/tmp/target1 bs=1 count=102400 debugfs /dev/hda2 -R "stat /tmp/target1" 2> /dev/null | tee -a /tmp/log done ---- Here is the "log" file containing debugfs "stat /tmp/target1" for each cycle. (I can see a definate pattern on some of these in which blocks are used, but again I'm trying to figure out what the algorithm is that causes it): ---- Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae1 -- Thu Oct 7 17:53:37 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae1 -- Thu Oct 7 17:53:37 2004 BLOCKS: (0-11):453994-454005, (IND):454006, (12-13):454007-454008, (14-24):455201-455211 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae5 -- Thu Oct 7 17:53:41 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae5 -- Thu Oct 7 17:53:41 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bae8 -- Thu Oct 7 17:53:44 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bae8 -- Thu Oct 7 17:53:44 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baec -- Thu Oct 7 17:53:48 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baec -- Thu Oct 7 17:53:48 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baef -- Thu Oct 7 17:53:51 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baef -- Thu Oct 7 17:53:51 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf2 -- Thu Oct 7 17:53:54 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf2 -- Thu Oct 7 17:53:54 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf6 -- Thu Oct 7 17:53:58 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf6 -- Thu Oct 7 17:53:58 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165baf9 -- Thu Oct 7 17:54:01 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165baf9 -- Thu Oct 7 17:54:01 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bafc -- Thu Oct 7 17:54:04 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bafc -- Thu Oct 7 17:54:04 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb00 -- Thu Oct 7 17:54:08 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb00 -- Thu Oct 7 17:54:08 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb03 -- Thu Oct 7 17:54:11 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb03 -- Thu Oct 7 17:54:11 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb07 -- Thu Oct 7 17:54:15 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb07 -- Thu Oct 7 17:54:15 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb0a -- Thu Oct 7 17:54:18 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb0a -- Thu Oct 7 17:54:18 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb0d -- Thu Oct 7 17:54:21 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb0d -- Thu Oct 7 17:54:21 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb11 -- Thu Oct 7 17:54:25 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb11 -- Thu Oct 7 17:54:25 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb14 -- Thu Oct 7 17:54:28 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb14 -- Thu Oct 7 17:54:28 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb17 -- Thu Oct 7 17:54:31 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb17 -- Thu Oct 7 17:54:31 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb1a -- Thu Oct 7 17:54:34 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb1a -- Thu Oct 7 17:54:34 2004 BLOCKS: (0-10):14775-14785, (11):14787, (IND):14788, (12-18):14789-14795, (19-24):14804-14809 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb1e -- Thu Oct 7 17:54:38 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb1e -- Thu Oct 7 17:54:38 2004 BLOCKS: (0-7):8924-8931, (8):8933, (9-11):14775-14777, (IND):14778, (12-18):14779-14785, (19-24):14787-14792 TOTAL: 26 Inode: 209219 Type: regular Mode: 0644 Flags: 0x0 Generation: 3550218717 User: 0 Group: 0 Size: 102400 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 208 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4165bb21 -- Thu Oct 7 17:54:41 2004 atime: 0x4165bade -- Thu Oct 7 17:53:34 2004 mtime: 0x4165bb21 -- Thu Oct 7 17:54:41 2004 BLOCKS: (0-1):8887-8888, (2):8890, (3-8):8892-8897, (9-11):8900-8902, (IND):8903, (12-14):8904-8906, (15-24):8908-8917 TOTAL: 26 __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhansongling at hotmail.com Tue Oct 12 02:28:35 2004 From: zhansongling at hotmail.com (zhan songling) Date: Tue, 12 Oct 2004 10:28:35 +0800 Subject: Ext3 used in server -- Thanks Message-ID: i want to use Ext3 as journal file system in a server, and there are several requirements: 1,the largest file should be no smaller than 1TB 2,the largest file syetem should be no smaller than 4TB 3,the max file name should be no less than 255 bytes does Ext3 fix it? or how could i do to solve this requirements? Thanks for your patience and expertise! _________________________________________________________________ ???? MSN Explorer: http://explorer.msn.com/lccn/ From adilger at clusterfs.com Tue Oct 12 07:33:41 2004 From: adilger at clusterfs.com (Andreas Dilger) Date: Tue, 12 Oct 2004 01:33:41 -0600 Subject: Ext3 used in server -- Thanks In-Reply-To: References: Message-ID: <20041012073341.GV2061@schnapps.adilger.int> On Oct 12, 2004 10:28 +0800, zhan songling wrote: > i want to use Ext3 as journal file system in a server, and there are > several requirements: > 1,the largest file should be no smaller than 1TB > 2,the largest file syetem should be no smaller than 4TB > 3,the max file name should be no less than 255 bytes > does Ext3 fix it? or how could i do to solve this requirements? Is this for a 2.4 or 2.6 kernel? In 2.4 you can't even have block devices larger than 2TB, and in 2.6 the maximum (32-bit CPU) limit is 16TB. In theory ext3 can support a 4TB filesystem and 1TB files. Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From sp0ck1701 at yahoo.com Fri Oct 8 22:18:48 2004 From: sp0ck1701 at yahoo.com (sp0ck1701) Date: Fri, 8 Oct 2004 15:18:48 -0700 (PDT) Subject: Multiple-pass overwrite of EXT3 file on a journalled fs Message-ID: <20041008221848.32521.qmail@web53310.mail.yahoo.com> Greetings all, I am curious if anyone knows why utilities such as 'GNU shred' (part of coreutils) and 'wipe' say they are not effective on journalled file systems- especially EXT3. Is it because you can't "guarantee" that the journal has been flushed/wiped (i.e. you have the journal 'between' you and the actual data blocks on the physical disk), or because of buffering, or some other reason? On the same note, does anyone know of any utility that does overwrite EXT3? Most specifically, I am Looking for an open source utility that will do a MULTIPLE-pass overwrite of any data blocks used by a file. (This is for a government related project and DOD says it's not sufficient to just do a single overwrite of all zero's). Any help/pointers are greatly appreciated. Thanks. Below is an excerpt from the shred man page which specifically mentions EXT3: ---- CAUTION: Note that shred relies on a very important assumption: that the filesystem overwrites data in place. This is the traditional way to do things, but many modern filesystem designs do not satisfy this assumption. The following are examples of filesystems on which shred is not effective: * log-structured or journaled filesystems, such as those supplied with AIX and Solaris (and JFS, ReiserFS, XFS, Ext3, etc.) ---- _______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com From sp0ck1701 at yahoo.com Fri Oct 8 22:05:59 2004 From: sp0ck1701 at yahoo.com (sp0ck1701) Date: Fri, 8 Oct 2004 15:05:59 -0700 (PDT) Subject: Ext 2/3 overwriting remnant data & use of data blocks - security In-Reply-To: <08399A9C7DB2294896E5ACBB120B607205E2CAB2@ex-mail-sea-01.ant.amazon.com> Message-ID: <20041008220559.29222.qmail@web53310.mail.yahoo.com> Eureka!!! The simplest answer is usually the right one. I tried a sanity check rewriting that script in a simple C prog using ?w?, ?w+?, and ?r+? and sure enough it looks like truncation is what I was seeing. (Never though to check the differences concerning truncation between the three; r+ works, btw, and ironically it is the only one of the three that says it doesn?t truncate, go figure.) The obvious answer may have been in front of my nose all the time? Thanks!! (BTW I am validating a product and haven?t actually seen the source code, but I?d bet 90% odds this is the root cause.) --- "Mermell, Todd" wrote: > Hi there, > > In your example, you are using dd, which is going to > truncate the file each time, resulting in > reallocation of the data blocks: > > open("./", > O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 1 > > (where is an existing file) > > You might want to check what method you are using to > overwrite the file, i.e. C, perl, shell, etc. If > you're making the read/write or pread/pwrite system > calls, then you might want to just check what flags > are being passed to "open." > > -Todd > > > -----Original Message----- > From: hell know [mailto:sp0ck1701 at yahoo.com] > Sent: Thu 10/7/2004 6:33 PM > To: ext3-users at redhat.com > Cc: > Subject: Ext 2/3 overwriting remnant data & use of > data blocks - security > Greetings all- > > I am conducting security testing on a device that > uses Linux 2.4 with ext3. I am testing secure > overwrite of remnant data in temporary files, but > have run into a real good stumpper in the way Ext > allocates data blocks. I've got 10 yrs of *NIX > behind me, several with Linux, and this has really > got me perplexed as I can't find any documentation > explaining the subject enough for me to figure out > what's going on so I'm hoping someone can help shed > some light on this... any help would be appreciated! > > BACKGROUND: > Device under test uses temporary spool files. When > those files are no longer needed, they are to be > overwritten by the three-pass DOD overwrite (pattern > '35', 'ca', '97'), then deleted. (Incase anyone out > there asks the obvious question, I am aware that Ext > supports a "secure" attribute but unfortunately that > isn't enough for our purposes. It HAS to be a > 3-pass overwrite... afterall that answer would be > TOO EASY ;-). Also, the file is written and > overwritten sequentially- that may be important to > know when I get to the problem. > > What is supposed to happen is this: > 1) File is created. Inode allocated. Data blocks > allocated, etc. Initial data is put into the file. > {For example, lets say the file occupies data blocks > 100 - 200 } > > 2) File is read/processed. (And is then no longer > needed) > > 3) File is overwritten three times from beginning > to end to overwrite the remnant data: > > 3a) with pattern '35' > { If I go in with dd and check data blocks 100-200 I > should see all '35' } > > 3b) with pattern 'ca' { data blocks 100-200 should > be all 'ca' } > { If I go in with dd and check data blocks 100-200 I > should see all 'ca' } > > > 3c) with pattern '97' { data blocks 100-200 should > be all '97 } > { If I go in with dd and check data blocks 100-200 I > should see all '97' } > > > 4) File is deleted (inode/dir entry zapped per > typical *NIX behavior) {data blocks 100-200 > deallocated } > { If I go in with dd and check data blocks 100-200 I > should see all '97' still, until the blocks are > reused by another file } > > > Makes good sense, right? ;-) > > PROBLEM: > I'm not seeing the behavior described above. Linux > keeps "shifting" the data blocks around each time > the file is written to. So using our hypothetical > data block numbering, I see something like this > occurring: > > 1) File is created. Inode allocated. Data blocks > allocated, etc. Initial data is put into the file. > {Data blocks 100 - 200 } > > 2) File is read/processed. (And is then no longer > needed) > > 3) File is overwritten three times from beginning > to end to overwrite the remnant data: > > 3a) with pattern '35' > { data blocks 300-400 get written with all '35', not > 100-200 } > { If I go in with dd and check data blocks 100-200 I > still see the original data } > > > 3b) with pattern 'ca' > { data blocks 500-600 get written with all 'ca', not > 100-200 } > { If I go in with dd and check data blocks 100-200 I > still see the original data } > > > > 3c) with pattern '97' { data blocks 100-200 should > be all '97 } > > { data blocks 700-800 get written with all '97', not > 100-200 } > { If I go in with dd and check data blocks 100-200 I > still see the original data } > > > > 4) File is deleted (inode/dir entry zapped per > typical *NIX behavior) {data blocks 100-200 > deallocated } > > { If I go in with dd and check data blocks 100-200 I > still see the original data until the data blocks > are overwritten by another file } > > > So you can see my problem. I suspect it has > something to do with the way Ext 2/3 preallocates > blocks, and it's use of "block groups". But I have > been unable to locate any good docs which clearly > explain exactly what algorithm logic controls this, > and how to either change it or work with it somehow > for the overwrite process. > > Can anyone shed any light into this? Am I heading > in the right direction here? I'm not afraid to read > any docs if you can point me to them. (Or if it can > be explained simply and you're willing to type it > out, that's great too!) > > MORE INFO > As proof of concept, I wrote the simple script > below. As you can see all it does is open a file, > write a chunk of data to it (simulating the initial > data write), and then repeats that process 25 times > (simulating cycles of overwrites); each time it > grabs the list of data blocks using debugfs and its > "stat" command. And for my immediate purposes, the > first 12 direct blocks are sufficient-- don't want > to even think about indirect blocks until I get this > behavior figured out. > > Before I get to the tty dumps, I want to say THANK > YOU to anyone who's read this post this far... and > of course, thanks in advance to any help!!! > > The pertinent info: > # uname -a > Linux vrh90 2.4.20-8 #1 Thu Mar 13 17:18:24 EST 2003 > i686 athlon i386 GNU/Linux > # mount > /dev/hda2 on / type ext3 (rw) > none on /proc type proc (rw) > /dev/hda1 on /boot type ext3 (rw) > none on /dev/pts type devpts (rw,gid=5,mode=620) > none on /dev/shm type tmpfs (rw) > //192.168.0.1/data1 on /mnt type smbfs (0) > > ---- > for (( i=0;i<20;i=i+1 )); do > echo Pass $i > dd if=/dev/urandom of=/tmp/target1 bs=1 count=102400 > debugfs /dev/hda2 -R "stat /tmp/target1" 2> > /dev/null | tee -a /tmp/log > done > ---- > > Here is the "log" file containing debugfs "stat > /tmp/target1" for each cycle. (I can see a definate > pattern on some of these in which blocks are used, > but again I'm trying to figure out what the > algorithm is that causes it): > > ---- > Inode: 209219 Type: regular Mode: 0644 > Flags: === message truncated === __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From sp0ck1701 at yahoo.com Tue Oct 12 02:25:35 2004 From: sp0ck1701 at yahoo.com (sp0ck1701) Date: Mon, 11 Oct 2004 19:25:35 -0700 (PDT) Subject: function to read a block? Message-ID: <20041012022535.50537.qmail@web53302.mail.yahoo.com> Does anyone know if there is an ext2fs function to read the contents of a given data block directly from an ext2/3 file system? I need to dump the contents of data blocks directly from an ext2/3 file system itself. I know I can use fseek on the raw device to do this, but I am wondering if the ext library has any function that accepts a block number as a parameter and reads a block. (The work I am doing requires me to talk directly to an ext3 file system and data blocks). Also timing of the dumps is critical as it works on files in real-time, so I am thinking it would be fastest to use a direct call. Any info is greatly appreciated. Thanks in advance for any info. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From wangchen at nanjing-fnst.com Wed Oct 13 06:49:55 2004 From: wangchen at nanjing-fnst.com (Wang Chen) Date: Wed, 13 Oct 2004 14:49:55 +0800 Subject: Linux device number extension patch Message-ID: <12a101c4b0f0$d95cbe30$1701a8c0@wangchen> Hello, I have a question about device number extension in "e2fsprogs-1.35". In Linux kernel 2.6, device number is extended from 16-bit to 32-bit. All utilities and libraries should make corresponding extension for this new feature in kernel 2.6. Following operation defines type "kdev_t" as "unsigned short" type, thus the device number has only 16-bit. In file lib/ext2fs/jfs_user.h: 4 typedef unsigned short kdev_t; In file misc/jfs_user.h: 4 typedef unsigned short kdev_t; In file debugfs/jfs_user.h: 4 typedef unsigned short kdev_t; It seems not to correspond to device number extension.I think Kdev_t should be defined as 32-bit long int. The attached patch is for device number extension. As following: --- e2fsprogs-1.35-orig/debugfs/jfs_user.h 2004-08-25 13:42:41.118920528 -0400 +++ e2fsprogs-1.35/debugfs/jfs_user.h 2004-08-25 13:47:00.732453272 -0400 @@ -1,7 +1,7 @@ #ifndef _JFS_USER_H #define _JFS_USER_H -typedef unsigned short kdev_t; +typedef unsigned long kdev_t; #include --- e2fsprogs-1.35-orig/lib/ext2fs/jfs_user.h 2004-08-25 13:42:41.136917792 -0400 +++ e2fsprogs-1.35/lib/ext2fs/jfs_user.h 2004-08-25 13:43:42.790545016 -0400 @@ -1,7 +1,7 @@ #ifndef _JFS_USER_H #define _JFS_USER_H -typedef unsigned short kdev_t; +typedef unsigned long kdev_t; #include "kernel-jbd.h" --- e2fsprogs-1.35-orig/misc/jfs_user.h 2004-08-25 13:42:41.247900920 -0400 +++ e2fsprogs-1.35/misc/jfs_user.h 2004-08-25 13:46:38.840781312 -0400 @@ -1,7 +1,7 @@ #ifndef _JFS_USER_H #define _JFS_USER_H -typedef unsigned short kdev_t; +typedef unsigned long kdev_t; #include Best Regards, Wang Chen ---------------------------------------------------------- Wang Chen Dept. of Technology and Development Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 16-5, Guangzhou Rd., Nanjing, P.R.China 210008 PHONE : +86+25-86630523-636 FUJITSU INTERNAL: 79955-636 FAX : +86+25-83317685 Mail : wangchen at nanjing-fnst.com ---------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: e2fsprogs-1.35-device_number.patch Type: application/octet-stream Size: 929 bytes Desc: not available URL: From adilger at clusterfs.com Wed Oct 13 07:26:35 2004 From: adilger at clusterfs.com (Andreas Dilger) Date: Wed, 13 Oct 2004 01:26:35 -0600 Subject: Multiple-pass overwrite of EXT3 file on a journalled fs In-Reply-To: <20041008221848.32521.qmail@web53310.mail.yahoo.com> References: <20041008221848.32521.qmail@web53310.mail.yahoo.com> Message-ID: <20041013072635.GI2061@schnapps.adilger.int> On Oct 08, 2004 15:18 -0700, sp0ck1701 wrote: > I am curious if anyone knows why utilities such as > 'GNU shred' (part of coreutils) and 'wipe' say they > are not effective on journalled file systems- > especially EXT3. > > Below is an excerpt from the shred man page which > specifically mentions EXT3: > > ---- > CAUTION: Note that shred relies on a very important > assumption: that the filesystem overwrites data in > place. This is the traditional way to do things, but > many modern filesystem designs do not satisfy this > assumption. The following are examples of filesystems > on which shred is not effective: > > * log-structured or journaled filesystems, such as > those supplied with > > AIX and Solaris (and JFS, ReiserFS, XFS, Ext3, > etc.) > ---- I don't know why they would say this. With very early versions of ext3 (or filesystems mounted with data=journal) all data writes would also go to the journal, so you would get two copies of your data on disk. The "shred" overwrite would in theory only erase the copy in the filesystem. It is also possible to get a copy of the last partial page of a truncated file being put into the journal. Of course, this neglects the fact that the journal is continually being overwritten so the only time there might be data in the journal (in the non-default data=journal mode) is after it was just written to disk. In normal filesystem usage this would be overwritten hundreds of times, and the user-space overwriting tool would itself cause the journal to be overwritten in this case (taking relative sizes of files and journal into account). I know reiser4 on the other hand is a log-structured filesystem, so overwriting a file has no guarantee that even the data blocks will be overwritten. Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From jaten at ucla.edu Sun Oct 17 21:34:48 2004 From: jaten at ucla.edu (Jason Aten) Date: Sun, 17 Oct 2004 15:34:48 -0600 (MDT) Subject: hole in ext3 filesystem - how to repair? Message-ID: <20041017153316.I14990@empire.acmeshells.com> I'm running debian with an ext3 filesystem in a 2.4.20 kernel on x86. A disk crash in my RAID array resulted in some ext3 filesystem corruption which "e2fsck -y -b 32768 /dev/sda1" does not repair. The filesystem mounts okay at first, but then reverts to read-only with the follow error in /var/log/messages: kernel: directory #213122574 contains a hole at offset 3596288 Can you point me in the direction of how to repair a hole in the directory? Thank you! - Jason From mark at reibert.com Fri Oct 15 04:11:20 2004 From: mark at reibert.com (Mark Reibert) Date: Thu, 14 Oct 2004 21:11:20 -0700 Subject: corrupt orphan inode list Message-ID: <1097813480.1323.22.camel@hellcat.home.reibert.com> Hello, I am running kernel 2.6.4 on XScale/ARM with BusyBox. Quite frequently after shutdown I have a number of orphaned inodes on the ext3 filesystem. (I have yet to figure out the precise cause of the orphans, but I believe it has to do with umount failing at shutdown due to a bug in BusyBox init that does not properly kill processes.) It is my understanding the orphaned inode list should be replayed on the next mount (after the journal is recovered), however this often fails due to a corrupt orphaned inode list (as reported by e2fsck). In extreme cases, then, I end up with a "clean" filesystem, but e2fsck -f fixes tens of thousands of orphaned inodes. What causes the corruption in the orphaned inode list itself? Is this why the orphaned inode list is not being cleaned up on mount? Is this at all related to the number of ext3 fixes noted in the kernel changelogs since 2.6.4? Thanks, Mark -- ---------------------- Mark S. Reibert, Ph.D. mark at reibert.com ---------------------- From cchan at outblaze.com Thu Oct 28 13:38:53 2004 From: cchan at outblaze.com (Christopher Chan) Date: Thu, 28 Oct 2004 21:38:53 +0800 Subject: kernel oops with latest FC2 kernel Message-ID: <4180F66D.4020901@outblaze.com> Unable to handle kernel NULL pointer dereference at virtual address 00000004 printing eip: 42839407 *pde = 00003001 Oops: 0002 [#1] SMP Modules linked in: md5 ipv6 autofs4 e100 mii ipt_REJECT iptable_filter ip_tables floppy sg scsi_mod microcode dm_mod ohci_hcd ext3 jbd raid1 raid0 CPU: 1 EIP: 0060:[<42839407>] Not tainted EFLAGS: 00010202 (2.6.8-1.521smp) EIP is at journal_commit_transaction+0x70c/0x13f7 [jbd] eax: 0fba338c ebx: 00000000 ecx: 0b8744c0 edx: 00000000 esi: 0fba338c edi: 41fc2200 ebp: 3dc09e60 esp: 3fd40da0 ds: 007b es: 007b ss: 0068 Process kjournald (pid: 910, threadinfo=3fd40000 task=3fe4b850) Stack: 00000000 00000000 00000000 00000000 00000000 1a6bd53c 41fc2200 1197029c 00001cc5 00000001 3fd40e14 0211c589 000005de ffffffff 00000001 00000080 00000300 00000300 00000300 00000000 0240d02c 00000000 00000000 3fe4b850 Call Trace: [<0211c589>] find_busiest_group+0xe6/0x2b7 [<0211ebdc>] autoremove_wake_function+0x0/0x2d [<0211c589>] find_busiest_group+0xe6/0x2b7 [<0211ebdc>] autoremove_wake_function+0x0/0x2d [<022d661a>] schedule+0x65e/0x6e9 [<02128521>] del_timer_sync+0x6f/0x87 [<4283c18f>] kjournald+0x10d/0x326 [jbd] [<0211ebdc>] autoremove_wake_function+0x0/0x2d [<0211ebdc>] autoremove_wake_function+0x0/0x2d [<4283c07c>] commit_timeout+0x0/0x5 [jbd] [<4283c082>] kjournald+0x0/0x326 [jbd] [<021041f1>] kernel_thread_helper+0x5/0xb Code: f0 ff 43 04 8b 03 a8 04 0f 84 b1 00 00 00 8b 4c 24 18 b2 01