From fuller at droog.sdf-eu.org Thu Apr 2 12:08:00 2009 From: fuller at droog.sdf-eu.org (Felix Resch) Date: Thu, 2 Apr 2009 12:08:00 +0000 Subject: external journal lost In-Reply-To: <20090331144755.GA14155@droog.sdf-eu.org> References: <20090331144755.GA14155@droog.sdf-eu.org> Message-ID: <20090402120800.GB6618@droog.sdf-eu.org> Hi, today i realised that tune2fs is able to change the uuid of the journal (partition/fs?) so i was able to fsck the volume and i am looking forward to get it back online within the day. Thouh i am still unclear about the semantics of the 'force' flag of tune2fs. Any hints? greets Felix Resch From lists at nerdbynature.de Thu Apr 2 21:12:53 2009 From: lists at nerdbynature.de (Christian Kujau) Date: Thu, 2 Apr 2009 14:12:53 -0700 (PDT) Subject: external journal lost In-Reply-To: <20090402120800.GB6618@droog.sdf-eu.org> References: <20090331144755.GA14155@droog.sdf-eu.org> <20090402120800.GB6618@droog.sdf-eu.org> Message-ID: On Thu, 2 Apr 2009, Felix Resch wrote: > Thouh i am still unclear about the semantics of the 'force' flag of > tune2fs. Hm, I never had to forcefully remove a journal (yet), but from reading the manpage I'd indeed expect to get this removed even if the filesystem is in error. The attached patch to e2fsprogs (latest git) makes tune2fs removing the journal when "-f" is supplied. And it seems to work so far: http://nerdbynature.de/bits/tune2fs/ Comments? Christian. -- Anyone who makes love to Bruce Schneier discovers a 0-day flaw in a crypto protocol the next day. -------------- next part -------------- A non-text attachment was scrubbed... Name: tune2fs_fflag.diff Type: text/x-diff Size: 558 bytes Desc: URL: From pilsl at goldfisch.at Fri Apr 3 12:42:44 2009 From: pilsl at goldfisch.at (peter pilsl) Date: Fri, 3 Apr 2009 13:42:44 +0100 (GMT+01:00) Subject: filesystem not full, but 0% available In-Reply-To: <274341890.21091238762386025.JavaMail.root@zimbra.goldfisch.at> Message-ID: <1952932029.21111238762564024.JavaMail.root@zimbra.goldfisch.at> I ran into a curious problem today: Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 684172708 652718132 0 100% /mnt/sda3 While there should be 30GB available it shows that its full. The partition is my /-partation and party of my linux (2.6.24.19) thinks that this partition is really full. Syslogd/Klogd are not working and /tmp is mounted as overflow-Ramdisk on boot. I did a forced fsck twice by now, and I booted into a brand new 2.6.28-11-kernel and ext2-tools version 1.41.4 and the same problem. Whats going on here? Anything left I can to debug or solve this problem? Next step is to backup all data on the disk, reformat the partition and restore the files, but actually this is not what I like to do on a regular base :) fdisk does not complain about any harddisk/partition-troubles : # fdisk -l /dev/sda Disk /dev/sda: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00083824 Device Boot Start End Blocks Id System /dev/sda1 * 1 4863 39062016 83 Linux /dev/sda2 4864 5349 3903795 82 Linux swap / Solaris /dev/sda3 5350 91201 689606190 83 Linux and there is a second ext3-partition on the same disk (sda1) which doesnt have any similar problem. Would removing and adding the journal help? (I did this once but I cant remember why and especially how I did it) Any help greatly appretiated, I add the result of tune2fs -l /dev/sda3 at the end of this posting. Maybe it helps. thnx a lot, peter # tune2fs -l /dev/sda3 tune2fs 1.41.4 (27-Jan-2009) Filesystem volume name: Last mounted on: Filesystem UUID: 3098e1e4-e120-469e-9e7b-41c2bb730b8f Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 43106304 Block count: 172401547 Reserved block count: 8620077 Free blocks: 7863644 Free inodes: 40182634 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 982 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 256 Filesystem created: Wed Jun 18 10:19:42 2008 Last mount time: Fri Apr 3 12:21:20 2009 Last write time: Fri Apr 3 12:21:20 2009 Mount count: 1 Maximum mount count: 34 Last checked: Fri Apr 3 11:09:13 2009 Check interval: 15552000 (6 months) Next check after: Wed Sep 30 11:09:13 2009 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 3536ec11-ab59-45b4-a4a4-6f1a37901363 Journal backup: inode blocks From Colin.vanNiekerk at mimecast.co.za Fri Apr 3 14:19:08 2009 From: Colin.vanNiekerk at mimecast.co.za (Colin van Niekerk) Date: Fri, 3 Apr 2009 16:19:08 +0200 Subject: filesystem not full, but 0% available In-Reply-To: <1952932029.21111238762564024.JavaMail.root@zimbra.goldfisch.at> References: <274341890.21091238762386025.JavaMail.root@zimbra.goldfisch.at> <1952932029.21111238762564024.JavaMail.root@zimbra.goldfisch.at> Message-ID: Hi there, I would say this is probably because the normal behaviour when formatting partitions is to reserve 5% of the partition for UID 0 user (root) and GID 0 (root) group. The other partition is very much smaller so the effects of this would be much less noticeable. If you take the number of reserved blocks and multiply them by the size of each block you get just a little more than 30GB. Reserved block count: 8620077 Block size: 4096 The 'Available' column show the amount of space available to non-root users. Does the following command work? Dd if=/dev/zero of=/mnt/sda3/testfile bs=1024 count=10000 This should create a 10MB file called /mnt/sda3/testfile and will let us know if root can still write data to the device. Are these processes running as root? Please could you provide the output of the following commands: cat /etc/redhat-release uname -a ps -ef |egrep "syslog[d]|klog[d]" ll `which klogd` `which syslogd` mount |grep quota Both these processes normally run as root, but it is possible to alter this behaviour so let's check these things. Regards, Colin -----Original Message----- From: ext3-users-bounces at redhat.com [mailto:ext3-users-bounces at redhat.com] On Behalf Of peter pilsl Sent: 03 April 2009 02:43 PM To: ext3-users at redhat.com Subject: filesystem not full, but 0% available I ran into a curious problem today: Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 684172708 652718132 0 100% /mnt/sda3 While there should be 30GB available it shows that its full. The partition is my /-partation and party of my linux (2.6.24.19) thinks that this partition is really full. Syslogd/Klogd are not working and /tmp is mounted as overflow-Ramdisk on boot. I did a forced fsck twice by now, and I booted into a brand new 2.6.28-11-kernel and ext2-tools version 1.41.4 and the same problem. Whats going on here? Anything left I can to debug or solve this problem? Next step is to backup all data on the disk, reformat the partition and restore the files, but actually this is not what I like to do on a regular base :) fdisk does not complain about any harddisk/partition-troubles : # fdisk -l /dev/sda Disk /dev/sda: 750.1 GB, 750156374016 bytes 255 heads, 63 sectors/track, 91201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00083824 Device Boot Start End Blocks Id System /dev/sda1 * 1 4863 39062016 83 Linux /dev/sda2 4864 5349 3903795 82 Linux swap / Solaris /dev/sda3 5350 91201 689606190 83 Linux and there is a second ext3-partition on the same disk (sda1) which doesnt have any similar problem. Would removing and adding the journal help? (I did this once but I cant remember why and especially how I did it) Any help greatly appretiated, I add the result of tune2fs -l /dev/sda3 at the end of this posting. Maybe it helps. thnx a lot, peter # tune2fs -l /dev/sda3 tune2fs 1.41.4 (27-Jan-2009) Filesystem volume name: Last mounted on: Filesystem UUID: 3098e1e4-e120-469e-9e7b-41c2bb730b8f Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 43106304 Block count: 172401547 Reserved block count: 8620077 Free blocks: 7863644 Free inodes: 40182634 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 982 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 256 Filesystem created: Wed Jun 18 10:19:42 2008 Last mount time: Fri Apr 3 12:21:20 2009 Last write time: Fri Apr 3 12:21:20 2009 Mount count: 1 Maximum mount count: 34 Last checked: Fri Apr 3 11:09:13 2009 Check interval: 15552000 (6 months) Next check after: Wed Sep 30 11:09:13 2009 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 3536ec11-ab59-45b4-a4a4-6f1a37901363 Journal backup: inode blocks _______________________________________________ Ext3-users mailing list Ext3-users at redhat.com https://www.redhat.com/mailman/listinfo/ext3-users From thorsten.henrici at gfd.de Fri Apr 3 21:28:27 2009 From: thorsten.henrici at gfd.de (thorsten.henrici at gfd.de) Date: Fri, 3 Apr 2009 23:28:27 +0200 Subject: =?iso-8859-1?q?Thorsten_Henrici_ist_au=DFer_Haus=2E?= Message-ID: Ich werde ab 03.04.2009 nicht im B?ro sein. Ich kehre zur?ck am 04.04.2009. Ich werde Ihre Nachricht nach meiner R?ckkehr beantworten. In dringenden F?llen wenden Sie sich bitte an Herrn Nadrowski. I'm out of office until the 19th of February. In urgent cases please contact Mr. Nadrowski. -- IMPORTANT NOTICE: This email is confidential, may be legally privileged, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone else is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. From lists at nerdbynature.de Wed Apr 8 08:40:22 2009 From: lists at nerdbynature.de (Christian Kujau) Date: Wed, 8 Apr 2009 01:40:22 -0700 (PDT) Subject: external journal lost [PATCH] In-Reply-To: References: <20090331144755.GA14155@droog.sdf-eu.org> <20090402120800.GB6618@droog.sdf-eu.org> Message-ID: On Thu, 2 Apr 2009, Christian Kujau wrote: > Hm, I never had to forcefully remove a journal (yet), but from reading the > manpage I'd indeed expect to get this removed even if the filesystem is in > error. The attached patch to e2fsprogs (latest git) makes tune2fs removing > the journal when "-f" is supplied. And it seems to work so far: > > http://nerdbynature.de/bits/tune2fs/ > > Comments? Hm, I was really hoping someone would comment on this, as I'm a) puzzled as well, why the -force option does not work as expected and b) if this is the right thing to do: --- e2fsprogs-git/misc/tune2fs.c.ORIG 2009-04-02 22:40:58.740218188 +0200 +++ e2fsprogs-git/misc/tune2fs.c 2009-04-02 22:42:37.221830335 +0200 @@ -374,8 +374,8 @@ static void update_feature_set(ext2_fils "read-only.\n"), stderr); exit(1); } - if (sb->s_feature_incompat & - EXT3_FEATURE_INCOMPAT_RECOVER) { + if ((sb->s_feature_incompat & + EXT3_FEATURE_INCOMPAT_RECOVER) && (!f_flag)) { fputs(_("The needs_recovery flag is set. " "Please run e2fsck before clearing\n" "the has_journal flag.\n"), stderr); -- Bruce Schneier knows the secret formula for Coca-Cola. From adilger at sun.com Thu Apr 9 04:31:21 2009 From: adilger at sun.com (Andreas Dilger) Date: Wed, 08 Apr 2009 22:31:21 -0600 Subject: external journal lost [PATCH] In-Reply-To: References: <20090331144755.GA14155@droog.sdf-eu.org> <20090402120800.GB6618@droog.sdf-eu.org> Message-ID: <20090409043121.GR17302@webber.adilger.int> On Apr 08, 2009 01:40 -0700, Christian Kujau wrote: > On Thu, 2 Apr 2009, Christian Kujau wrote: > > Hm, I never had to forcefully remove a journal (yet), but from reading the > > manpage I'd indeed expect to get this removed even if the filesystem is in > > error. The attached patch to e2fsprogs (latest git) makes tune2fs removing > > the journal when "-f" is supplied. And it seems to work so far: > > > > http://nerdbynature.de/bits/tune2fs/ > > > > Comments? > > Hm, I was really hoping someone would comment on this, as I'm a) puzzled > as well, why the -force option does not work as expected and b) if this is > the right thing to do: Proabably the right thing to do. > --- e2fsprogs-git/misc/tune2fs.c.ORIG 2009-04-02 22:40:58.740218188 +0200 > +++ e2fsprogs-git/misc/tune2fs.c 2009-04-02 22:42:37.221830335 +0200 > @@ -374,8 +374,8 @@ static void update_feature_set(ext2_fils > "read-only.\n"), stderr); > exit(1); > } > - if (sb->s_feature_incompat & > - EXT3_FEATURE_INCOMPAT_RECOVER) { > + if ((sb->s_feature_incompat & > + EXT3_FEATURE_INCOMPAT_RECOVER) && (!f_flag)) { > fputs(_("The needs_recovery flag is set. " > "Please run e2fsck before clearing\n" > "the has_journal flag.\n"), stderr); > > -- > Bruce Schneier knows the secret formula for Coca-Cola. > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From josh at moonfruit.com Sat Apr 11 00:32:30 2009 From: josh at moonfruit.com (Hiren Joshi) Date: Sat, 11 Apr 2009 01:32:30 +0100 Subject: Filesystem gone readonly Message-ID: <49DFE51E.7060403@moonfruit.com> I've got a really weird situation here. I'm using RHEL 4 and connecting to an EMC storage device using fibre and qla2300. The luns are put into LVM and we have a number of 400G partitions coming off that, I made a snapshot and ran fsck -yn on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 2392655. IGNORED. Inode 2392655, i_blocks is 672, should be 664. Fix? no Inode 29507663 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 29507663. IGNORED. Inode 32918223 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 32918223. IGNORED. Inode 33768943 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33768943. IGNORED. Inode 33771951 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33771951. IGNORED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Duplicate/bad block(s) in inode 37063007: 74791522 Duplicate/bad block(s) in inode 37063010: 74791523 74791524 74791525 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 11 inodes containing duplicate/bad blocks.) File /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 10 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Clone duplicate/bad blocks? no Delete file? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? no Free blocks count wrong for group #2282 (0, counted=18). Fix? no Free blocks count wrong (22802537, counted=22802555). Fix? no Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? no Free inodes count wrong for group #643 (15860, counted=15846). Fix? no Free inodes count wrong (47980634, counted=47980620). Fix? no webspace2: ********** WARNING: Filesystem still has errors ********** webspace2: 4448166/52428800 files (3.4% non-contiguous), 82055063/104857600 blocks So, I took the partition offline and ran fsck -yf on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 2392655. CLEARED. Inode 2392655, i_blocks is 672, should be 664. Fix? yes Inode 29507663 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 29507663. CLEARED. Inode 32918223 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 32918223. CLEARED. Inode 33768943 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33768943. CLEARED. Inode 33771951 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33771951. CLEARED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Duplicate/bad block(s) in inode 37062984: 74791523 Duplicate/bad block(s) in inode 37063012: 74791524 74791525 Duplicate/bad block(s) in inode 37290160: 74791522 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 12 inodes containing duplicate/bad blocks.) File /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) has 2 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 11 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Duplicated blocks already reassigned or cloned. Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #9. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? yes Free blocks count wrong for group #2 (16059, counted=16040). Fix? yes Free blocks count wrong for group #146 (7244, counted=7245). Fix? yes Free blocks count wrong for group #2282 (0, counted=18). Fix? yes Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4451892/52428800 files (3.5% non-contiguous), 82037602/104857600 blocks Then I saw this in /var/log/messages: Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 4096 Apr 9 21:53:56 kernel: Aborting journal on device dm-4. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 8192 Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 12288 Apr 9 21:53:56 kernel: ext3_abort called. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal Apr 9 21:53:56 kernel: Remounting filesystem read-only So I took it off-line and ran fsck -yf again with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) webspace2: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #1. Allocate? yes Directory inode 29507663 has an unallocated block #2. Allocate? yes Directory inode 29507663 has an unallocated block #3. Allocate? yes Directory inode 29507663 has an unallocated block #4. Allocate? yes Directory inode 29507663 has an unallocated block #5. Allocate? yes Directory inode 29507663 has an unallocated block #6. Allocate? yes Directory inode 29507663 has an unallocated block #7. Allocate? yes Directory inode 29507663 has an unallocated block #8. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong for group #2 (16029, counted=16021). Fix? yes Free blocks count wrong (22763960, counted=22763952). Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4454345/52428800 files (3.5% non-contiguous), 82093648/104857600 blocks What on earth is going on?! Can I expect the partition to go read-only again? Also, over the last week a number of other partitions from the same lvm and EMC setup and they all came out with: Illegal block #9 (4101620032) in inode 803439. CLEARED. In the fsck...... From dkg_004 at verizon.net Tue Apr 14 12:06:58 2009 From: dkg_004 at verizon.net (dkg_004 at verizon.net) Date: Tue, 14 Apr 2009 07:06:58 -0500 (CDT) Subject: kjournald stuck in DW< state Message-ID: <9438125.535952.1239710818462.JavaMail.root@vms076.mailsrvcs.net> An HTML attachment was scrubbed... URL: From tytso at mit.edu Tue Apr 14 14:16:15 2009 From: tytso at mit.edu (Theodore Tso) Date: Tue, 14 Apr 2009 10:16:15 -0400 Subject: kjournald stuck in DW< state In-Reply-To: <9438125.535952.1239710818462.JavaMail.root@vms076.mailsrvcs.net> References: <9438125.535952.1239710818462.JavaMail.root@vms076.mailsrvcs.net> Message-ID: <20090414141615.GE955@mit.edu> On Tue, Apr 14, 2009 at 07:06:58AM -0500, dkg_004 at verizon.net wrote: >kjournald D C0284784 0 408 6 (L-TLB) >[] (schedule+0x0/0x64c) from [] (journal_commit_transaction+0x16c/0x1568) >[] (journal_commit_transaction+0x0/0x1568) from [] (kjournald+0xbc/0x260) >[] (kjournald+0x0/0x260) from [] (kthread+0xe8/0x128) >[] (kthread+0x0/0x128) from [] (do_exit+0x0/0x8c8) Are you using any other patches or modifications to the kernel? It looks like the journal_commit_transaction() is blocked waiting for some transaction to finish: while (commit_transaction->t_updates) { DEFINE_WAIT(wait); prepare_to_wait(&journal->j_wait_updates, &wait, TASK_UNINTERRUPTIBLE); if (commit_transaction->t_updates) { spin_unlock(&commit_transaction->t_handle_lock); spin_unlock(&journal->j_state_lock); schedule(); <------------ spin_lock(&journal->j_state_lock); spin_lock(&commit_transaction->t_handle_lock); } finish_wait(&journal->j_wait_updates, &wait); } This can happen if the kernel has OOPS'ed while in the middle of an ext3 operation, or if there is some bug in ext3. The 2.6.21 kernel is quite old, so there are one or two bug fixes that might possibly apply to your problem. So the first easy thing to tryy would be to upgrade to the latest kernel (2.6.29) and see if that fixes your problem. You might also check your logs to see if there were any OOPS messages before the system locked up. And, I would try compiling with LOCKDEP enabled to see if that shows the problem. - Ted From dkg_004 at verizon.net Tue Apr 14 14:57:42 2009 From: dkg_004 at verizon.net (dkg_004 at verizon.net) Date: Tue, 14 Apr 2009 09:57:42 -0500 (CDT) Subject: kjournald stuck in DW< state Message-ID: <906082012.1058800.1239721062772.JavaMail.root@vms170003.mailsrvcs.net> An HTML attachment was scrubbed... URL: From sandeen at redhat.com Tue Apr 14 15:02:43 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Tue, 14 Apr 2009 10:02:43 -0500 Subject: kjournald stuck in DW< state In-Reply-To: <906082012.1058800.1239721062772.JavaMail.root@vms170003.mailsrvcs.net> References: <906082012.1058800.1239721062772.JavaMail.root@vms170003.mailsrvcs.net> Message-ID: <49E4A593.8090202@redhat.com> dkg_004 at verizon.net wrote: > Ted, > > There are not OOPS messages. > The kernel is provided by the CPU vendor. From what I know - the Ext3 > code has not been modified. > At the moment I do not have list of modifications the vendor made, and > not sure I will be able to get such list. I have the source code so > probably I can compare with the stock kernel and see what has been changed. > > Upgrading the kernel to 2.6.29 will be difficult. Will it be possible to > upgrade / patch the Ext3 code only? Will that work? It'd be tough. I've been playing with arm vendor kernels lately (probably the same codebase) and doing some filesystem backporting... it's possible but not for the faint-hearted. You did sysrq-t; if sysrq-w is supported in the kernel it will give you only the tasks in blocked state, which may show a deadlock more clearly? -Eric > I will try with LOCKDEP and see what happens. > > Thanks a lot. > > Dimitar From dkg_004 at verizon.net Tue Apr 14 15:07:07 2009 From: dkg_004 at verizon.net (dkg_004 at verizon.net) Date: Tue, 14 Apr 2009 10:07:07 -0500 (CDT) Subject: kjournald stuck in DW< state Message-ID: <615043250.1060153.1239721627571.JavaMail.root@vms170003.mailsrvcs.net> An HTML attachment was scrubbed... URL: From jordi.prats at gmail.com Wed Apr 15 13:57:57 2009 From: jordi.prats at gmail.com (Jordi Prats) Date: Wed, 15 Apr 2009 15:57:57 +0200 Subject: about tune2fs Message-ID: <1908f30904150657u57cb7b1ar6240f0536acd0a2c@mail.gmail.com> Hi all, In which cases using tune2fs should I unmount the filesystem? On the manpage there's no reference about online or offline operations. It would be nice to be specified there, wouldn't? regards, -- Jordi From kyle at kbrandt.com Thu Apr 16 11:53:59 2009 From: kyle at kbrandt.com (Kyle Brandt) Date: Thu, 16 Apr 2009 07:53:59 -0400 Subject: MTBF of Ext3 and Partition Size Message-ID: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> Hi All, On several of my servers I seem to have a high rate of server crashes do to file system errors. So I have some questions related to this: Is there any Mean Time Between Failure ( MTBF) data for the ext3 file-system? Does increased partition size cause a higher risk of the partition being corrupted? If so, is there any data on the ratio between partition size and the likely hood of failure? Does ext3 on hardware raid (10) increase the possibility of file system corruption? Does anyone recommend any partitioning schemes that might help the server stay up in the event of corruption? Are there any mount options that might help? Thank you, Kyle -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandeen at redhat.com Thu Apr 16 15:52:57 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Thu, 16 Apr 2009 10:52:57 -0500 Subject: MTBF of Ext3 and Partition Size In-Reply-To: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> References: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> Message-ID: <49E75459.90102@redhat.com> Kyle Brandt wrote: > Hi All, > > On several of my servers I seem to have a high rate of server crashes do > to file system errors. So I have some questions related to this: > > Is there any Mean Time Between Failure ( MTBF) data for the ext3 > file-system? > > Does increased partition size cause a higher risk of the partition being > corrupted? If so, is there any data on the ratio between partition size > and the likely hood of failure? > > Does ext3 on hardware raid (10) increase the possibility of file system > corruption? > > Does anyone recommend any partitioning schemes that might help the > server stay up in the event of corruption? > > Are there any mount options that might help? > > Thank you, > Kyle Rather than answer the above questions, I'd rather say that if ext3 is getting corrupted it's a bug somewhere (or bad hardware) - and if it's a bug then it needs to be investigated and fixed. Even before any temporary workaround could be suggested, it'd be most helpful if you could say what errors you are encountering, with as much detail as possible (i.e. kernel logs, e2fsck output) and what kernel you are using. -Eric From tytso at mit.edu Thu Apr 16 16:51:36 2009 From: tytso at mit.edu (Theodore Tso) Date: Thu, 16 Apr 2009 12:51:36 -0400 Subject: MTBF of Ext3 and Partition Size In-Reply-To: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> References: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> Message-ID: <20090416165136.GN21586@mit.edu> On Thu, Apr 16, 2009 at 07:53:59AM -0400, Kyle Brandt wrote: > > On several of my servers I seem to have a high rate of server crashes do to > file system errors. So I have some questions related to this: > > Is there any Mean Time Between Failure ( MTBF) data for the ext3 > file-system? > > Does increased partition size cause a higher risk of the partition being > corrupted? If so, is there any data on the ratio between partition size and > the likely hood of failure? The probability of these sorts of filesystem problems is going to be dominated by hardware induced corruptions --- so it's not going to make a lot of sense to talk about MTBF failures without having a specific hardware context in mind. If you have lousy memory, or a lousy disk controller cable, or a cable connector which is loose then corruptions will happen often. If you are are located some place where there is a strong alpha particle source, then you will have a much greater percentage chance of bit flips. If you use ECC memory, and do very careful hardware selection, with enterprise-quality disks that trade off disk capacity for a much stronger level of ECC codes, then of course the MBTF will be much less. (For example, there was the imfamous story in the early 1990's when Sun had a spate of bad memory; I think it was ultimately traced to radioactive contamination of the ceramic materials used to make their memory chips; this caused alpha particles to cause "bit flips" and which had the result of making their customers rather antsy, especially since Sun tried todeny there was even a problem for quite some time.) So if you are having a high rate of server crashes, the first thing I would do is to make sure you have the latest distribution updates; it's possible it's caused by a kernel bug that has since been fixed, but it's somewhat unlikely. The next thing I would do is take one of the machines that has been cashing off line, and try running a 36-48 hour memory test. > Does ext3 on hardware raid (10) increase the possibility of file system > corruption? No, it shouldn't --- unless you have a buggy or otherwise dodgy hardware raid controller. - Ted From rwheeler at redhat.com Thu Apr 16 16:55:37 2009 From: rwheeler at redhat.com (Ric Wheeler) Date: Thu, 16 Apr 2009 12:55:37 -0400 Subject: MTBF of Ext3 and Partition Size In-Reply-To: <20090416165136.GN21586@mit.edu> References: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> <20090416165136.GN21586@mit.edu> Message-ID: <49E76309.50802@redhat.com> Theodore Tso wrote: > On Thu, Apr 16, 2009 at 07:53:59AM -0400, Kyle Brandt wrote: >> On several of my servers I seem to have a high rate of server crashes do to >> file system errors. So I have some questions related to this: >> >> Is there any Mean Time Between Failure ( MTBF) data for the ext3 >> file-system? >> >> Does increased partition size cause a higher risk of the partition being >> corrupted? If so, is there any data on the ratio between partition size and >> the likely hood of failure? > > The probability of these sorts of filesystem problems is going to be > dominated by hardware induced corruptions --- so it's not going to > make a lot of sense to talk about MTBF failures without having a > specific hardware context in mind. If you have lousy memory, or a > lousy disk controller cable, or a cable connector which is loose then > corruptions will happen often. If you are are located some place > where there is a strong alpha particle source, then you will have a > much greater percentage chance of bit flips. If you use ECC memory, > and do very careful hardware selection, with enterprise-quality disks > that trade off disk capacity for a much stronger level of ECC codes, > then of course the MBTF will be much less. > > (For example, there was the imfamous story in the early 1990's when > Sun had a spate of bad memory; I think it was ultimately traced to > radioactive contamination of the ceramic materials used to make their > memory chips; this caused alpha particles to cause "bit flips" and > which had the result of making their customers rather antsy, > especially since Sun tried todeny there was even a problem for quite > some time.) > > So if you are having a high rate of server crashes, the first thing I > would do is to make sure you have the latest distribution updates; > it's possible it's caused by a kernel bug that has since been fixed, > but it's somewhat unlikely. The next thing I would do is take one of > the machines that has been cashing off line, and try running a 36-48 > hour memory test. > >> Does ext3 on hardware raid (10) increase the possibility of file system >> corruption? > > No, it shouldn't --- unless you have a buggy or otherwise dodgy > hardware raid controller. > > - Ted > One note is that the file system will often be the first notification that your hardware RAID has done something wrong - you should have a careful look at any logs/errors/etc that your storage maintains for you. Can you share specifics of your system - what is the storage, which kernel, etc? Regards, Ric Ric From lists at nerdbynature.de Fri Apr 17 03:05:42 2009 From: lists at nerdbynature.de (Christian Kujau) Date: Thu, 16 Apr 2009 20:05:42 -0700 (PDT) Subject: about tune2fs In-Reply-To: <1908f30904150657u57cb7b1ar6240f0536acd0a2c@mail.gmail.com> References: <1908f30904150657u57cb7b1ar6240f0536acd0a2c@mail.gmail.com> Message-ID: On Wed, 15 Apr 2009, Jordi Prats wrote: > In which cases using tune2fs should I unmount the filesystem? The manpage mentions that when creating an ext3 journal there's a difference (.journal file vs. internal inode). All other options should be safe to alter with the filesystem mounted. However personally I'd recommend to unmount (any) fileystem first before tuning it - if possible. Christian. -- Crytanalysis doesn't break cryptosystems. Bruce Schneier breaks cryptosystems. From lists at nerdbynature.de Fri Apr 17 03:08:52 2009 From: lists at nerdbynature.de (Christian Kujau) Date: Thu, 16 Apr 2009 20:08:52 -0700 (PDT) Subject: MTBF of Ext3 and Partition Size In-Reply-To: <49E76309.50802@redhat.com> References: <9ee385320904160453qb4ad7a3g17f6a316bff2ecff@mail.gmail.com> <20090416165136.GN21586@mit.edu> <49E76309.50802@redhat.com> Message-ID: On Thu, 16 Apr 2009, Ric Wheeler wrote: > One note is that the file system will often be the first notification that > your hardware RAID has done something wrong Sadly so - while it *should* be the other way around and the RAID controller telling one that something is wrong before harming the filesystem. Sigh... Christian. -- Crytanalysis doesn't break cryptosystems. Bruce Schneier breaks cryptosystems. From rdavidson at bdi.org.au Tue Apr 21 09:17:24 2009 From: rdavidson at bdi.org.au (Robert Davidson) Date: Tue, 21 Apr 2009 19:17:24 +1000 Subject: Superblock missing, FS corruption after power outage Message-ID: <20090421090757.M5309@bdi.org.au> Hi all, I'm wondering if its possible that an ext3 filesystem with journaling enabled and so on (defaults really) could lose its primary superblock and lose, in this case, a whole directory which contained a virtual server. It is possible in this case that a fair chunk of data could have been lost as the RAID card has a 256mb cache with no battery backup for cached data, but it seems somewhat extreme that the superblock went missing (according to mount the filesystem didn't exist, according to fsck.ext3 it had to use a backup superblock), and that a whole directory disappeared and didn't end up in lost+found. During the fsck there were lots of claims of inodes claiming the same data blocks as some other inodes and as such the fsck chose to clone those blocks. In some instances - well, at least one, a directory that contained PostgreSQL databses became a symbolic link to some python file after the fsck. So I'd like to know if all of this is possible just from a power failure, or if there is something else going on that was possibly causing file system corruption that wasn't noticed until the reboot. Thanks. -- Regards, Robert Davidson. From adilger at sun.com Tue Apr 21 12:01:30 2009 From: adilger at sun.com (Andreas Dilger) Date: Tue, 21 Apr 2009 06:01:30 -0600 Subject: Superblock missing, FS corruption after power outage In-Reply-To: <20090421090757.M5309@bdi.org.au> References: <20090421090757.M5309@bdi.org.au> Message-ID: <20090421120130.GR3209@webber.adilger.int> On Apr 21, 2009 19:17 +1000, Robert Davidson wrote: > It is possible in this case that a fair chunk of data could have been lost as > the RAID card has a 256mb cache with no battery backup for cached data, At this point, all bets are off - ext3+jbd expects when the disk reports that data is safe on disk that it really is safe. > seems somewhat extreme that the superblock went missing (according to mount > the filesystem didn't exist, according to fsck.ext3 it had to use a backup > superblock), and that a whole directory disappeared and didn't end up in > lost+found. Well, if you lost 256MB at the start of the filesystem then that could easily contain the whole directory, which might only be a few blocks in size. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From hiren.joshi at totalise.co.uk Sat Apr 11 00:29:49 2009 From: hiren.joshi at totalise.co.uk (Hiren Joshi) Date: Sat, 11 Apr 2009 01:29:49 +0100 Subject: Filesystem gone readonly Message-ID: <49DFE47D.1070008@totalise.co.uk> I've got a really weird situation here. I'm using RHEL 4 and connecting to an EMC storage device using fibre and qla2300. The luns are put into LVM and we have a number of 400G partitions coming off that, I made a snapshot and ran fsck -yn on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 2392655. IGNORED. Inode 2392655, i_blocks is 672, should be 664. Fix? no Inode 29507663 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 29507663. IGNORED. Inode 32918223 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 32918223. IGNORED. Inode 33768943 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33768943. IGNORED. Inode 33771951 has illegal block(s). Clear? no Illegal block #9 (4101620032) in inode 33771951. IGNORED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Illegal block number passed to ext2fs_test_block_bitmap #4101620032 for multiply claimed block map Duplicate/bad block(s) in inode 37063007: 74791522 Duplicate/bad block(s) in inode 37063010: 74791523 74791524 74791525 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 11 inodes containing duplicate/bad blocks.) File /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? no Delete file? no File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 10 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/921/648/pages/html-4532555619 (inode #37063010, mod time Wed Apr 8 18:48:15 2009) /8/004/006/176/558/pages/html-4523408551 (inode #37063007, mod time Wed Apr 8 19:25:11 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Clone duplicate/bad blocks? no Delete file? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? no Free blocks count wrong for group #2282 (0, counted=18). Fix? no Free blocks count wrong (22802537, counted=22802555). Fix? no Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? no Free inodes count wrong for group #643 (15860, counted=15846). Fix? no Free inodes count wrong (47980634, counted=47980620). Fix? no webspace2: ********** WARNING: Filesystem still has errors ********** webspace2: 4448166/52428800 files (3.4% non-contiguous), 82055063/104857600 blocks So, I took the partition offline and ran fsck -yf on it with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 2392655 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 2392655. CLEARED. Inode 2392655, i_blocks is 672, should be 664. Fix? yes Inode 29507663 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 29507663. CLEARED. Inode 32918223 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 32918223. CLEARED. Inode 33768943 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33768943. CLEARED. Inode 33771951 has illegal block(s). Clear? yes Illegal block #9 (4101620032) in inode 33771951. CLEARED. Duplicate blocks found... invoking duplicate block passes. Pass 1B: Rescan for duplicate/bad blocks Duplicate/bad block(s) in inode 37062984: 74791523 Duplicate/bad block(s) in inode 37063012: 74791524 74791525 Duplicate/bad block(s) in inode 37290160: 74791522 Duplicate/bad block(s) in inode 37389296: 74791530 Duplicate/bad block(s) in inode 37389297: 74791531 Duplicate/bad block(s) in inode 37389298: 74791532 74791533 74791537 74791538 74791543 Duplicate/bad block(s) in inode 37389300: 74791544 Duplicate/bad block(s) in inode 37389304: 74791521 Duplicate/bad block(s) in inode 37389305: 74791527 Duplicate/bad block(s) in inode 37389307: 74791520 Duplicate/bad block(s) in inode 37389325: 74791529 74791545 74791547 Duplicate/bad block(s) in inode 37389367: 74791520 74791521 74791522 74791523 74791524 74791525 74791527 74791529 74791530 74791531 74791532 74791533 74791537 74791538 74791543 74791544 74791545 74791547 Pass 1C: Scan directories for inodes with dup blocks. Pass 1D: Reconciling duplicate blocks (There are 12 inodes containing duplicate/bad blocks.) File /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) has 2 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) has 5 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) has 1 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) has 3 duplicate block(s), shared with 1 file(s): /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) Clone duplicate/bad blocks? yes File /8/000/001/064/488/images/4523639157.jpg (inode #37389367, mod time Sun Nov 9 13:49:59 2008) has 18 duplicate block(s), shared with 11 file(s): /8/004/006/875/928/pages/4532395783-Apr__7_2009_12:04:08:120PM (inode #37389325, mod time Tue Apr 7 13:07:38 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:15:43:736AM (inode #37389300, mod time Mon Apr 6 09:15:33 2009) /8/004/006/990/488/pages/4533372293-Apr__6_2009__8:12:30:586AM (inode #37389298, mod time Mon Apr 6 09:12:20 2009) /8/004/006/990/488/pages/4533377531-Apr__6_2009__8:12:10:800AM (inode #37389297, mod time Mon Apr 6 09:12:00 2009) /8/004/006/990/488/pages/4533377522-Apr__6_2009__8:10:49:616AM (inode #37389296, mod time Mon Apr 6 09:11:33 2009) /8/004/006/875/928/pages/4533008375-Apr__7_2009_10:22:30:456AM (inode #37389305, mod time Tue Apr 7 11:26:00 2009) /8/004/006/176/558/pages/4523428551-Apr__9_2009_10:22:19:750AM (inode #37063012, mod time Thu Apr 9 11:25:50 2009) /8/004/006/730/448/pages/html-4530801363 (inode #37062984, mod time Thu Apr 9 15:12:45 2009) /8/004/006/389/058/pages/html-4527727459 (inode #37290160, mod time Thu Apr 9 14:37:51 2009) /8/004/006/931/218/pages/4532668438-Apr__6_2009__8:08:37:743PM (inode #37389304, mod time Mon Apr 6 21:12:07 2009) /8/004/006/875/928/images/4525650407_pre.jpg (inode #37389307, mod time Tue Apr 7 11:29:54 2009) Duplicated blocks already reassigned or cloned. Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #9. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -4806293 Fix? yes Free blocks count wrong for group #2 (16059, counted=16040). Fix? yes Free blocks count wrong for group #146 (7244, counted=7245). Fix? yes Free blocks count wrong for group #2282 (0, counted=18). Fix? yes Inode bitmap differences: -10549863 -10549865 -(10549871--10549873) -(10549876--10549879) -10549883 -(10549885--10549888) Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4451892/52428800 files (3.5% non-contiguous), 82037602/104857600 blocks Then I saw this in /var/log/messages: Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 4096 Apr 9 21:53:56 kernel: Aborting journal on device dm-4. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 8192 Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_readdir: directory #29507663 contains a hole at offset 12288 Apr 9 21:53:56 kernel: ext3_abort called. Apr 9 21:53:56 kernel: EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal Apr 9 21:53:56 kernel: Remounting filesystem read-only So I took it off-line and ran fsck -yf again with the following output: fsck 1.35 (28-Feb-2004) e2fsck 1.35 (28-Feb-2004) webspace2: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Directory inode 29507663 has an unallocated block #1. Allocate? yes Directory inode 29507663 has an unallocated block #2. Allocate? yes Directory inode 29507663 has an unallocated block #3. Allocate? yes Directory inode 29507663 has an unallocated block #4. Allocate? yes Directory inode 29507663 has an unallocated block #5. Allocate? yes Directory inode 29507663 has an unallocated block #6. Allocate? yes Directory inode 29507663 has an unallocated block #7. Allocate? yes Directory inode 29507663 has an unallocated block #8. Allocate? yes Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong for group #2 (16029, counted=16021). Fix? yes Free blocks count wrong (22763960, counted=22763952). Fix? yes webspace2: ***** FILE SYSTEM WAS MODIFIED ***** webspace2: 4454345/52428800 files (3.5% non-contiguous), 82093648/104857600 blocks What on earth is going on?! Can I expect the partition to go read-only again? Also, over the last week a number of other partitions from the same lvm and EMC setup and they all came out with: Illegal block #9 (4101620032) in inode 803439. CLEARED. In the fsck...... From articpenguin3800 at gmail.com Fri Apr 17 03:19:22 2009 From: articpenguin3800 at gmail.com (John Nelson) Date: Thu, 16 Apr 2009 23:19:22 -0400 Subject: E2fsck and large file Message-ID: <49E7F53A.5090509@gmail.com> How big is a file that e2fsck considers it to be a large file? 814611 blocks used (42.79%) 0 bad blocks 1 large file <----- that Thanks John Nelson From lists at nerdbynature.de Sat Apr 25 03:38:34 2009 From: lists at nerdbynature.de (Christian Kujau) Date: Fri, 24 Apr 2009 20:38:34 -0700 (PDT) Subject: Filesystem gone readonly In-Reply-To: <49DFE47D.1070008@totalise.co.uk> References: <49DFE47D.1070008@totalise.co.uk> Message-ID: On Sat, 11 Apr 2009, Hiren Joshi wrote: > The luns are put into LVM and we have a number of 400G partitions coming off > that, I made a snapshot and ran fsck -yn on it with the following output: Why did you run fsck? Did you suspect filesystem errors, i.e. were there other reasons to run fsck in the first place? (unclean shutdown, power outages, etc.) > fsck 1.35 (28-Feb-2004) > e2fsck 1.35 (28-Feb-2004) e2fsprogs-1.41.5 has been released[0] today, you may want to upgrade these before doing anything to the filesystem. Also, was there anything devices related in the kernel logs? Can you read off the raw LVM device w/o errors? When did the errors start / when was your last fsck with no errors reported? It's strange that the filesystem starts to show errors "out of the blue", so I'm fishing for hardware related issues, because: if it's software related, you'd have to upgrade the ext3 "driver", i.e. the kernel - probably not possible for w/o breaking ot of RHEL4. Christian. [0] http://e2fsprogs.sf.net/ -- Bruce Schneier isn't fooled by decoy states. From sandeen at redhat.com Sat Apr 25 04:01:56 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Fri, 24 Apr 2009 23:01:56 -0500 Subject: E2fsck and large file In-Reply-To: <49E7F53A.5090509@gmail.com> References: <49E7F53A.5090509@gmail.com> Message-ID: <49F28B34.8030307@redhat.com> John Nelson wrote: > How big is a file that e2fsck considers it to be a large file? > > 814611 blocks used (42.79%) > 0 bad blocks > 1 large file <----- that > > > Thanks > John Nelson if (LINUX_S_ISREG(inode->i_mode) && (inode->i_size_high || inode->i_size & 0x80000000UL)) ctx->large_files++; 2G or greater. -Eric From sandeen at redhat.com Sat Apr 25 04:04:14 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Fri, 24 Apr 2009 23:04:14 -0500 Subject: Filesystem gone readonly In-Reply-To: <49DFE47D.1070008@totalise.co.uk> References: <49DFE47D.1070008@totalise.co.uk> Message-ID: <49F28BBE.7030500@redhat.com> Hiren Joshi wrote: > I've got a really weird situation here. I'm using RHEL 4 and connecting > to an EMC storage device using fibre and qla2300. > > The luns are put into LVM and we have a number of 400G partitions coming > off that, I made a snapshot and ran fsck -yn on it with the following > output: > > fsck 1.35 (28-Feb-2004) > e2fsck 1.35 (28-Feb-2004) > Pass 1: Checking inodes, blocks, and sizes > Inode 2392655 has illegal block(s). Clear? no > > Illegal block #9 (4101620032) in inode 2392655. IGNORED. > Inode 2392655, i_blocks is 672, should be 664. Fix? no Is there any chance that some other node on the san has this lun mounted, or is writing to it? -Eric From lists at nerdbynature.de Sat Apr 25 07:19:57 2009 From: lists at nerdbynature.de (Christian Kujau) Date: Sat, 25 Apr 2009 00:19:57 -0700 (PDT) Subject: E2fsck and large file In-Reply-To: <49F28B34.8030307@redhat.com> References: <49E7F53A.5090509@gmail.com> <49F28B34.8030307@redhat.com> Message-ID: On Fri, 24 Apr 2009, Eric Sandeen wrote: >> How big is a file that e2fsck considers it to be a large file? > if (LINUX_S_ISREG(inode->i_mode) && > (inode->i_size_high || inode->i_size & 0x80000000UL)) > ctx->large_files++; > 2G or greater. I was tempted to reply the same, but: # mkfs.ext4 -q /dev/md0 # fsck.ext4 -fv /dev/md0 e2fsck 1.41.5 (23-Apr-2009) [...] 11 inodes used (0.00%) 0 non-contiguous files (0.0%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 1 33640 blocks used (3.45%) 0 bad blocks 1 large file <--- so, there's at least one file (inode?) considered a large file right from the beginning? C. -- A vigenere cipher with the Key "BRUCESCHNEIER" is in fact unbreakable. From cax0cn at gmail.com Sat Apr 25 14:09:21 2009 From: cax0cn at gmail.com (Joseph Chen) Date: Sat, 25 Apr 2009 22:09:21 +0800 Subject: Filesystem gone readonly In-Reply-To: <49F28BBE.7030500@redhat.com> References: <49DFE47D.1070008@totalise.co.uk> <49F28BBE.7030500@redhat.com> Message-ID: <8d423b320904250709uf07dc44mf14a3cf856861697@mail.gmail.com> Hi, Hiren, I agree with Christian, before running fsck, you should have a look on the hardware status. If it's not a file system issue, running fsck may cause terrible amount of files lost. Additionally, you need to create a backup with the help of dd before any fsck, so that you are able to recover some lost files. Good luck, Joe.c @ http://admon.org/ On Sat, Apr 25, 2009 at 12:04 PM, Eric Sandeen wrote: > Hiren Joshi wrote: > > I've got a really weird situation here. I'm using RHEL 4 and connecting > > to an EMC storage device using fibre and qla2300. > > > > The luns are put into LVM and we have a number of 400G partitions coming > > off that, I made a snapshot and ran fsck -yn on it with the following > > output: > > > > fsck 1.35 (28-Feb-2004) > > e2fsck 1.35 (28-Feb-2004) > > Pass 1: Checking inodes, blocks, and sizes > > Inode 2392655 has illegal block(s). Clear? no > > > > Illegal block #9 (4101620032) in inode 2392655. IGNORED. > > Inode 2392655, i_blocks is 672, should be 664. Fix? no > > Is there any chance that some other node on the san has this lun > mounted, or is writing to it? > > -Eric > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users > -- Sponser and operater: Linux monitoring solution: http://admon.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Sat Apr 25 16:42:50 2009 From: tytso at mit.edu (Theodore Tso) Date: Sat, 25 Apr 2009 12:42:50 -0400 Subject: E2fsck and large file In-Reply-To: <49E7F53A.5090509@gmail.com> References: <49E7F53A.5090509@gmail.com> Message-ID: <20090425164250.GI13608@mit.edu> On Thu, Apr 16, 2009 at 11:19:22PM -0400, John Nelson wrote: > How big is a file that e2fsck considers it to be a large file? > > 814611 blocks used (42.79%) > 0 bad blocks > 1 large file <----- that 2GB. This is the size at which point the LARGEFILE read-only compat flag is set. It's there so that really old kernels (pre Linux version 2.4) don't try to modify, and thus corrupt filesystems that have > 2GB files, since Linux 2.2 kernels didn't support files larger than 2 gigabytes. It's largely a historical issue in this day and age, but it does have a very specific meaning. - Ted From tytso at mit.edu Sat Apr 25 16:47:17 2009 From: tytso at mit.edu (Theodore Tso) Date: Sat, 25 Apr 2009 12:47:17 -0400 Subject: E2fsck and large file In-Reply-To: References: <49E7F53A.5090509@gmail.com> <49F28B34.8030307@redhat.com> Message-ID: <20090425164717.GJ13608@mit.edu> On Sat, Apr 25, 2009 at 12:19:57AM -0700, Christian Kujau wrote: > On Fri, 24 Apr 2009, Eric Sandeen wrote: > >> How big is a file that e2fsck considers it to be a large file? > > if (LINUX_S_ISREG(inode->i_mode) && > > (inode->i_size_high || inode->i_size & 0x80000000UL)) > > ctx->large_files++; > > 2G or greater. > > I was tempted to reply the same, but: > > # mkfs.ext4 -q /dev/md0 > # fsck.ext4 -fv /dev/md0 > e2fsck 1.41.5 (23-Apr-2009) > [...] > 11 inodes used (0.00%) > 0 non-contiguous files (0.0%) > 0 non-contiguous directories (0.0%) > # of inodes with ind/dind/tind blocks: 0/0/0 > Extent depth histogram: 1 > 33640 blocks used (3.45%) > 0 bad blocks > 1 large file <--- so, there's at least one file (inode?) > considered a large file right from the > beginning? It's the resize inode; because of how it's built, it appears to be a very large sparse file. - Ted From Ralf.Hildebrandt at charite.de Sun Apr 26 11:10:44 2009 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Sun, 26 Apr 2009 13:10:44 +0200 Subject: ext4 mount fails with "resize inode not valid" after a reboot Message-ID: <20090426111044.GC5116@charite.de> With kernel 2.6.30-rc2-git6 and prior I am having problems mounting ext4 partitions after reboot. A successful mount looks like this: /dev/cciss/c0d0p8 on /squid-cache0 type ext4 (rw,noexec,nodev,noatime,data=writeback,errors=panic) /dev/cciss/c0d0p9 on /squid-cache1 type ext4 (rw,noexec,nodev,noatime,data=writeback,errors=panic) /dev/cciss/c0d0p10 on /squid-data type ext4 (rw,noexec,nodev,noatime,data=writeback,errors=panic) Dmesg reports: [ 242.297469] EXT4-fs: barriers enabled [ 242.302098] kjournald2 starting: pid 2509, dev cciss!c0d0p8:8, commit interval 5 seconds [ 242.311272] EXT4 FS on cciss!c0d0p8, internal journal on cciss!c0d0p8:8 [ 242.311331] EXT4-fs: delayed allocation enabled [ 242.311386] EXT4-fs: file extents enabled [ 242.311562] EXT4-fs: mballoc enabled [ 242.311627] EXT4-fs: mounted filesystem cciss!c0d0p8 with writeback data mode [ 242.329741] EXT4-fs: barriers enabled [ 242.337949] kjournald2 starting: pid 2510, dev cciss!c0d0p9:8, commit interval 5 seconds [ 242.343248] EXT4 FS on cciss!c0d0p9, internal journal on cciss!c0d0p9:8 [ 242.343317] EXT4-fs: delayed allocation enabled [ 242.343383] EXT4-fs: file extents enabled [ 242.343588] EXT4-fs: mballoc enabled [ 242.343662] EXT4-fs: mounted filesystem cciss!c0d0p9 with writeback data mode [ 242.366470] EXT4-fs: barriers enabled [ 242.371982] kjournald2 starting: pid 2511, dev cciss!c0d0p10:8, commit interval 5 seconds [ 242.379278] EXT4 FS on cciss!c0d0p10, internal journal on cciss!c0d0p10:8 [ 242.379345] EXT4-fs: delayed allocation enabled [ 242.379410] EXT4-fs: file extents enabled [ 242.379594] EXT4-fs: mballoc enabled [ 242.379662] EXT4-fs: mounted filesystem cciss!c0d0p10 with writeback data mode During boot I get these errors (Resize inod not valid): http://www.arschkrebs.de/images/ext4-1.png and the manual fsck reports: http://www.arschkrebs.de/images/ext4-2.png WTF is going on? This hapens even when I just recreate the filesystem. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12200 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 Ralf.Hildebrandt at charite.de | http://www.charite.de From sandeen at redhat.com Sun Apr 26 14:10:21 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Sun, 26 Apr 2009 09:10:21 -0500 Subject: ext4 mount fails with "resize inode not valid" after a reboot In-Reply-To: <20090426111044.GC5116@charite.de> References: <20090426111044.GC5116@charite.de> Message-ID: <49F46B4D.6090409@redhat.com> Ralf Hildebrandt wrote: > With kernel 2.6.30-rc2-git6 and prior I am having problems mounting > ext4 partitions after reboot. > ... > WTF is going on? This hapens even when I just recreate the filesystem. So mkfs followed immediately by fsck shows it, or only with a mount in between? Does e2fsprogs-1.41.5 (released about 2 days ago) still show the problem? If so can you mkfs & provide the dumpe2fs -h output right after the mkfs, perhaps there is something unique about your fs geometry that trips a bug. -Eric From Ralf.Hildebrandt at charite.de Sun Apr 26 14:15:18 2009 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Sun, 26 Apr 2009 16:15:18 +0200 Subject: ext4 mount fails with "resize inode not valid" after a reboot In-Reply-To: <49F46B4D.6090409@redhat.com> References: <20090426111044.GC5116@charite.de> <49F46B4D.6090409@redhat.com> Message-ID: <20090426141518.GA6243@charite.de> * Eric Sandeen : > Ralf Hildebrandt wrote: > > With kernel 2.6.30-rc2-git6 and prior I am having problems mounting > > ext4 partitions after reboot. > > > ... > > > WTF is going on? This hapens even when I just recreate the filesystem. > > So mkfs followed immediately by fsck shows it, or only with a mount in > between? Does e2fsprogs-1.41.5 (released about 2 days ago) still show > the problem? If so can you mkfs & provide the dumpe2fs -h output right > after the mkfs, perhaps there is something unique about your fs geometry > that trips a bug. Right now I updated e2fsprogs-1.41.5 to and rebooted all my 4 machines and the error did not occur. I'll have to report back. -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12200 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 Ralf.Hildebrandt at charite.de | http://www.charite.de From hiren.joshi at totalise.co.uk Mon Apr 27 09:47:11 2009 From: hiren.joshi at totalise.co.uk (Hiren Joshi) Date: Mon, 27 Apr 2009 10:47:11 +0100 Subject: Filesystem gone readonly In-Reply-To: <8d423b320904250709uf07dc44mf14a3cf856861697@mail.gmail.com> References: <49DFE47D.1070008@totalise.co.uk> <49F28BBE.7030500@redhat.com> <8d423b320904250709uf07dc44mf14a3cf856861697@mail.gmail.com> Message-ID: <000301c9c71d$25996a40$2002a8c0@greekattic.local> I did just this and it looks like the memory is shot so on one of the nodes so I'm doing all the work on the other node..... thanks for the pointers! Josh. _____ From: ext3-users-bounces at redhat.com [mailto:ext3-users-bounces at redhat.com] On Behalf Of Joseph Chen Sent: 25 April 2009 15:09 To: Eric Sandeen Cc: ext3-users at redhat.com Subject: Re: Filesystem gone readonly Hi, Hiren, I agree with Christian, before running fsck, you should have a look on the hardware status. If it's not a file system issue, running fsck may cause terrible amount of files lost. Additionally, you need to create a backup with the help of dd before any fsck, so that you are able to recover some lost files. Good luck, Joe.c @ http://admon.org/ On Sat, Apr 25, 2009 at 12:04 PM, Eric Sandeen wrote: Hiren Joshi wrote: > I've got a really weird situation here. I'm using RHEL 4 and connecting > to an EMC storage device using fibre and qla2300. > > The luns are put into LVM and we have a number of 400G partitions coming > off that, I made a snapshot and ran fsck -yn on it with the following > output: > > fsck 1.35 (28-Feb-2004) > e2fsck 1.35 (28-Feb-2004) > Pass 1: Checking inodes, blocks, and sizes > Inode 2392655 has illegal block(s). Clear? no > > Illegal block #9 (4101620032) in inode 2392655. IGNORED. > Inode 2392655, i_blocks is 672, should be 664. Fix? no Is there any chance that some other node on the san has this lun mounted, or is writing to it? -Eric _______________________________________________ Ext3-users mailing list Ext3-users at redhat.com https://www.redhat.com/mailman/listinfo/ext3-users -- Sponser and operater: Linux monitoring solution: http://admon.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ranjithkannikara at gmail.com Wed Apr 29 00:45:59 2009 From: ranjithkannikara at gmail.com (ranjith kannikara) Date: Wed, 29 Apr 2009 06:15:59 +0530 Subject: EXT3 file recovery Message-ID: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> HiI am a pre-final year computer science engineering student. We are doing an open source project to make an application to recover the deleted files in an ext3 filesystem. In between we are having some doubts can any one make some clarification? * Is it possible to manipulate an inode. ie is it possible if I want to edit the content of an inode . ie I know what should be the content of an inode and I want to change the content of the inode manually is there any way to do this..? or * Can some one say from where I can get the procedure for deletion in an ext3 filesystem. ie i would like to know where is the procedure for deletion in an ext3 filesystem is located. is it there in the system itself or can I get the exact procedure online...? Expecting replies Ranju. -- http://www.ranjithkannikara.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruno at wolff.to Wed Apr 29 03:08:24 2009 From: bruno at wolff.to (Bruno Wolff III) Date: Tue, 28 Apr 2009 22:08:24 -0500 Subject: EXT3 file recovery In-Reply-To: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> References: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> Message-ID: <20090429030824.GA9485@wolff.to> On Wed, Apr 29, 2009 at 06:15:59 +0530, ranjith kannikara wrote: > HiI am a pre-final year computer science engineering student. We are doing > an open source project to make an application to recover the deleted files > in an ext3 filesystem. In between we are having some doubts can any one make > some clarification? A good start would probably be to look at what ext3grep does. From rbock at eudoxos.de Wed Apr 29 10:56:00 2009 From: rbock at eudoxos.de (Roland Bock) Date: Wed, 29 Apr 2009 12:56:00 +0200 Subject: Ext3: Faster deletion for big files? Message-ID: <49F83240.3000703@eudoxos.de> Hi, is there a way to speed up the deletion of big files on an Ext3 file system? I have to delete two rather big files (200GB each) once per day. The deletion of these two files takes almost 15 minutes on a Raid6 with 16 Disks. During this time the IO system of the machine is pretty much occupied... Any suggestions? Regards, Roland From adilger at sun.com Wed Apr 29 11:39:07 2009 From: adilger at sun.com (Andreas Dilger) Date: Wed, 29 Apr 2009 05:39:07 -0600 Subject: EXT3 file recovery In-Reply-To: <20090429030824.GA9485@wolff.to> References: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> <20090429030824.GA9485@wolff.to> Message-ID: <20090429113907.GD3209@webber.adilger.int> On Apr 28, 2009 22:08 -0500, Bruno Wolff III wrote: > On Wed, Apr 29, 2009 at 06:15:59 +0530, > ranjith kannikara wrote: > > HiI am a pre-final year computer science engineering student. We are doing > > an open source project to make an application to recover the deleted files > > in an ext3 filesystem. In between we are having some doubts can any one make > > some clarification? > > A good start would probably be to look at what ext3grep does. A good finish would be to take ext3grep and enhance it to do more than it currently does (e.g. add ext4 support) instead of making a duplicate tool. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. From tytso at mit.edu Wed Apr 29 13:29:53 2009 From: tytso at mit.edu (Theodore Tso) Date: Wed, 29 Apr 2009 09:29:53 -0400 Subject: EXT3 file recovery In-Reply-To: <20090429113907.GD3209@webber.adilger.int> References: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> <20090429030824.GA9485@wolff.to> <20090429113907.GD3209@webber.adilger.int> Message-ID: <20090429132953.GA14264@mit.edu> On Wed, Apr 29, 2009 at 05:39:07AM -0600, Andreas Dilger wrote: > On Apr 28, 2009 22:08 -0500, Bruno Wolff III wrote: > > On Wed, Apr 29, 2009 at 06:15:59 +0530, > > ranjith kannikara wrote: > > > HiI am a pre-final year computer science engineering student. We are doing > > > an open source project to make an application to recover the deleted files > > > in an ext3 filesystem. In between we are having some doubts can any one make > > > some clarification? > > > > A good start would probably be to look at what ext3grep does. > > A good finish would be to take ext3grep and enhance it to do more than it > currently does (e.g. add ext4 support) instead of making a duplicate tool. Note that converting it to use libext2fs would for more of its low-level functions would be one of the easier ways of accomplishing this, since all of the low-level extent manipulation functions are coded in libext2fs. On the other hand, you'll learn more about the low-level filesystem layout if you do it yourself; sone one is probably better from an engineering perspective in terms of making a robust and useful and long-term more easily maintained tool --- and the other approach is better from a pedagogical perspective. - Ted From tytso at mit.edu Wed Apr 29 13:34:32 2009 From: tytso at mit.edu (Theodore Tso) Date: Wed, 29 Apr 2009 09:34:32 -0400 Subject: Ext3: Faster deletion for big files? In-Reply-To: <49F83240.3000703@eudoxos.de> References: <49F83240.3000703@eudoxos.de> Message-ID: <20090429133432.GB14264@mit.edu> On Wed, Apr 29, 2009 at 12:56:00PM +0200, Roland Bock wrote: > Hi, > > is there a way to speed up the deletion of big files on an Ext3 file > system? > > I have to delete two rather big files (200GB each) once per day. > > The deletion of these two files takes almost 15 minutes on a Raid6 with > 16 Disks. During this time the IO system of the machine is pretty much > occupied... Unfortunately, no, not really. Upgrading to ext4 will definitely help. If you're not willing to do change filesystems, another approach might be to put the two files on a separate LVM volume, point symlinks at them, and then recreate the volume each day. - Ted From ranjithkannikara at gmail.com Wed Apr 29 15:56:29 2009 From: ranjithkannikara at gmail.com (ranjith kannikara) Date: Wed, 29 Apr 2009 21:26:29 +0530 Subject: EXT3 file recovery In-Reply-To: <20090429132953.GA14264@mit.edu> References: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> <20090429030824.GA9485@wolff.to> <20090429113907.GD3209@webber.adilger.int> <20090429132953.GA14264@mit.edu> Message-ID: <20aa8c370904290856o4da24a9fxd7aa85c7bd3d8db1@mail.gmail.com> Hi all, Thanks to get such fast and useful replies. I would like to ask one more that Can some one tell me where to find the code for the procedure 'delete'. ie the code that runs when a file is deleted from an ext3 filesystem.. where is this code placed. Regards Ranju. On Wed, Apr 29, 2009 at 6:59 PM, Theodore Tso wrote: > On Wed, Apr 29, 2009 at 05:39:07AM -0600, Andreas Dilger wrote: > > On Apr 28, 2009 22:08 -0500, Bruno Wolff III wrote: > > > On Wed, Apr 29, 2009 at 06:15:59 +0530, > > > ranjith kannikara wrote: > > > > HiI am a pre-final year computer science engineering student. We are > doing > > > > an open source project to make an application to recover the deleted > files > > > > in an ext3 filesystem. In between we are having some doubts can any > one make > > > > some clarification? > > > > > > A good start would probably be to look at what ext3grep does. > > > > A good finish would be to take ext3grep and enhance it to do more than it > > currently does (e.g. add ext4 support) instead of making a duplicate > tool. > > Note that converting it to use libext2fs would for more of its > low-level functions would be one of the easier ways of accomplishing > this, since all of the low-level extent manipulation functions are > coded in libext2fs. On the other hand, you'll learn more about the > low-level filesystem layout if you do it yourself; sone one is > probably better from an engineering perspective in terms of making a > robust and useful and long-term more easily maintained tool --- and > the other approach is better from a pedagogical perspective. > > - Ted > -- http://www.ranjithkannikara.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Wed Apr 29 19:51:11 2009 From: tytso at mit.edu (Theodore Tso) Date: Wed, 29 Apr 2009 15:51:11 -0400 Subject: EXT3 file recovery In-Reply-To: <20aa8c370904290856o4da24a9fxd7aa85c7bd3d8db1@mail.gmail.com> References: <20aa8c370904281745u16ae3ac0ud8f7f0e3694d8ab2@mail.gmail.com> <20090429030824.GA9485@wolff.to> <20090429113907.GD3209@webber.adilger.int> <20090429132953.GA14264@mit.edu> <20aa8c370904290856o4da24a9fxd7aa85c7bd3d8db1@mail.gmail.com> Message-ID: <20090429195111.GB17797@mit.edu> On Wed, Apr 29, 2009 at 09:26:29PM +0530, ranjith kannikara wrote: > Hi all, > > Thanks to get such fast and useful replies. I would like to ask one > more that Can some one tell me where to find the code for the > procedure 'delete'. ie the code that runs when a file is deleted > from an ext3 filesystem.. where is this code placed. I'm not sure why you would find that all that helpful, but please see ext3_unlink(), ext3_delete_entry, ext3_delete_inode(), ext3_truncate(), ext3_orphan_add(), and ext3_orphan_cleanup(). The actual work of deleting a file is scattered across all of these functions, plus work at the VFS layer. It may be simpler for you to see what happens when you delete a file by looking at the userspace code in e2fsprogs. If you look at do_rm() in debugfs/debugfs.c, and then trace through its function calls, starting with unlink_file_by_name() and kill_file_by_inode() in debugfs.c, and seeing how it calls out to the libext2fs functions ext2fs_unlink(), ext2fs_block_iterate(), and in the callback release_blocks_proc() passed to ext2fs_block_iterate, ext2fs_block_alloc_stas(), that might be easier to understand. Or, of course, I recommend you look at the various papers that talk about the ext3 and ext4 filesystem formats at: http://ext4.wiki.kernel.org/index.php/Publications Best regards, - Ted From number9652 at yahoo.com Thu Apr 30 20:17:02 2009 From: number9652 at yahoo.com (Number9652) Date: Thu, 30 Apr 2009 13:17:02 -0700 (PDT) Subject: Undeletion utililty for ext3/4 Message-ID: <938911.2430.qm@web43504.mail.sp1.yahoo.com> I have recently released a project on sourceforge ( http://extundelete.sourceforge.net ) that can undelete a file from an ext3 or ext4 partition. It uses code from ext3grep to parse command-line options, and uses libext2fs to read the partitions. Instead of reading the entire partition, as ext3grep does, it reads only the journal file and is able to restore a deleted file from the information there and in (possibly deleted) directory blocks. I hope it is of some use. -N From ramesh at arasan.com Wed Apr 29 05:33:26 2009 From: ramesh at arasan.com (Ramesh) Date: Wed, 29 Apr 2009 11:03:26 +0530 (IST) Subject: File System Selection Message-ID: <1240983206.446412847@192.168.1.35> Hi All, I am developing a SD Block Driver. As per old specification (SD Spec 2.0 ) Maximum size of SD memory card is 32 GB. - We used ext2 file system. By referring the new Specification (SD Spec 3.0) SD memory card size is reached upto and including 2TB (Terra Byte) - Block size strictly limited to 512 only (as per specification). My Questions. 1. For 2TB disk with Block size 512, Which file system is preferred (ext3/ext4) 2. In a 32 bit machine, If I installed the Fedora 10 ( having ext4), am I able to use it as effectively ( for the maximum disk/file size usage). To utilize 2TB or more size hard disk, is this allowable to use 32 bit machine with Ext4 fs? Thanks in advance. Regards, Ramesh ATTENTION: The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited by law. If you have received this message in error, please immediately notify the sender and/or Arasan Chip Systems, Inc. by telephone at (408) 282-1600 and delete or destroy any copy of this message.