From markjballard at googlemail.com Thu Aug 29 11:56:35 2013 From: markjballard at googlemail.com (Mark Ballard) Date: Thu, 29 Aug 2013 12:56:35 +0100 Subject: ext3 / ext4 on USB flash drive? Message-ID: I think this is really an attempt at user feedback, rather than user discussion. But there's no such thing as a user-feedback mail list. Nevertheless, others may find this pertinent: why doesn't mke2fs handle USB's competently? And if it does, why doesn't it reassure me so? And how can I handle a linux-formatted USB flash drive in the absence of my system giving me any guidance? I cannot accept the usual advice you see in the forums: simply format it FAT16 or whatever. I need it to use a linux file system. But linux may have problems, apparently, with the device: http://lwn.net/Articles/428584/ Plenty of people say in the forums that ext3 and ext4 will wear your USB out quickly, effectively thrashing it with their journaling. But the above-linked article suggests it may depend on the device. Some people say ext4 is simply okay, some that it is actually advisable to use it. I'm none the wiser. If it is true that ext4 will thrash your USB then it could important if it is also true that, for example, the life-expectancy of a USB can really as low as 10,000 writes: http://linux.koolsolutions.com/2009/01/29/installing-linux-on-usb-part-3-which-linux-filesystem-for-usb-devices/. I spoke to Integral Memory, the manufacturer of my USB drive. Their technical support line said their device had a life expectancy of 1000 writes, and came with a two-year warranty. He didn't know what he was talking about. None of Integral's publically available spec-sheets tell you anything useful. They give an approximate write-speed and that's about it. Nothing about block size, whether optimized for linear write or not, what method of wear distribution they use - all the things you are led to believe may be important in deciding whether you should format your USB ext4 or not. Of course, I shouldn't have to know about any of these things as a user anyway. But I'm left with no choice but to lower myself into this arcane realm and get in a tangle. One of the things I'm led to believe I must consider when formatting a USB drive is its physical block size. This appears to be something of a mystery, but someone reckons he has worked out a simple method of discerning it: http://kim.oyhus.no/FlashBlockSize.html. Using that method, and assuming Kim G. S. ?yhus is correct, my drive seems to have 4k block size. Now perhaps if I format my drive with a filesystem that uses a 4k block size I might stop it being mashed by ext4. Perhaps. It does seem strange to me though that mke2fs doesn't already do this for me: discerns what's going on underneath the casing of my USB, adjust its formatting parameters accordingly - and tell me what its doing so I can be reassured. Or that the help pages at least tell me definitively whether and in what circumstances it can be used with USB so I don't have to waste so much time trying to unravel the mystery by going like a pinball from one contradictory, incomplete or well-meaning nincompoop forum message to another; and so I don't mash my USB, and lose my data, because I should have listened to the pessimists and not put an ext4 filesystem on my USB. I actually don't want to believe them. But they shout loudest. mb. From tytso at mit.edu Thu Aug 29 15:46:57 2013 From: tytso at mit.edu (Theodore Ts'o) Date: Thu, 29 Aug 2013 11:46:57 -0400 Subject: ext3 / ext4 on USB flash drive? In-Reply-To: References: Message-ID: <20130829154657.GC30918@thunk.org> On Thu, Aug 29, 2013 at 12:56:35PM +0100, Mark Ballard wrote: > I think this is really an attempt at user feedback, rather than user > discussion. But there's no such thing as a user-feedback mail list. > > Nevertheless, others may find this pertinent: why doesn't mke2fs > handle USB's competently? And if it does, why doesn't it reassure me > so? And how can I handle a linux-formatted USB flash drive in the > absence of my system giving me any guidance? The problem is that it really depends on the quality of the flash drive. Two main things drive this: 1) Whether the flash used in the drive is SLC. MLC, or TLC. This will tell you how many write cycles the flash can support. This is where you will see numbers such as 10,000 write cycles, all the way down to 1,000 write depending on the quality of the flash used. 2) The quality of the Flash Translation Layer (FTL). The important thing to remember here is that not all parts of the storage device will get used to the same amount. Some files will rarely change once they are installed on the device; others will change a huge amount. So a good Flash Translation Layer will spread out the wear so that if you have a 120GB device, with flash cells that support 10,000 write cycles, that ideally you can support up to 10,000 times 120GB == 120 TB worth of writes. Of course, that assumes 100% efficiency, which is unrealistic. But some FTL's are incredibly bad, and in the worst case, if they aren't spreading the wear around at all, if you write a single block 1,000 times, and the flash only supports 1,000 write cycles, then that block will go bad. Unfortunately, the cheapest USB flash devices that you find in the bargain bin at the checkout counter of the Micro Center tend to use the crappiest flash media possible, and the crappist FTL's. Some even will only have 512MB of flash even though it is advertised as having 8GB of flash, such that if you ever write more than 512MB worth of files on the flash, you will start losing data. (Fortunately most people don't actually use the full capacity of their USB sticks, which is why the manufacturers can get away with this kind of fraud.) So this is fundamentally a problem with the quality of the hardware, and that's not something the file system can really compensate for. And there's no way to tell whether a particular USB device has has a high quality flash device, or is a craptastic flash device. It's not like we can query the device for "I ripped off the purchaser" bit. :-) Regards, - Ted From sandeen at redhat.com Thu Aug 29 16:31:03 2013 From: sandeen at redhat.com (Eric Sandeen) Date: Thu, 29 Aug 2013 11:31:03 -0500 Subject: ext3 / ext4 on USB flash drive? In-Reply-To: <20130829154657.GC30918@thunk.org> References: <20130829154657.GC30918@thunk.org> Message-ID: <521F7747.2000003@redhat.com> On 8/29/13 10:46 AM, Theodore Ts'o wrote: > So this is fundamentally a problem with the quality of the hardware, > and that's not something the file system can really compensate for. > And there's no way to tell whether a particular USB device has has a > high quality flash device, or is a craptastic flash device. It's not > like we can query the device for "I ripped off the purchaser" bit. :-) Yes, this is the fundamental problem. What's the best setting for the random blob of silicon on the end of your usb plug? e2fsprogs can't call tech support, even to get bad answers. We'd like to help in software, but we can't; we have no reliable way of knowing most of the necessary details of this class of hardware. It's not exported in any way. So unfortunately we are as in the dark as you are in this case. -Eric From prichard at med.wayne.edu Fri Aug 30 01:48:42 2013 From: prichard at med.wayne.edu (Richards, Paul Franklin) Date: Fri, 30 Aug 2013 01:48:42 +0000 Subject: Strange fsck.ext3 behavior - infinite loop Message-ID: <90D96432E685D84E9A62AF83962A689737E7AA68@MED-CORE07A.med.wayne.edu> Greetings! Need your help fellow penguins! Strange behavior with fsck.ext3: how to remove a long orphaned inode list? After copying data over from one old RAID to another new RAID with rsync, the dump command would not complete because of filesystem errors on the new RAID. So I ran fsck.ext3 with the -y option and it would just run in an infinite loop restarting itself and then trying to correct the same inodes over and over again. Some of the errors were lots of orphaned inodes. So I ran a tar tape backup and reformatted it with mkfs.ext3. After restoring the tar backup, I got the same errors when I ran fsck.ext3 -f. Again fsck.ext3 -y would run in an infinite loop trying to correct the problem. So I formatted it again and ran fsck immediately afterwards and it's still detecting corrupted orphaned inode lists. This is on what should be a pristine filesystem with basically no files. Is this a hardware problem with the RAID or a bug somewhere or just normal behavior? [root at myhost /]# mkfs.ext3 /dev/sda1 mke2fs 1.35 (28-Feb-2004) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 268435456 inodes, 536868352 blocks 26843417 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 16384 block groups 32768 blocks per group, 32768 fragments per group 16384 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 24 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. [root at myhost /]# fsck fsck fsck.cramfs fsck.ext2 fsck.ext3 fsck.msdos fsck.vfat [root at myhost /]# man fsck.ext3 [root at myhost /]# fsck.ext3 -C /root/completion /dev/sda1 e2fsck 1.35 (28-Feb-2004) /dev/sda1: clean, 11/268435456 files, 8450084/536868352 blocks [root at myhost /]# fsck.ext3 -f -C /root/completion /dev/sda1 e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inodes that were part of a corrupted orphan linked list found. Fix? yes Inode 26732609 was part of the orphaned inode list. FIXED. Inode 26732609 has imagic flag set. Clear? yes Inode 26732610 is in use, but has dtime set. Fix? yes Inode 26732611 is in use, but has dtime set. Fix? yes Inode 26732611 has imagic flag set. Clear? yes Inode 26732612 is in use, but has dtime set. Fix? yes Inode 26732613 is in use, but has dtime set. Fix? yes Inode 26732613 has imagic flag set. Clear? /dev/sda1: e2fsck canceled. /dev/sda1: ***** FILE SYSTEM WAS MODIFIED ***** [root at myhost /]# fsck.ext3 -f /dev/sda1 e2fsck 1.35 (28-Feb-2004) Pass 1: Checking inodes, blocks, and sizes Inode 26732609 is in use, but has dtime set. Fix? -------------- next part -------------- An HTML attachment was scrubbed... URL: From markjballard at googlemail.com Fri Aug 30 09:41:30 2013 From: markjballard at googlemail.com (Mark Ballard) Date: Fri, 30 Aug 2013 10:41:30 +0100 Subject: ext3 / ext4 on USB flash drive? In-Reply-To: <20130829154657.GC30918@thunk.org> References: <20130829154657.GC30918@thunk.org> Message-ID: Agh. This is great to have some of this clarified, Ted. It does unfortunately reinorce my cynicism. But it also fills my heart. That is, though it might be bad news to hear that I have most likely bought a piece of crap, and that there's no way I can really tell what's inside the cover unless either I do some kind of low-level alchemy of a kind that mere users would normally be well advised to avoid, or the manufacturer fesses up to its scam by publishing the specs; it is nevertheless the case that this news has come from the filesystem maintainer himself, and my faith in humanity is thus restored by this simple proof of floss's egalitarian creed. Amen. That said, the user forums are full of lost souls, despairing for want of an answer to the simple question of whether they could well or should not format their USB in ext3/4. Are you implying that it cannot be known unless the manuf gives you the specs and if this is the case and you've bought a bargain bucket flash drive (and assuming that without a decent flash h/w controller it is true that ext3/4 will thrash your drive) then you should stick to ext2? What then are the features a USB flash spec you should look for as the minimum required before installing ext4? cheers mb. On 29 August 2013 16:46, Theodore Ts'o wrote: > On Thu, Aug 29, 2013 at 12:56:35PM +0100, Mark Ballard wrote: >> I think this is really an attempt at user feedback, rather than user >> discussion. But there's no such thing as a user-feedback mail list. >> >> Nevertheless, others may find this pertinent: why doesn't mke2fs >> handle USB's competently? And if it does, why doesn't it reassure me >> so? And how can I handle a linux-formatted USB flash drive in the >> absence of my system giving me any guidance? > > The problem is that it really depends on the quality of the flash > drive. Two main things drive this: > > 1) Whether the flash used in the drive is SLC. MLC, or TLC. This > will tell you how many write cycles the flash can support. This is > where you will see numbers such as 10,000 write cycles, all the way > down to 1,000 write depending on the quality of the flash used. > > 2) The quality of the Flash Translation Layer (FTL). The important > thing to remember here is that not all parts of the storage device > will get used to the same amount. Some files will rarely change once > they are installed on the device; others will change a huge amount. > So a good Flash Translation Layer will spread out the wear so that if > you have a 120GB device, with flash cells that support 10,000 write > cycles, that ideally you can support up to 10,000 times 120GB == 120 > TB worth of writes. Of course, that assumes 100% efficiency, which is > unrealistic. But some FTL's are incredibly bad, and in the worst > case, if they aren't spreading the wear around at all, if you write a > single block 1,000 times, and the flash only supports 1,000 write > cycles, then that block will go bad. > > Unfortunately, the cheapest USB flash devices that you find in the > bargain bin at the checkout counter of the Micro Center tend to use > the crappiest flash media possible, and the crappist FTL's. Some even > will only have 512MB of flash even though it is advertised as having > 8GB of flash, such that if you ever write more than 512MB worth of > files on the flash, you will start losing data. (Fortunately most > people don't actually use the full capacity of their USB sticks, which > is why the manufacturers can get away with this kind of fraud.) > > So this is fundamentally a problem with the quality of the hardware, > and that's not something the file system can really compensate for. > And there's no way to tell whether a particular USB device has has a > high quality flash device, or is a craptastic flash device. It's not > like we can query the device for "I ripped off the purchaser" bit. :-) > > Regards, > > - Ted From markjballard at googlemail.com Fri Aug 30 09:56:17 2013 From: markjballard at googlemail.com (Mark Ballard) Date: Fri, 30 Aug 2013 10:56:17 +0100 Subject: ext3 / ext4 on USB flash drive? In-Reply-To: <521F7747.2000003@redhat.com> References: <20130829154657.GC30918@thunk.org> <521F7747.2000003@redhat.com> Message-ID: > We'd like to help in software, but we can't; we have no reliable way > of knowing most of the necessary details of this class of hardware. > It's not exported in any way. > > So unfortunately we are as in the dark as you are in this case. This is incredible, Mr Sandeen. You mean USB flash manufacturers (what's their body - the USB Implementer's Forum?) have simply not provided a means for software to query the underlying hardware in a USB flash? Have software producers asked them for this? mb. From tytso at mit.edu Fri Aug 30 13:32:24 2013 From: tytso at mit.edu (Theodore Ts'o) Date: Fri, 30 Aug 2013 09:32:24 -0400 Subject: ext3 / ext4 on USB flash drive? In-Reply-To: References: <20130829154657.GC30918@thunk.org> <521F7747.2000003@redhat.com> Message-ID: <20130830133224.GB27699@thunk.org> On Fri, Aug 30, 2013 at 10:56:17AM +0100, Mark Ballard wrote: > > This is incredible, Mr Sandeen. You mean USB flash manufacturers > (what's their body - the USB Implementer's Forum?) have simply not > provided a means for software to query the underlying hardware in a > USB flash? Have software producers asked them for this? No, they haven't. And yes we have, since there are some things (such as the erase block size) which would be useful for tuning file system performance. And the technical people I've talked to at various Flash manufacturers all agree it's pointless to hide this information, but the product managers tend to be the roadblock. If you are buying several million eMMC flash devices for a mobile handset (i.e., for an Android device), you can find out this information, under NDA. But otherwise, the only way you can find it out is by carrying out timing attacks against the flash device. Even if they did provide this sort of information, there are lots of things which are not quantifiable. Cars don't have interface for telling you whether they have the crash protection of a Ford Pinto, or a Tesla S. Part of this is because product managers aren't going to want to advertise that they have only provided 512MB worth of flash on a device which is labelled as containing 8GB (and which will claim to contain 8GB even if you query it via software; it's just that the 512MB + 4k write will cause some random block to disappear). But the other part is that there isn't a good single metric for "crash safety" (crash safety against head-on collisions? crash safety versus side impact collisions? etc.) Similarly, even for a non-fradulent USB stick or SD card, there is no single way to measure "FTL quality". An FTL which is optimized for use in digital cameras (where writes are usually sequential, and for large jpg images) may be horrible for use a general purpose root file system. There are sites that will do technical analysis and ratings for various flash media (such as www.anandtech.com). For example a quick google search turns up: http://www.anandtech.com/show/4523/usb-30-flash-drive-roundup These are going to be the pricier devices, though. They aren't going to be the five dollar crap that you find at the checkout counter of a computer store. But then again, you get what you pay for.... - Ted From tytso at mit.edu Fri Aug 30 13:35:34 2013 From: tytso at mit.edu (Theodore Ts'o) Date: Fri, 30 Aug 2013 09:35:34 -0400 Subject: ext3 / ext4 on USB flash drive? In-Reply-To: References: <20130829154657.GC30918@thunk.org> Message-ID: <20130830133534.GC27699@thunk.org> On Fri, Aug 30, 2013 at 10:41:30AM +0100, Mark Ballard wrote: > That said, the user forums are full of lost souls, despairing for want > of an answer to the simple question of whether they could well or > should not format their USB in ext3/4. Are you implying that it cannot > be known unless the manuf gives you the specs and if this is the case > and you've bought a bargain bucket flash drive (and assuming that > without a decent flash h/w controller it is true that ext3/4 will > thrash your drive) then you should stick to ext2? What then are the > features a USB flash spec you should look for as the minimum required > before installing ext4? The cheapest crap is going to be "unsafe at any speed". It might be OK if you are using it for the occasional transfer of files from one computer to another, but if you are going to be doing lots of updates, your data is going to be at risk no matter what file system you use. Some filesystems are more performant than others, so keep in mind that a simple way to double the lifetime of some cheap crap is to slow down the write speed by a factor of two. :-) So I wouldn't so much focus on whether some storage device is "good enough for ext3/4", but rather, "is it good enough given the value of the data of the data I'm storing on it, and the read/write patterns I'm going to be subjecting the drive to"? Cheers, - Ted From adilger at dilger.ca Fri Aug 30 18:07:22 2013 From: adilger at dilger.ca (Andreas Dilger) Date: Fri, 30 Aug 2013 12:07:22 -0600 Subject: Strange fsck.ext3 behavior - infinite loop In-Reply-To: <90D96432E685D84E9A62AF83962A689737E7AA68@MED-CORE07A.med.wayne.edu> References: <90D96432E685D84E9A62AF83962A689737E7AA68@MED-CORE07A.med.wayne.edu> Message-ID: <4764C2B2-63C9-4FC5-A99B-3D8BEB004995@dilger.ca> On 2013-08-29, at 7:48 PM, Richards, Paul Franklin wrote: > Strange behavior with fsck.ext3: how to remove a long orphaned inode list? > > After copying data over from one old RAID to another new RAID with rsync, the dump command would not complete because of filesystem errors on the new RAID. So I ran fsck.ext3 with the -y option and it would just run in an infinite loop restarting itself and then trying to correct the same inodes over and over again. Some of the errors were lots of orphaned inodes. So I ran a tar tape backup and reformatted it with mkfs.ext3. After restoring the tar backup, I got the same errors when I ran fsck.ext3 -f. Again fsck.ext3 -y would run in an infinite loop trying to correct the problem. So I formatted it again and ran fsck immediately afterwards and it's still detecting corrupted orphaned inode lists. This is on what should be a pristine filesystem with basically no files. Is this a hardware problem with the RAID or a bug somewhere or just normal behavior? Definitely a bug somewhere. > [root at myhost /]# mkfs.ext3 /dev/sda1 > mke2fs 1.35 (28-Feb-2004) First thing I would suggest is to update to a newer version of e2fsprogs, since this one is 9+ years old and that is a lot of water under the bridge. Cheers, Andreas > Filesystem label= > OS type: Linux > Block size=4096 (log=2) > Fragment size=4096 (log=2) > 268435456 inodes, 536868352 blocks > 26843417 blocks (5.00%) reserved for the super user > First data block=0 > Maximum filesystem blocks=4294967296 > 16384 block groups > 32768 blocks per group, 32768 fragments per group > 16384 inodes per group > Superblock backups stored on blocks: > 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, > 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, > 102400000, 214990848, 512000000 > > Writing inode tables: done > Creating journal (8192 blocks): done > Writing superblocks and filesystem accounting information: done > > This filesystem will be automatically checked every 24 mounts or > 180 days, whichever comes first. Use tune2fs -c or -i to override. > [root at myhost /]# fsck > fsck fsck.cramfs fsck.ext2 fsck.ext3 fsck.msdos fsck.vfat > [root at myhost /]# man fsck.ext3 > [root at myhost /]# fsck.ext3 -C /root/completion /dev/sda1 > e2fsck 1.35 (28-Feb-2004) > /dev/sda1: clean, 11/268435456 files, 8450084/536868352 blocks > [root at myhost /]# fsck.ext3 -f -C /root/completion /dev/sda1 > e2fsck 1.35 (28-Feb-2004) > Pass 1: Checking inodes, blocks, and sizes > Inodes that were part of a corrupted orphan linked list found. Fix? yes > > Inode 26732609 was part of the orphaned inode list. FIXED. > Inode 26732609 has imagic flag set. Clear? yes > > Inode 26732610 is in use, but has dtime set. Fix? yes > > Inode 26732611 is in use, but has dtime set. Fix? yes > > Inode 26732611 has imagic flag set. Clear? yes > > Inode 26732612 is in use, but has dtime set. Fix? yes > > Inode 26732613 is in use, but has dtime set. Fix? yes > > Inode 26732613 has imagic flag set. Clear? > > /dev/sda1: e2fsck canceled. > > /dev/sda1: ***** FILE SYSTEM WAS MODIFIED ***** > [root at myhost /]# fsck.ext3 -f /dev/sda1 > e2fsck 1.35 (28-Feb-2004) > Pass 1: Checking inodes, blocks, and sizes > Inode 26732609 is in use, but has dtime set. Fix? > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users Cheers, Andreas From tytso at mit.edu Fri Aug 30 18:23:02 2013 From: tytso at mit.edu (Theodore Ts'o) Date: Fri, 30 Aug 2013 14:23:02 -0400 Subject: Strange fsck.ext3 behavior - infinite loop In-Reply-To: <4764C2B2-63C9-4FC5-A99B-3D8BEB004995@dilger.ca> References: <90D96432E685D84E9A62AF83962A689737E7AA68@MED-CORE07A.med.wayne.edu> <4764C2B2-63C9-4FC5-A99B-3D8BEB004995@dilger.ca> Message-ID: <20130830182302.GE30385@thunk.org> On Fri, Aug 30, 2013 at 12:07:22PM -0600, Andreas Dilger wrote: > > > [root at myhost /]# mkfs.ext3 /dev/sda1 > > mke2fs 1.35 (28-Feb-2004) > > First thing I would suggest is to update to a newer version of e2fsprogs, since this one is 9+ years old and that is a lot of > water under the bridge. That's definitely good advice, but even with e2fsprogs 1.35, if e2fsck -f is finding errors when run immediately after running mke2fs, it would make me suspect the storage device. Are you sure the RAID controller (is this a hw raid, or software raid) is working correctly? - Ted