From kristen.eisenberg at yahoo.com Sat Nov 5 13:06:57 2011 From: kristen.eisenberg at yahoo.com (Kristen Eisenberg) Date: Sat, 5 Nov 2011 06:06:57 -0700 (PDT) Subject: High CPU Utilization When Copying to Ext4 Message-ID: <1320498417.54327.YahooMailNeo@web122301.mail.ne1.yahoo.com> Sorry if this is not the correct mailing list for ext4 questions. Kristen Eisenberg Billige Fl?ge Marketing GmbH Emanuelstr. 3, 10317 Berlin Deutschland Telefon: +49 (33) 5310967 Email: utebachmeier at gmail.com Site: http://flug.airego.de - Billige Fl?ge vergleichen -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Sat Nov 5 17:21:07 2011 From: tytso at mit.edu (Ted Ts'o) Date: Sat, 5 Nov 2011 13:21:07 -0400 Subject: High CPU Utilization When Copying to Ext4 In-Reply-To: <1320498417.54327.YahooMailNeo@web122301.mail.ne1.yahoo.com> References: <1320498417.54327.YahooMailNeo@web122301.mail.ne1.yahoo.com> Message-ID: <20111105172106.GA4801@thunk.org> On Sat, Nov 05, 2011 at 06:06:57AM -0700, Kristen Eisenberg wrote: > Sorry if this is not the correct mailing list for ext4 questions. You might get a better discussion going on the linux-ext4 at vger.kernel.org developer's list, but I do listen in on ext3-users. What do you consider high CPU utilization, and how are you writing to ext4? And what kernel version are you using? Regards, - Ted From feldmann_markus at gmx.de Sat Nov 26 14:15:42 2011 From: feldmann_markus at gmx.de (Markus Feldmann) Date: Sat, 26 Nov 2011 15:15:42 +0100 Subject: damaged encrypted LUKS device Message-ID: Hi people, i had created encrypted device with cryptsetup/LUKS which i setup with an ext4 filesystem. This device is an external USB harddisk. When i plugin this device it will be automatically mounted by my gnome3 system (Debian Wheezey/Testing), but this week i got an error. I did ask the ask the LUKS developers what is the problem and they told me that this is an ext4 problem. The error message from gnome is: > Einh?ngen von AIRY_1TB nicht m?glich > > Error mounting: mount: wrong fs type, bad option, bad superblock on /dev/mapper/udisks-luks-uuid-70a2aedf-ce7e-4e12-8f0b-ec7974ebdbd4-uid1000, > missing codepage or helper program, or other error > In some cases useful info is found in syslog - try > dmesg | tail or so And from i got the message: > [ 3094.394043] EXT4-fs (dm-0): ext4_check_descriptors: Inode bitmap for group 3328 not in group (block 1235654634)! > [ 3094.394056] EXT4-fs (dm-0): group descriptors corrupted! Further on, when i try to i get: > fsck from util-linux 2.19.1 > fsck: fsck.crypto_LUKS: not found > fsck: Error 2 while executing fsck.crypto_LUKS for /dev/sdb But cryptsetup is installed, only lvm2 is missing, but i dont have a LVM device and as i researched at "debian.org" there is no package which provides the command . Any hints/instructions for me? regards Markus From mnalis-ml at voyager.hr Sat Nov 26 16:40:04 2011 From: mnalis-ml at voyager.hr (Matija Nalis) Date: Sat, 26 Nov 2011 17:40:04 +0100 Subject: damaged encrypted LUKS device In-Reply-To: References: Message-ID: <20111126164004.GA13390@eagle102.home.lan> On Sat, Nov 26, 2011 at 03:15:42PM +0100, Markus Feldmann wrote: > And from i got the message: > >[ 3094.394043] EXT4-fs (dm-0): ext4_check_descriptors: Inode bitmap for group 3328 not in group (block 1235654634)! > >[ 3094.394056] EXT4-fs (dm-0): group descriptors corrupted! > > Further on, when i try to i get: > >fsck from util-linux 2.19.1 > >fsck: fsck.crypto_LUKS: not found > >fsck: Error 2 while executing fsck.crypto_LUKS for /dev/sdb > > But cryptsetup is installed, only lvm2 is missing, but i dont have a > LVM device and as i researched at "debian.org" there is no package > which provides the command . > > Any hints/instructions for me? try "e2fsck -C0 /dev/dm-0" (assuming /dev/dm-0 is your block device after decryption). And do full block-by-block backup of your external USB disk first! (using dd(1)) -- Opinions above are GNU-copylefted. From puhuri at iki.fi Mon Nov 28 10:17:25 2011 From: puhuri at iki.fi (Markus Peuhkuri) Date: Mon, 28 Nov 2011 12:17:25 +0200 Subject: damaged encrypted LUKS device In-Reply-To: <20111126164004.GA13390@eagle102.home.lan> References: <20111126164004.GA13390@eagle102.home.lan> Message-ID: <4ED35FB5.1080506@iki.fi> On 26.11.2011 18:40, Matija Nalis wrote: > And do full block-by-block backup of your external USB disk first! > (using dd(1)) And if it happens to abort on error, try http://safecopy.sourceforge.net (available on package in recent distros) that makes quite good work on reading disks with read errors. Worked better than any dd{_recover,rescue}. After doing the 3-stage copy, I was able copy files from ext4 partition inside LVM that was in qcow2 virtual machine image that was on ext3 partition on LVM on the failed disk. t. Markus From feldmann_markus at gmx.de Mon Nov 28 14:42:56 2011 From: feldmann_markus at gmx.de (Markus Feldmann) Date: Mon, 28 Nov 2011 15:42:56 +0100 Subject: damaged encrypted LUKS device In-Reply-To: <20111126164004.GA13390@eagle102.home.lan> References: <20111126164004.GA13390@eagle102.home.lan> Message-ID: Am 26.11.2011 17:40, schrieb Matija Nalis: > > try "e2fsck -C0 /dev/dm-0" (assuming /dev/dm-0 is your block device > after decryption). How can i be sure, that my cryptsetup/LUKS harddisk is still decrypted? gives me: > root at feld-bert:/home/markus# cryptsetup status /dev/dm-1 > /dev/mapper//dev/dm-1 is active. > type: LUKS1 > cipher: aes-xts-plain > keysize: 256 bits > device: /dev/sdb > offset: 4096 sectors > size: 1953521072 sectors > mode: read/write From bothie at gmx.de Mon Nov 28 15:58:06 2011 From: bothie at gmx.de (Bodo Thiesen) Date: Mon, 28 Nov 2011 16:58:06 +0100 Subject: damaged encrypted LUKS device In-Reply-To: References: <20111126164004.GA13390@eagle102.home.lan> Message-ID: <20111128165806.7a2d0227@gmx.de> Hello Markus. * Markus Feldmann hat geschrieben: >* Am 26.11.2011 17:40, schrieb Matija Nalis: >> try "e2fsck -C0 /dev/dm-0" (assuming /dev/dm-0 is your block device >> after decryption). > How can i be sure, that my cryptsetup/LUKS harddisk is still decrypted? > gives me: >> root at feld-bert:/home/markus# cryptsetup status /dev/dm-1 >> /dev/mapper//dev/dm-1 is active. That line tells you ;) So, try a e2fsck -n -C0 /dev/dm-1 (the -n causes no changes to be made to the file system - errors will be reported only). If it's not too much, but you're unsure what to do, maybe you want to post them here (use command LANG=C e2fsck -n /dev/dm-1 > some-file.txt for that and post the contents of some-file.txt). If it's too much maybe you want to pastebin them and post the link here. In both cases, remember to strip path names and similar data unless you don't care to publish them. (However - why are you encrypting them then in the first place?) Additionally, I suggest following Markus Peuhkuri's tip to first create a block by block copy before doing anything else. I suggest copying the encrypted version (i.e. /dev/sdb) not the decrypted one (i.e. /dev/dm-1). Regards, Bodo From feldmann_markus at gmx.de Mon Nov 28 17:09:34 2011 From: feldmann_markus at gmx.de (Markus Feldmann) Date: Mon, 28 Nov 2011 18:09:34 +0100 Subject: damaged encrypted LUKS device In-Reply-To: <20111128165806.7a2d0227@gmx.de> References: <20111126164004.GA13390@eagle102.home.lan> <20111128165806.7a2d0227@gmx.de> Message-ID: Here comes my new output from http://pastebin.com/raw.php?i=x0siBmrj I will buy another 1 TByte harddisk these days and then make a byte-to-byte backup. regards Markus From bothie at gmx.de Mon Nov 28 18:26:52 2011 From: bothie at gmx.de (Bodo Thiesen) Date: Mon, 28 Nov 2011 19:26:52 +0100 Subject: damaged encrypted LUKS device In-Reply-To: References: <20111126164004.GA13390@eagle102.home.lan> <20111128165806.7a2d0227@gmx.de> Message-ID: <20111128192652.29638259@gmx.de> * Markus Feldmann hat geschrieben: > Here comes my new output from > > http://pastebin.com/raw.php?i=x0siBmrj Ok, the summary of that paste is: | root at feld-bert:/home/markus# LANG=C e2fsck -n -C0 /dev/dm-1 | e2fsck 1.42-WIP (16-Oct-2011) | e2fsck: Group descriptors look bad... trying backup blocks... | Error writing block 1 (Attempt to write block to filesystem | resulted in short write). Ignore error? no (more lines of this kind) | e2fsck: Device or resource busy while trying to open /dev/dm-1 | Filesystem mounted or opened exclusively by another program? And the bug report is here: https://sourceforge.net/tracker/?func=detail&aid=3444351&group_id=2406&atid=102406 | 1. e2fsck should not try to write when using -n. | 2. ignore error is not a question to be answered with "no" when | using -n but with "yes" instead. | 3. I do not know what that "Device or resource busy" error is about | (someone else reported that problem on ext3-users at redhat.com, I'm | merely relaying that bug report), but it doesn't matter either: -n | means don't write and reading worked obviously. > I will buy another 1 TByte harddisk these days and then make a > byte-to-byte backup. Do that. But before you start the copy, reboot your system, maybe there is some processes waiting in disk-io on the problem disk (task state D in process lists). At least, that's what | e2fsck: Device or resource busy while trying to open /dev/dm-1 | Filesystem mounted or opened exclusively by another program? suggest. However, that message may as well be caused by the other bug in e2fsck. Maybe it could be useful to init 1, wait for the X server and other stuff to stop and then log in on the text console and do the copy there, so that all that normally fancy but for this problem nasty auto mounting stuff is not active when you connect the disk(s) to your system. Regards, Bodo From tytso at mit.edu Tue Nov 29 23:54:23 2011 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 29 Nov 2011 18:54:23 -0500 Subject: E2fsprogs 1.42 is released! Message-ID: I'm happy to announce the release of e2fsprogs 1.42. You can download the tarball from here: http://prdownloads.sourceforge.net/e2fsprogs/e2fsprogs-1.42.tar.gz Or you can download it from the git repository on git.kernel.org. git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git. Or, you can wait for your favorite distribution to package it. :-) The release notes are attached below. Many thanks to all of the contributors to e2fsprogs, whose employers include Canonical, Ctera Networks, Fujitsu, Google, IBM, Red Hat, SuSE, Tao Bao, Whamcloud, and of course to all of the users of e2fsprogs. Many thanks for your support, bug reports, code contributions, and translations over the years. Regards, - Ted E2fsprogs 1.42 (November 29, 2011) ================================== This release of e2fsprogs has support for file systems > 16TB. Online resize requires kernel support which will hopefully be in Linux version 3.2. Offline support is not yet available for > 16TB file systems, but will be coming. This release of e2fsprogs has support for clustered allocation. This reduces the number of block (now cluster) bitmaps by allocating and deallocating space in contiguous power-of-2 collections of blocks, which are called clustered. This is a file system level feature, called 'bigalloc', which must be enabled when the file system is initially formatted. It is not backwards compatible with older kernels. Added support for the Multi-Mount Protection (MMP) feature. E2fsck more efficiently uses scratch files for really big file systems. (This is a feature that has to be turned on explicitly; see [scratch_files] in the e2fsck.conf man page.) Fix a bug in e2fsck where if the free blocks and inodes counts are incorrect, e2fsck would fix them without printing an error message. This would cause a "*** FILE SYSTEM WAS MODIFIED ***" message without any explanation of what was fixed. E2fsck will no longer attempt to clone an extended attribute block in pass1b handling if the file system does not support extended attributes. E2fsck will be more careful accidentally asking the user to continue if the file system is mounted, so that an escape sequence won't cause a false positive. (Addresses Debian Bug: #619859) E2fsck now uses less cpu time in pass 5 when large portions of the bitmaps are uninitialized. E2fsck will no longer segault when a corrupted file system has a bad extent, and removing it leads to a block needing to be deallocated. (Addresses SourceForge Bug: #2971800) E2fsck will catch termination signals (segfaults, bus errors, sigfpe) and print debugging information to make it easier to find potential problems. E2fsck will check to see if the bad block inode looks insane, and will skip trying to use if it certain fields which should be zero look non-zero. This is to avoid a corrupted bad block inode causing e2fsck to get confused and do more harm than good. If e2fsck modifies certain superblock fields which the kernel doesn't look at, it will now mark the superblock as dirty without marking the file system as changed. This avoids signalling the init scripts that a reboot is necessary, since the kernel isn't going to look at those fields, so it won't care if they have been changed. Fixed a bug in the libext2fs library (in the binary search routine of the icount abstraction) that could (very, very rarely) cause e2fsck to die in the middle of pass 1 or pass 2 processing. E2fsck will not try to do a discard operation if the -n option was specified on the command line. E2fsck now supports an extended "discard" option which will cause e2fsck to attempt discard all unused blocks after a full, successful file system check. E2fsck will check for the bad block inode to make sure it looks sane before trusting it, to avoid causing more harm than good to the file system. E2fsck now returns additional status bits in its exit code if it aborts early in the e2fsck run. E2fsck now correctly calculates the maximum file size in the case of the huge_file file system feature enabled without extents. The mke2fs and e2fsck programs now tries to use the punch hole command as a "discard" when operating on normal files. The e2image program now supports the qcow2 format, which is a more efficient way of capturing file system dumps. Mke2fs now supports the [devices] stanza in mke2fs.conf which allows per-device defaults to be specified in the configuration file. Mke2fs now supports the reserved_ratio relation in the [defaults] and [fs_types] section in mke2fs.conf. Mke2fs now creates extent-mapped directories for the root and lost+found directories. Mke2fs will skip zero'ing the journal if the extended option "lazy_journal_init" is specified. This can save a lot of time, but it does add a small amount of risk if the system crashes before the journal is overwritten entirely once. It is epsecially useful for testing. Mke2fs will now create file systems that enable user namespace extended attributes and with time- and mount count-based file system checks disabled. Mke2fs will not set a stride or strip size of one block based on block device attributes obtained from sysfs. Mke2fs now displays a progress report during the discard process. Mke2fs now handles extreme file system parameters correctly which previously caused the inodes per group to drop below 8, leading to a segfault. (The inodes per group must be a multiple of 8, but the code didn't correctly deal with an inodes per group count less than 8.) Mke2fs and tune2fs previously would give an error if the user tried setting the stride and stripe-width parameters to zero; but this is necessary to disable the stride and stripe-width settings. So allow setting these superblock fields to zero. (Addresses Google Bug: #4988557) Mke2fs now gives a warning if the auto-detected block size exceeds the system's page size. If the enable_periodic_fsck option is false in /etc/mke2fs.conf (which is the default), mke2fs will now set the s_max_mnt_count superblock field to -1, instead of 0. Kernels older then 3.0 will print a spurious message on each mount then they see a s_max_mnt_count set to 0, which will annoy users. (Addresses Debian Bug: #632637) The default mke2fs.conf now has entries for "big" and "huge", which are needed for very big file systems. The resize2fs program now has support for a new online resize ioctl that can support file systems > 16TB, once it arrives in v3.x kernels. Fixed bug which caused resize2fs to fail when shrinking an empty file system down to its minimal size. (Addresses Sourceforge Bug #3404051) Fixed tune2fs's mount options parsing. (Addresses Debian Bug: #641667) Allow tune2fs to remove external journals if the device can not be found. Debugfs's icheck will now correctly find inodes which use the searched-for block as an extended attribute block. Debugfs now has a new "punch" command which remove blocks from the middle of an inode. Debugfs now has a new "e2freefrag" command which analyzes the free space fragmentation of the file system, using the same code as the e2freefrag program. Debugfs now has a "filefrag" command which displays information about a file's fragmentation. Add support to build a metadata-only, read-only, stripped-down version of debugfs called rdebugfs. Fixed a potential stack overrun bug in debugfs. The badblocks program now correctly recovers from I/O errors when direct I/O is being used. The badblocks command now also supports a -B option which forces the use of buffered I/O, and the -v option will provide a more detailed breakdown of read, write, and failed comparison errors. Added e4defrag tool which uses the EXT4_IOC_MOVE_EXT ioctl. Added support for journals larger than 2GB. Support using both hard links and symlinks when installing e2fsprogs. Add overflow checking to tune2fs -i's fsck interval, which must fit in a 32-bit field. The debugfs command now has a new 'blocks' command which prints out data blocks of a particular inode in a format which is useful for scripting. Filefrag will report 0 extents correctly in verbose mode. (Addresses RedHat Bugzilla: #653234) Filefrag has been fixed so its -v report prints the correct expected block number (previously there had been an off-by-one error). In addition, it will now display the number of contiguous extents when -v is not specified. This makes it consistent with the number of extents printed when the -v option was specified. In addition, the number of contiguous extents is far more interesting/useful than the number of physical extents for very large files. (Addresses Debian Bug: #631498, #644792) Logsave's usage message has been fixed. (Addresses Debian Bug: #619788) Avoid an infinite loop in ext2fs_find_block_device() if there are symlink loops in /dev caused by a buggy udev. Added a useful "fallocate" program to the contrib directory. Fixed an ABI compatibility problem in libext2fs which broke the dump program. Also added back some macros which dump needed so it could compile against the latest version of ext2_fs.h (Addresses Debian Bug: #636418) Fixed parsing of MNTOPT_ options for tune2fs and debugfs (Addresses Debian Bug: #641667) Added internationalization support for libcom_err error table strings. Fixed various spelling mistakes found in various output strings found by I18N translators. Update translations: French, Chinese, Germany, Indonesian, Swedish, Vietnamese, Polish, Dutch, Czech. (Addresses Debian Bugs: #520985, #620659) Fixed various Debian Packaging issues. (Addresses Debian Bugs: #614662, #632169, #641838, #627535, #629355) Updated/clarified man pages. (Addresses Debian Bugs: #639411, #642193, #634883) Programming notes ----------------- Initial support for quota as an integrated feature, where the quota files are hidden system files that are automatically maintained by e2fsck is present, although disabled by default. It must be enabled by using the configure option --enable-quota. Reserved the on-disk fields for the metadata checksum and snapshot features, which are currently in development. The ext2fs library now has the new functions ext2fs_punch(), ext2fs_get_memzero() and ext2fs_file_get_inode(). The ext2fs library now has support for calculating the crc32c checksum (via the new functions ext2fs_crc32c_be and ext2fs_crc32c_le). The I/O manager now supports the discard operation. Reserved file system code points for new 1st class quota feature. Shortened the compilation lines by moving the autoconf defines to a config.h header file. Fixed a potential free of an unitialized pointer in ext2fs_update_bb_inode(). Fixed miscellaneous gcc -Wall and coverity warnings. Fixed portability issues for Mac OS X, Hurd, and FreeBSD. Fixed a build failure when OMIT_COM_ERR is defined. Improved error checking and fixed memory leaks caused by error return paths. Add ext2fs_flush2() and ext2fs_close2() which takes a flag to allow the fsync() to be skipped. Added a test for extent-mapped journals by mke2fs and tune2fs. Added a test for creating a large (over 4GB) journal using mke2fs. Added a test to make sure the inode size remains constant. The regression test script now prints the list of which tests failed. The regression test system now uses /tmp for its scratch files, which is often a tmpfs mounted file system and hence much faster. The i_e2image test was fixed so it works with valgrind. Tests that rely on debugfs are now skipped if debugfs wasn't built. Fixed the dependencies for "make check" so all required dependencies are built before running the gression tests. A link to com_err.h is installed in $(includedir) during a "make install". The po/*.gmo files are automatically rebuilt if they are missing or out of date. This helps out Debian packaging. Allow ext2fs_get_memalign() to compile on systems that don't have posix_memalign(). Fixed a namespace leak in libext2fs (tdb_null). E2fsprogs 1.41.14 (December 22, 2010) ===================================== Fix spurious complaint in mke2fs where it would complain if the file system type "default" is not defined in mke2fs.conf. The resize2fs program will no longer clear the resize_inode feature when the number reserved group descriptor blocks reaches zero. This allows for subsequent shrinks of the file system to work cleanly for flex_bg file systems. The resize2fs program now handles devices which are exactly 16T; previously it would give an error saying that the file system was too big. E2fsck (and the libext2fs library) will not use the extended rec_len encoding for file systems whose block size is less than 64k, for consistency with the kernel. Programming notes ----------------- E2fsprogs 1.41.13 would not compile on big-endian systems. This has been fixed. (Addresses Sourceforge Bug: #3138115) The ext2fs_block_iterator2() function passed an incorrect ref_offset to its callback function in the case of sparse files. (Addresses Sourceforge Bug: #3081087) Fix some type-punning warnings generated by newer versions of gcc. From sandeen at redhat.com Wed Nov 30 01:23:16 2011 From: sandeen at redhat.com (Eric Sandeen) Date: Tue, 29 Nov 2011 19:23:16 -0600 Subject: E2fsprogs 1.42 is released! In-Reply-To: References: Message-ID: <4ED58584.1050209@redhat.com> On 11/29/11 5:54 PM, Theodore Ts'o wrote: > > I'm happy to announce the release of e2fsprogs 1.42. > > You can download the tarball from here: > > http://prdownloads.sourceforge.net/e2fsprogs/e2fsprogs-1.42.tar.gz > > Or you can download it from the git repository on git.kernel.org. > > git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git. > > Or, you can wait for your favorite distribution to package it. :-) It's now built for Fedora Rawhide, for anyone feeling excited to try it. :) -Eric