From listudo at bestsolution.at Fri Aug 6 09:21:47 2010 From: listudo at bestsolution.at (Udo Rader) Date: Fri, 06 Aug 2010 11:21:47 +0200 Subject: complete fs corruption after fsck Message-ID: <4C5BD42B.4010801@bestsolution.at> Hi, we are currently facing the worst of all possible scenarios on one of our virtualized database servers (kvm, with a qcow2 image). After trying to repair a broken database, the kernel reported an ext3 error, put the filesystem in r/o mode and suggested an fsck. The story ends with a completely destroyed filesystem, the only folder existing is lost+found, containing huge amounts of #12345 style files and folders. Now my problem is that unfortunately most of the stuff that went into lost+found is just unuseable garbage, there is hardly any "regular" file in lost+found and its subdirectories. What I desperatly need is at least one of the many +400M backup files located on the server. They happen to be so called postsgresql "custom" dump files, identifyable by their "PGDMP" file header. In order to find out wether such files or fragments exist in lost+found I fgrep'ed the entire folder, w/o success. On the other hand, if I fgrep the qcow2 image file for the string, I get results soon. So, are there any options left to get more data back? I've played with lde [1], but I must admit that the magic required to understand lde is beyond my level of magic ... If there is a chance, we would also be willing to pay for the efforts required. Thanks in advance. Udo Rader [1] http://lde.sourceforge.net/ -- Udo Rader, CTO http://www.bestsolution.at http://riaschissl.blogspot.com From mnalis-ml at voyager.hr Sat Aug 7 21:02:32 2010 From: mnalis-ml at voyager.hr (Matija Nalis) Date: Sat, 7 Aug 2010 23:02:32 +0200 Subject: complete fs corruption after fsck In-Reply-To: <4C5BD42B.4010801@bestsolution.at> References: <4C5BD42B.4010801@bestsolution.at> Message-ID: <20100807210232.GA6429@eagle102.home.lan> On Fri, Aug 06, 2010 at 11:21:47AM +0200, Udo Rader wrote: > What I desperatly need is at least one of the many +400M backup files > located on the server. They happen to be so called postsgresql "custom" > dump files, identifyable by their "PGDMP" file header. They are probably too big to be in one piece on the disk, and since the pgdump custom format does not sound like something easily detectable as correct (it looks like it's usually gzip compressed, which means it would look like random data), they probably won't recover correctly. > On the other hand, if I fgrep the qcow2 image file for the string, I get > results soon. > > So, are there any options left to get more data back? I've played with > lde [1], but I must admit that the magic required to understand lde is > beyond my level of magic ... You can try one of the following: http://www.cgsecurity.org/wiki/TestDisk http://www.itu.dk/people/jobr/magicrescue/ http://www.sleuthkit.org/ http://www.keldix.com/keld/readme-salvage.html or some of the tools from: https://ext4.wiki.kernel.org/index.php/Undeletion you would want to make a copy of qcow2 image first, as some of the tools will damage it even more than it was in the beginning, so you want to start with fresh copy on each try. Good luck... -- Opinions above are GNU-copylefted. From listudo at bestsolution.at Mon Aug 9 09:27:57 2010 From: listudo at bestsolution.at (Udo Rader) Date: Mon, 09 Aug 2010 11:27:57 +0200 Subject: complete fs corruption after fsck In-Reply-To: <20100807210232.GA6429@eagle102.home.lan> References: <4C5BD42B.4010801@bestsolution.at> <20100807210232.GA6429@eagle102.home.lan> Message-ID: <4C5FCA1D.8000101@bestsolution.at> On 08/07/2010 11:02 PM, Matija Nalis wrote: > On Fri, Aug 06, 2010 at 11:21:47AM +0200, Udo Rader wrote: >> What I desperatly need is at least one of the many +400M backup files >> located on the server. They happen to be so called postsgresql "custom" >> dump files, identifyable by their "PGDMP" file header. > > They are probably too big to be in one piece on the disk, and since the > pgdump custom format does not sound like something easily detectable as > correct (it looks like it's usually gzip compressed, which means it would > look like random data), they probably won't recover correctly. > >> On the other hand, if I fgrep the qcow2 image file for the string, I get >> results soon. >> >> So, are there any options left to get more data back? I've played with >> lde [1], but I must admit that the magic required to understand lde is >> beyond my level of magic ... > > You can try one of the following: > > http://www.cgsecurity.org/wiki/TestDisk > http://www.itu.dk/people/jobr/magicrescue/ > http://www.sleuthkit.org/ > http://www.keldix.com/keld/readme-salvage.html > > or some of the tools from: > https://ext4.wiki.kernel.org/index.php/Undeletion > > you would want to make a copy of qcow2 image first, as some of the tools > will damage it even more than it was in the beginning, so you want to start > with fresh copy on each try. Thanks a lot for those links - as it seems some kind of real survivial kits for situations like mine. This is at least something more to try, although, as you write, my hope is limited due to the internal PGDMP format (that indeed gzip's data inside). Thanks again! Udo Rader -- Udo Rader, CTO http://www.bestsolution.at http://riaschissl.blogspot.com From Ralf.Hildebrandt at charite.de Fri Aug 13 10:00:55 2010 From: Ralf.Hildebrandt at charite.de (Ralf Hildebrandt) Date: Fri, 13 Aug 2010 12:00:55 +0200 Subject: Squid and first-level subdirectories & second-level subdirectories on ext3/4 Message-ID: <20100813100055.GT16572@charite.de> Squid, a proxy, is by its nature, storing large amounts of relatively small files in it's cache. As config optins it's offering: # 'Level-1' is the number of first-level subdirectories which # will be created under the 'Directory'. The default is 16. # # 'Level-2' is the number of second-level subdirectories which # will be created under each first-level directory. The default # is 256. Meaning one has /squid-cache/(16 dirs)/(256 dirs)/(the small files) so the total number of small files in the cache is (hopefully) evenly distributed to 16*256 directories. But is that optimal for an ext3/4 filesystem? What is the point of using 16 for the first level and 256 for the second? Wouldn't 64*64 (which equals 16*256) be better when it comes to finding the files on disk? -- Ralf Hildebrandt Gesch?ftsbereich IT | Abteilung Netzwerk Charit? - Universit?tsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt at charite.de | http://www.charite.de From adilger at dilger.ca Sun Aug 15 23:05:00 2010 From: adilger at dilger.ca (Andreas Dilger) Date: Sun, 15 Aug 2010 17:05:00 -0600 Subject: Squid and first-level subdirectories & second-level subdirectories on ext3/4 In-Reply-To: <20100813100055.GT16572@charite.de> References: <20100813100055.GT16572@charite.de> Message-ID: <6982E360-6DE7-4DEF-8F4D-E5031EB7151B@dilger.ca> On 2010-08-13, at 04:00, Ralf Hildebrandt wrote: > Squid, a proxy, is by its nature, storing large amounts of relatively > small files in it's cache. > > As config optins it's offering: > > # 'Level-1' is the number of first-level subdirectories which > # will be created under the 'Directory'. The default is 16. > # > # 'Level-2' is the number of second-level subdirectories which > # will be created under each first-level directory. The default > # is 256. > > Meaning one has > > /squid-cache/(16 dirs)/(256 dirs)/(the small files) > > so the total number of small files in the cache is (hopefully) evenly > distributed to 16*256 directories. > > But is that optimal for an ext3/4 filesystem? What is the point of > using 16 for the first level and 256 for the second? In ext3/4 the top-level inodes are spread around the filesystem, on the assumtion that something like /home or / is allocating trees of unrelated subdirectories at the top level, but that files within those subdirectories ARE related and should be allocated together. Depending on how many files are in your cache, the 256 * {small files} is likely too big to fit into a single block group (32k inodes, 32k blocks) so you may want to consider marking the first level of subdirectories with the "TOPDIR" flag, that indicates the second-level (256) subdirs should also be spread around the disk. > Wouldn't 64*64 (which equals 16*256) be better when it comes to > finding the files on disk? Benchmarking tells all... Cheers, Andreas From samuel at bcgreen.com Tue Aug 24 20:32:45 2010 From: samuel at bcgreen.com (Stephen Samuel) Date: Tue, 24 Aug 2010 13:32:45 -0700 Subject: problem with custom fs that utilizes ext3 In-Reply-To: References: Message-ID: As far as I'm concerned, I consider the fact that you can extend the size of the inode and get back old data to be the bug. (it's a data security problem). Also: Storing data in the slack space of files is dangerous because, if someone else extends the same file, you're going to lose your data anyways. Depending on an undocumented mis-feature of a filesystem is a rather unsafe way to make a product. If this is intended for production use, I'd say that you're better off to just bite the bullet and create a *real* file for persistent strage. On Mon, Jul 19, 2010 at 12:39 PM, Ryan O'Neill wrote: > I have developed a file system, it is a virtual memory file system > with persistence -- the persistence essentially works by storing the > fs data (from memory) into the slack space > of ext3 files (We are working on CentOS 5.3 -- old I know). The > following details should be sufficient. > > I keep the inode size the same so that utilities don't see the hidden > data -- it appears do_sync_read which ext3 uses and the function that > it uses (such as generic_file_read) do not read past the size of the > inode. > ..... -- Stephen Samuel http://www.bcgreen.com Software, like love, 778-861-7641 grows when you give it away -------------- next part -------------- An HTML attachment was scrubbed... URL: