From jp at enix.org Wed Aug 12 12:48:34 2009 From: jp at enix.org (=?UTF-8?B?SsOpcsO0bWUgUGV0YXp6b25p?=) Date: Wed, 12 Aug 2009 14:48:34 +0200 Subject: Making an ext3 partition unmountable (intentionally) In-Reply-To: <539316.49835.qm@web111810.mail.gq1.yahoo.com> References: <539316.49835.qm@web111810.mail.gq1.yahoo.com> Message-ID: <4A82BA22.6060801@enix.org> Sri S wrote: > > Hello, > > I?d like to make an ext3 partition that cannot be mounted easily on > another system. I don?t want to use loop-aes to encrypt the partition > because of the performance hit. I also don?t need it to be very > secure, I just need to prevent someone from taking the drive from my > system and reading the data off of it. > > I was thinking about one or a combination of one of options: > > 1. Changing the partition?s ext3 magic number to some other magic > number that only I recognize > 2. Changing the location of the super-block and backup super blocks > Maybe you could simply use a loop-device with the "offset" option. It would be trivial to mount the partition (one would just have to issue the correct options to mount and/or losetup), but recognizing the beginning of the filesystem and guessing the correct offset would require some (minimal) knowledge about EXT3 layout. HTH, From youness254 at gmail.com Tue Aug 18 14:36:40 2009 From: youness254 at gmail.com (Youness HAFFANE) Date: Tue, 18 Aug 2009 16:36:40 +0200 Subject: ext3 real disk occupation of a file Message-ID: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> Hi, I'm trying to understand why extra blocks are added when you create a file in the ext3 fs: for example, let's create a file of 1111 blocks (4096 bytes for each block) $ dd if=/dev/zero of=foo bs=4096 count=1111 the real occupation disk is then: $ du -B 4096 foo 1114 instead of 1111 So where the 3 extra blocks come from? Thanks for help! youness -- Y. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandeen at redhat.com Tue Aug 18 14:45:47 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Tue, 18 Aug 2009 09:45:47 -0500 Subject: ext3 real disk occupation of a file In-Reply-To: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> References: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> Message-ID: <4A8ABE9B.4030606@redhat.com> Youness HAFFANE wrote: > > Hi, > > I'm trying to understand why extra blocks are added when you create a > file in the ext3 fs: > > for example, let's create a file of 1111 blocks (4096 bytes for each block) > > $ dd if=/dev/zero of=foo bs=4096 count=1111 > > the real occupation disk is then: > > $ du -B 4096 foo > 1114 > > instead of 1111 > > So where the 3 extra blocks come from? from the indirect blocks. This is how ext2/3 keep track of the layout of the file. see http://e2fsprogs.sourceforge.net/ext2intro.html and in particular the diagram: http://e2fsprogs.sourceforge.net/ext2-inode.gif in that paper. -Eric > Thanks for help! > > youness > > -- From youness254 at gmail.com Tue Aug 18 15:51:51 2009 From: youness254 at gmail.com (Youness HAFFANE) Date: Tue, 18 Aug 2009 17:51:51 +0200 Subject: ext3 real disk occupation of a file In-Reply-To: <4A8ABE9B.4030606@redhat.com> References: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> <4A8ABE9B.4030606@redhat.com> Message-ID: <1fd0c7cc0908180851g5fc97e49m71e495f49e036eb9@mail.gmail.com> Hi Eric, I have understood from the document, that for each file, we have 3 extra blocks (indirect) that refer to : - the group descriptor - the block bitmap - the inode bitmap Is that right? If not, to what the indirect blocks refer to? and are there always only 3 blocks for each file in ext2/3fs? Best regards youness On Tue, Aug 18, 2009 at 4:45 PM, Eric Sandeen wrote: > Youness HAFFANE wrote: > > > > Hi, > > > > I'm trying to understand why extra blocks are added when you create a > > file in the ext3 fs: > > > > for example, let's create a file of 1111 blocks (4096 bytes for each > block) > > > > $ dd if=/dev/zero of=foo bs=4096 count=1111 > > > > the real occupation disk is then: > > > > $ du -B 4096 foo > > 1114 > > > > instead of 1111 > > > > So where the 3 extra blocks come from? > > from the indirect blocks. This is how ext2/3 keep track of the layout > of the file. > > see http://e2fsprogs.sourceforge.net/ext2intro.html > and in particular the diagram: > http://e2fsprogs.sourceforge.net/ext2-inode.gif > in that paper. > > -Eric > > > Thanks for help! > > > > youness > > > > -- > > -- Y. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandeen at redhat.com Tue Aug 18 18:05:42 2009 From: sandeen at redhat.com (Eric Sandeen) Date: Tue, 18 Aug 2009 13:05:42 -0500 Subject: ext3 real disk occupation of a file In-Reply-To: <1fd0c7cc0908180851g5fc97e49m71e495f49e036eb9@mail.gmail.com> References: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> <4A8ABE9B.4030606@redhat.com> <1fd0c7cc0908180851g5fc97e49m71e495f49e036eb9@mail.gmail.com> Message-ID: <4A8AED76.807@redhat.com> Youness HAFFANE wrote: > Hi Eric, > > I have understood from the document, that for each file, we have 3 extra > blocks (indirect) that refer to : > > - the group descriptor > - the block bitmap > - the inode bitmap > > Is that right? nope :) > If not, to what the indirect blocks refer to? and are there always only > 3 blocks for each file in ext2/3fs? No. basically the extra blocks are for keeping track of where the data blocks are, or for keeping track of where more indirect blocks are, which in turn keep track of where the data blocks are. Maybe http://www.nongnu.org/ext2-doc/ext2.html#DEF-INODES has a better explanation: "There are pointers to the first 12 blocks which contain the file's data in the inode. There is a pointer to an indirect block (which contains pointers to the next set of blocks), a pointer to a doubly-indirect block (which contains pointers to indirect blocks) and a pointer to a trebly-indirect block (which contains pointers to doubly-indirect blocks). " It's all these indirect blocks that you're seeing. -Eric From youness254 at gmail.com Tue Aug 18 21:43:15 2009 From: youness254 at gmail.com (Youness HAFFANE) Date: Tue, 18 Aug 2009 23:43:15 +0200 Subject: ext3 real disk occupation of a file In-Reply-To: <4A8AED76.807@redhat.com> References: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> <4A8ABE9B.4030606@redhat.com> <1fd0c7cc0908180851g5fc97e49m71e495f49e036eb9@mail.gmail.com> <4A8AED76.807@redhat.com> Message-ID: <1fd0c7cc0908181443n21c1288ay937776b984ec6c98@mail.gmail.com> I see, So for only the first 12 blocks, there is no need for extra blocks, but I couldn't find any mathematical logic for the rest. I explain : I've tried to create many files to which I assigned a fixed number of blocks, and the results are : for a 13 block file till a 1036 block file I had 3 extra blocks added and then it's 4 extra blocks added (for example a 3060 block file) and for a 4000 block file, it's 5 extra blocks for 5000 it's 6 and for 10 000 it's 11 blocks added! so is there any logic? or it depends on the machine (regardless if it is ext3, or another fs)? Best regards youness On Tue, Aug 18, 2009 at 8:05 PM, Eric Sandeen wrote: > Youness HAFFANE wrote: > > Hi Eric, > > > > I have understood from the document, that for each file, we have 3 extra > > blocks (indirect) that refer to : > > > > - the group descriptor > > - the block bitmap > > - the inode bitmap > > > > Is that right? > > nope :) > > > If not, to what the indirect blocks refer to? and are there always only > > 3 blocks for each file in ext2/3fs? > > No. basically the extra blocks are for keeping track of where the data > blocks are, or for keeping track of where more indirect blocks are, > which in turn keep track of where the data blocks are. > > Maybe http://www.nongnu.org/ext2-doc/ext2.html#DEF-INODES has a better > explanation: > > "There are pointers to the first 12 blocks which contain the file's data > in the inode. There is a pointer to an indirect block (which contains > pointers to the next set of blocks), a pointer to a doubly-indirect > block (which contains pointers to indirect blocks) and a pointer to a > trebly-indirect block (which contains pointers to doubly-indirect blocks). > " > > It's all these indirect blocks that you're seeing. > > -Eric > -- Y. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Tue Aug 18 22:03:29 2009 From: tytso at mit.edu (Theodore Tso) Date: Tue, 18 Aug 2009 18:03:29 -0400 Subject: ext3 real disk occupation of a file In-Reply-To: <1fd0c7cc0908181443n21c1288ay937776b984ec6c98@mail.gmail.com> References: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> <4A8ABE9B.4030606@redhat.com> <1fd0c7cc0908180851g5fc97e49m71e495f49e036eb9@mail.gmail.com> <4A8AED76.807@redhat.com> <1fd0c7cc0908181443n21c1288ay937776b984ec6c98@mail.gmail.com> Message-ID: <20090818220329.GL28560@mit.edu> On Tue, Aug 18, 2009 at 11:43:15PM +0200, Youness HAFFANE wrote: > I see, > So for only the first 12 blocks, there is no need for extra blocks, but I > couldn't find any mathematical logic for the rest. > I explain : > I've tried to create many files to which I assigned a fixed number of > blocks, and the results are : > for a 13 block file till a 1036 block file I had 3 extra blocks added > and then it's 4 extra blocks added (for example a 3060 block file) > and for a 4000 block file, it's 5 extra blocks > for 5000 it's 6 > and for 10 000 it's 11 blocks added! All of this assumes a filesystem with a 4k block size.... In the inode there are 12 direct block pointers, and then one indirect block pointer, one double indirect block pointer, and one triple indirect block pointer. A 4k block has room for 1024 block numbers. Hence, so files that have between 13 and 1036 data blocks (12+1024), will need a single indirect block. Beyond that a single double indirect block has room for 1024 indirect block pointers. So for the next 1024 blocks, one double indirect block, plus an indirect block is needed. For the next 1024 data blocks, another indirect block is needed. After the 1024 slots in the double indirect blocks are filled, then we use a triple indirect block, which has room for 1024 double indirect blocks, and so on. If the filesystem is using 1k block sizes, then there is room for 256 block numbers in each block, so the indirect block can address 256 data blocks, and then the double indirect block in the inode can can address (256*256) data blocks while using 256 indirect numbers, and the triple indirect block can address 256 double indirect blocks, which when using 256*256 indirect blocks, can address up to 256*256*256 data blocks. With sparse files, indirect blocks and double indirect blocks are aren't needed to address data blocks won't be allocated. But for contiguous files it's relatively easy to predict in advance how much space is needed for the indirect, double indirect, and triple indirect blocks. - Ted From youness254 at gmail.com Tue Aug 18 23:09:20 2009 From: youness254 at gmail.com (Youness HAFFANE) Date: Wed, 19 Aug 2009 01:09:20 +0200 Subject: ext3 real disk occupation of a file In-Reply-To: <20090818220329.GL28560@mit.edu> References: <1fd0c7cc0908180736k3b97fe66vc2b5f1434987e667@mail.gmail.com> <4A8ABE9B.4030606@redhat.com> <1fd0c7cc0908180851g5fc97e49m71e495f49e036eb9@mail.gmail.com> <4A8AED76.807@redhat.com> <1fd0c7cc0908181443n21c1288ay937776b984ec6c98@mail.gmail.com> <20090818220329.GL28560@mit.edu> Message-ID: <1fd0c7cc0908181609g6a94557dm19633b56ba49e1e@mail.gmail.com> Now I can see great logic after this perfect explanation. Thanks a lot Eric and Ted. Best regards youness On Wed, Aug 19, 2009 at 12:03 AM, Theodore Tso wrote: > On Tue, Aug 18, 2009 at 11:43:15PM +0200, Youness HAFFANE wrote: > > I see, > > So for only the first 12 blocks, there is no need for extra blocks, but I > > couldn't find any mathematical logic for the rest. > > I explain : > > I've tried to create many files to which I assigned a fixed number of > > blocks, and the results are : > > for a 13 block file till a 1036 block file I had 3 extra blocks added > > and then it's 4 extra blocks added (for example a 3060 block file) > > and for a 4000 block file, it's 5 extra blocks > > for 5000 it's 6 > > and for 10 000 it's 11 blocks added! > > All of this assumes a filesystem with a 4k block size.... > > In the inode there are 12 direct block pointers, and then one indirect > block pointer, one double indirect block pointer, and one triple > indirect block pointer. > > A 4k block has room for 1024 block numbers. Hence, so files that have > between 13 and 1036 data blocks (12+1024), will need a single indirect > block. Beyond that a single double indirect block has room for 1024 > indirect block pointers. So for the next 1024 blocks, one double > indirect block, plus an indirect block is needed. For the next 1024 > data blocks, another indirect block is needed. After the 1024 slots > in the double indirect blocks are filled, then we use a triple > indirect block, which has room for 1024 double indirect blocks, and so > on. > > If the filesystem is using 1k block sizes, then there is room for 256 > block numbers in each block, so the indirect block can address 256 > data blocks, and then the double indirect block in the inode can can > address (256*256) data blocks while using 256 indirect numbers, and > the triple indirect block can address 256 double indirect blocks, > which when using 256*256 indirect blocks, can address up to > 256*256*256 data blocks. > > With sparse files, indirect blocks and double indirect blocks are > aren't needed to address data blocks won't be allocated. But for > contiguous files it's relatively easy to predict in advance how much > space is needed for the indirect, double indirect, and triple indirect > blocks. > > - Ted > -- Y. -------------- next part -------------- An HTML attachment was scrubbed... URL: From web2009 at zeroreality.com Sun Aug 30 16:00:12 2009 From: web2009 at zeroreality.com (z0diac) Date: Sun, 30 Aug 2009 09:00:12 -0700 (PDT) Subject: How many files can I have safely in a subdirectory? Message-ID: <25212801.post@talk.nabble.com> Ok, I'm running a vBulletin forum (3.8.4) and found that all user attachments go into 1 single directory for each user. For each attached file in the forum, there's 2 files on disk (*.attach and *.thumb), for pictures that is. One user already has over 100,000 attachements, thus, over 200,000 files in his attach directory. Someone recently told me to 'keep an eye on it' because certain setups can't hold more than X number of files in a single directory. Yet someone else said I could have over 1 trillion files in a single directory if the HDD was large enough... Here's my setup: linux version: 2.6.18-92.1.10.el5 php: 5.1.6 mySQL: 5.0.45 File system: ext3 vB support has told me any limitation there might be, will not be the result of vB, so now I'm looking at either Linux and the way it handles files, or the ext3 file system. Does anyone know if I can just keep going with putting files into one directory? (there will be over 1million probably by year's end. Hopefully not more than 5 million ever). And, will having so many files in a single directory cause any performance problems? (ie: slowdowns) My only option is to hire a coder to somehow have it split the 1M+ files into several subdirs, say 50,000 per subdir. But even though it's messy, if it really doesn't make a difference in the end whether they're in 50 subdirs, or just 1 dir, then I won't bother (and can sigh a breath of relief) Thanks in advance!! z0diac is offline Looking for Linux Hosting? Click Here. -- View this message in context: http://www.nabble.com/How-many-files-can-I-have-safely-in-a-subdirectory--tp25212801p25212801.html Sent from the Ext3 - User mailing list archive at Nabble.com.