Setting up large (12.5 TB) filesystem howto?

[Nigel Wade]

Götz Reinicke - IT-Koordinator wrote:
> Hi,
> 
> I'm trying to set up an iscsi 12.5 TB storage for some data backup.
> 
> Doing so, I had some difficulties to find the right tool, maybe it's
> also a question of the system settings...
> 
> The server is a 32Bit Red Hat EL 5.3 with the recent updates. Ths iscsi
> connection can be establised.
> 
> fdisk and parted fail to create any information on the device or fail
> completely.
> 
> using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly
> create a logical volume:
> 
> lvdisplay /dev/VolGroup02/lvol0
>   --- Logical volume ---
>   LV Name                /dev/VolGroup02/lvol0
>   VG Name                VolGroup02
>   LV UUID                h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x
>   LV Write Access        read/write
>   LV Status              available
>   # open                 0
>   LV Size                12,73 TB
>   Current LE             3337487
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     256
>   Block device           253:4
> 
> 
> But, I can't create a filesystem on it:
> 
> mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0
> /dev/mapper/VolGroup02-lvol0
> 
> 
> mke2fs 1.39 (29-May-2006)
> mkfs.ext3: Filesystem too large.  No more than 2**31-1 blocks
> 	 (8TB using a blocksize of 4k) are currently supported.
> 
> 
> The limits information provided by red hat say, that RH EL 5.1 supports
> 16 TB filesystems:
> 
> http://www.redhat.com/rhel/compare/
> 
> 	-> Maximum filesystem size (Ext3): 16TB in 5.1
> 
> Using a block size of 8192 gives a warning, that this size is to large
> for that system.
> 
> 
> So my question: What is my missunderstanding or what's wrong with my
> system? Where are the real limits? Do I have to switch the OS to 64 Bit?

The problem is a limitation in ext2/ext3. With a PAGESIZE of 4kB (which is the 
default on most PC architectures) the limit to the size of an ext2/ext3 
filesystem is 8TB. If you are able to increase the PAGESIZE then you can 
probably increase the max. filesystem size.

 From some very cursory reading it seems that the Alpha, Itanium and PowerPC 
processors may support larger PAGESIZE, but I am not certain of that. I don't 
know whether the normal 4kB PAGESIZE is a hardware limit or just an optimal setting.

If you really do need a large filesystem then you will have to do some reading 
up on whether your system can support a larger PAGESIZE. Otherwise you may be 
able to split the external storage into multiple LUNs (if the hardware provides 
that option) and create multiple smaller filesystems, one on each LUN.

-- 
Nigel Wade, System Administrator, Space Plasma Physics Group,
             University of Leicester, Leicester, LE1 7RH, UK
E-mail :    nmw at ion.le.ac.uk
Phone :     +44 (0)116 2523548, Fax : +44 (0)116 2523555