cpio to ext4 seems much slower than to ext2, ext3 or xfs

[Ric Wheeler]

On 11/12/2009 03:27 PM, Eric Sandeen wrote:
> Roberto Ragusa wrote:
>> Ric Wheeler wrote:
>>> In our testing with f12, I build a 60TB ext4 file system with 1 billion
>>> small files. A forced fsck of ext4 finished in 2.5 hours give or take a
>>> bit :-) The fill was artificial and the file system was not aged, so
>>> real world results will probably be slower.
>>>
>>> fsck time scales mostly with the number of allocated files in my
>>> experience. Allocated blocks (fewer very large files) are quite quick.
>>>
>>
>> What kind of machine did you use?
>>
>> With 60TB a simple allocation bitmap for 4k-blocks takes almost 2GB;
>> and this is just to detect free space or double allocation of blocks.
>> Wow.
>>
>
> The box did have a lot of memory, it's true :)
>
> But ext4 also uses the "uninit_bg" feature:
>
> uninit_bg
>    Create  a filesystem without initializing all of the
>    block groups.  This feature also  enables  checksums
>    and  highest-inode-used  statistics  in  each block-
>    group.  This feature can speed  up  filesystem  cre-
>    ation   time   noticeably  (if  lazy_itable_init  is
>    enabled), and can also reduce e2fsck  time  dramati-
>    cally.   It is only supported by the ext4 filesystem
>    in recent Linux kernels.
>
> -Eric
>

A lot in this case was 40GB of DRAM - fsck (iirc) consumed about 13GB of virtual 
space during the run?

Ric