parallel I/O on shared-memory multi-CPU machines

[Andreas Dilger]

On Apr 02, 2007  05:57 -0700, i.vlad at yahoo.com wrote:
> My question is -- can the filesystem actually read/write to files truly
> in parallel (more processors -- faster read/write), or if I write such
> a code the I/O commands from the CPUs will just queue after each other
> and it would be the same thing as using a single CPU? If parallel I/O
> is possible, can it be accomplished entirely transparently, or using
> special libraries, or only in special circumstances, like reading in
> parallel with N CPUs from N different physical disks? Or only on some
> types of hardware? Is there a max nr of threads/processes that can
> write to disk in parallel? If ext3 does not do this, which (stable)
> Linux filesystem does it?

You can do this with Lustre (www.lustre.org), which is a GPL parallel,
distributed filesystem that runs on top of ext3 (essentially ext4 though).
Since Lustre does its own distributed extent locking on a file, it is
possible to do fully parallel IO to a single file from multiple nodes.

> I know that the vast majority of clusters use something else than ext3
> (NFS, Lustre, etc), but the question still stands because: (1) individual
> nodes in commodity clusters do have very often individual ext3 disks that
> are used for temporary files (intermediate computational results); (2)
> grid computers made of standalone user machines are likely to have the
> most common filesystem, ext3; (3) There are scientific data processing
> steps that need to be done on a single shared-memory machine because of
> intensive data exchange between CPUs. (4) Software development is easier
> to do on a single machine (i.e. powerful multi-core laptop).

We are looking to make even single-node file locking better than just
i_mutex (i.e. fine-grained extent locking in the VFS) but this is not
a high priority task for us, since we target large systems mostly.
If anyone is interested in doing such work we would definitely want
to help out with it.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.