question?

[James Wilkinson]

Dan Ladd wrote:
> Yeah this is the only email address i could find because I had a
> question about the fedora project.  I didn't konw where to direct it. 
> So here goes... I was wondering about the WinFS file management
> system.   If something like that (meaning the built in database
> system) was going to be included in fedora or other RedHat operating
> systems or if something like is already in an operating system with
> RedHat?  It seems like it is old technology because the AS/400 or now
> the iSeries has the built in database system to locate physical files
> that are stored in logical files.   If you could direct me to where i
> can have that answered that would be awesome.

Oohh. *BIG* question.

Background: Unix invented much of the "everything is a file, stored in a
file tree, and the OS just sees a normal file as a collection of bits"
philosophy that is now more or less standard.  Meanwhile, the relational
database became the standard way of storing smaller, structured data.

*Lots* of computer scientists have been wondering whether this split
between ways of storing data is ideal. So there has been a lot of work
done looking at the best ways to store general-purpose files in a
database.

It turns out that this is a Hard Problem. Storing files as opaque,
binary objects in a database isn't a problem, a lot of modern
filesystems effectively do this. The question is whether we can take
anything else from the database world.

Here you should understand that there is no agreed vision of how things
should work. This is the main point I want to make. So you will have to
work out what you want from a database filesystem, and see what provides
it.

The two big problems come under the heading of writing and reading.
Writing is relatively easy, since you can define the problem: It Would Be
Nice If Linux allowed multiple updates to one file or to many files to be
treated as a transaction. Even there, there is the unfortunate detail
of getting transactions to span filesystems.

Reading is more of a problem. Many file formats keep internal metadata
(author, image size, artist, etc.), and there is a demand to keep more
data against files (e.g. Access Control Lists). Many people think that
there should be a better way of finding all recordings of Vaughan
Williams' works than finding all MP3s, all OGGs, all Real Media, etc.
and running format-specific query programs against each file. (One still
has a problem if some-one entered "Old Hundredth, arr. RWV", but that
can be ignored in the first few versions...) Maybe icons for a file
should be stored against the file.

This is the metadata problem, or rather, series of problems. One is,
simply, how do you present metadata under Unix-like systems? Solaris
has a special system call and program to access the metadata: Hans
Reiser is proposing to allow you to access each file as a directory
with the metadata available underneath (obviously, this isn't practical
with real directories).

The other big problem is how much metadata should move around with a
file. It's obvious that you want to be able to export files in an
existing format, which will drop any metadata that isn't already in the
format.  (You still need to support existing filesystems, for example on
CDs).

Then when you're copying things around, some metadata (user to last
modify) should change, others (user to first modify) shouldn't. This
means that something is going to have to know a *lot* about the way that
metadata works, which means you are going to have a lot of per-filetype
programming and/or a lot of rigidity.

The main contender to "solve" both of these problems is the Reiser 4
filesystem. This is still very new, and has a number of problems with it.

 * It's very new, not fully debugged, and has a number of security and
   reliability problems.

 * It stores metadata to a file by treating the file as a directory, and
   putting metadata as pseudo-files in that directory. That changes the
   way users and programs think about files, and will invalidate a lot
   of assumputions.

See http://lwn.net/Articles/14035/ .

Other contenders include user-space plugins to the Gnome or KDE virtual
filesystems. These can be reasonably taught "this is an MP3, this is an
XML document", and retrieve the meta-data on demand.

It still isn't clear how best to make this visible to non-technical end-
users. It's largely those people who *aren't* happy with shell scripting
who would most benefit for easy ways to look for files with Vaughan
Williams recordings. (Those who can will probably have the sense to put
RVW in the pathname somewhere, and can use custom tools).

On top of this, maybe some files should be word-indexed. It doesn't make
sense doing this for Ogg files, though: Microsoft's Find Fast has long
done this in userspace with a separate database, and this does seem much
better than putting the suppot in the kernel.

I don't know much about OS/400: it always sounded as though they
implemented the database first, and then created the entire OS and
related applications around the database. They had the advantage that
everything knew it was going to be working with a database, and
progams on OS/400 probably really want a database backend anyway.

Linux doesn't have that, and is a much more general purpose OS.

I've also come across http://lwn.net/Articles/56923/ : you might want to
read that, too.

Note WinFS itself appears to be delayed until the end of the decade.

Sorry for the length of this e-mail: there's more I could say, but won't.

James.
-- 
E-mail address: james | Examiner: How does an AC motor start?
@westexe.demon.co.uk  | Student: vrrrrrrrrrrRrRRRRRRR...
                      | Examiner: Stop! Stop!
                      | Student: RRRRRRRmmmmm.