[Linux-cluster] gfs2 v. zfs?

[Wendy Cheng]

Comments in-line ...

On Mon, Jan 24, 2011 at 6:55 PM, Jankowski, Chris
<Chris.Jankowski at hp.com> wrote:
> A few comments, which might contrast uses of GFS2 and XFS in enterprise class production environments:
>
> 1.
> SAN snapshot is not a panacea, as it is only crash consistent and only within a single LUN.
> If you have your data or database spread over multiple LUNs each with its own filesystem,
> then you are on your own.

It depends on the SAN box. Some products have aggregate level
snapshots that can contain multiple LUNs.

However, the argument here is correct; that is, SAN snaphost is not a
panacea. Other than different SAN vendors may have different setup(s),
snapshot restore could require specific knowledge of the filesystem
involved (e.g. how the journal is replayed). So there are integration
and test  efforts required for the restore to work well.

>
> 2.
> Therefore, we still need at least OS level (filesystem level) consistent backup
> if the application itself does not provide a hot backup mechanism, which very few do.
> The consistent filesystem level backup requires freeze and thaw commands.
> XFS offers them, GFS2 does not.

I seem to see GFS2 having freeze/thaw patches in the past ? But for
backup to work well, it requires more than freeze/thaw.

>
> 3.
> GFS2 provides only tar(1) as a backup mechanism.
> Unfortunately, tar(1) does not cope efficiently with sparse files,
> which many applications create.
> As an exercise create a 10 TB sparse file with just one byte of non-null data at the end.
> Then try to back it up to disk using tar(1).
> The tar image will be correctly created, but it will take many, many hours.
> Dump(8) would do the job in a blink, but is not available for GFS2 filesystem.
> However, XFS does have XFS specific dump(8) command and will backup sparse files
> efficiently.
>
> 4.
> GFS2 is very convenient to use, as by its nature is clusterised.
> However, there is huge performance cost to pay for all this convenience.
> This cost stems from serialization imposed by distributed lock manager.
>
> 5.
> For these reason, for the HA applications running on one node at a time,
> I found that XFS on top of LVM gives me the best mix of performance and functionality:
> - high performance
> - efficient backup of sparse files
> - backup consistency through freeze/thaw
> - zero downtime backup through use of LVM snapshots
> - short failover times due to efficient XFS transaction logs
>
> So, for this type of HA applications (failover HA) and environment,
> it makes perfect sense to use XFS in a cluster instead of GFS2.
>
> Having said that, GFS2 can, in principle, be engineered to be much better
> for failover HA applications.
>
> It would require development of:
> - GFS2 specific dump(8)
> - GFS2 specific freeze and thaw commands
> - CLVM wide snapshots
> - more efficient DLM

You did a great summary here. By looking at the list, I would imagine
CLVM snapshoting is probably the easiest, technically and politically.
It's all up to GFS2 engineers to take the note.

>
> It certainly is possible to do. Digital/Compaq/HP TruCluster Cluster File System (CFS) built on top of AdvFS had all of these features and much, much more by circa year 2000.
>

Yep, I met a TruCluster developer 3 years ago. Based on his
description, I was impressed. Not sure HP is still marketing it
though.

Again, a great summary !

-- Wendy