[Linux-cluster] very poor rm performance with gfs2 (el6)

[Sophana K]

Hello

I'm currently testing gfs2 under EL6 (2.6.32-71.18.1.el6.x86_64) 11
nodes over gigabit  iscsi (target is IET on EL5)
Application is a compute cluster with lot of files of various sizes.
Application need close to write coherency and cache coherency.

I can't tell yet about performance. It looks not bad, iscsi target
shows a bandwidth of about 80Mbytes/s (mostly reads during compute). I
have to compare to NFS to tell if it is good.

Major problem: I noticed very very poor rm performance, down to 1 to 2
files erased per second in a rm -rf.
Some of these files are not very big: a few kilo bytes. Cluster is
idle during rm. mkfs.gfs2 was made just before compute test was made.

here are some small strace taken from rm -rf:
12:32:38.824237 unlinkat(6, "cycle.c", 0) = 0
12:32:39.226444 unlinkat(6, "cycle.h", 0) = 0
12:32:40.277366 unlinkat(6, "cycle_.h", 0) = 0

...

13:38:09.719444 unlinkat(5, "BoxGrp._d988887c.+.vlog.vlog", 0) = 0
13:38:10.125942 unlinkat(5, "BoxGrp._d8335657.+.vlog.vlog.tog", 0) = 0
13:38:10.126037 unlinkat(5, "BoxGrp._8f2f6edf.+.vlog.vlog", 0) = 0
13:38:11.139070 unlinkat(5, "BoxGrp._cfc41f42..pkl", 0) = 0
13:38:11.139217 unlinkat(5, "BoxGrp._aec657d9..hier", 0) = 0
13:38:11.147072 unlinkat(5, "BoxGrp._cfc41f42..hier", 0) = 0
13:38:11.353478 unlinkat(5, "BoxGrp._47b2df42.+.vlog.vlog.tog", 0) = 0
13:38:11.575804 unlinkat(5, "BoxGrp._32107b60..hier", 0) = 0
13:38:12.637765 unlinkat(5, "BoxGrp._2e795200.+.vlog.vlog", 0) = 0
13:38:13.049503 unlinkat(5, "BoxGrp._32107b60..pkl", 0) = 0
13:38:13.468287 unlinkat(5, "BoxGrp._403da4c3..pkl", 0) = 0
13:38:14.107020 unlinkat(5, "BoxGrp._f07dc01b..pkl", 0) = 0
13:38:14.112050 unlinkat(5, "BoxGrp._27228204..hier", 0) = 0
13:38:14.144840 unlinkat(5, "BoxGrp._35ef97b2.+.vlog.vlog", 0) = 0
13:38:15.593250 unlinkat(5, "BoxGrp._47b2df42..pkl", 0) = 0
13:38:15.593437 unlinkat(5, "BoxGrp._e8cd3f8d..hier", 0) = 0
13:38:15.830560 unlinkat(5, "BoxGrp._35ef97b2.+.vlog.vlog.tog", 0) = 0
13:38:16.414388 unlinkat(5, "BoxGrp._403da4c3.+.vlog.vlog", 0) = 0
13:38:16.619618 unlinkat(5, "BoxGrp._d8335657..hier", 0) = 0
13:38:16.824062 unlinkat(5, "BoxGrp._d988887c.+.vlog.vlog.tog", 0) = 0
13:38:16.824397 unlinkat(5, "BoxGrp._808fb9e4..pkl", 0) = 0
13:38:17.077052 unlinkat(5, "BoxGrp._341e8e1f..pkl", 0) = 0
13:38:17.087327 unlinkat(5, "BoxGrp._47b2df42.+.vlog.vlog", 0) = 0
13:38:17.899539 unlinkat(5, "BoxGrp._341e8e1f..hier", 0) = 0
13:38:18.100736 unlinkat(5, "BoxGrp._d8335657.+.vlog.vlog", 0) = 0
13:38:18.319269 unlinkat(5, "BoxGrp._70bf96ba..pkl", 0) = 0
13:38:18.757326 unlinkat(5, "BoxGrp._d988887c..hier", 0) = 0
13:38:18.975545 unlinkat(5, "BoxGrp._35ef97b2..pkl", 0) = 0
13:38:19.568433 unlinkat(5, "BoxGrp._bb280eee..pkl", 0) = 0


Please advise if there is a solution for this.

Thanks