[Linux-cluster] GFS volume filled to the brim - "No space left on device" although still data blocks free
Reiner Rottmann
rottmann at atix.de
Tue Jul 15 07:44:23 UTC 2008
Hello everyone,
I've experienced strange behavior on a 20 GB GFS formatted volume (although
same behaviour applies to smaller and larger sizes) when reaching the max
available disk space by writing lots of 256 byte files in a nested directory
structure (~15k files in one dir).
The expected behaviour would be that all free data blocks are transformed to
inodes and metadata as required but although there are still plenty
datablocks free, new 256 byte files cannot be created due to "No space left
on device".
After that, when creating sequential files via touch, it is expected that they
are created till all data blocks are transformed in inodes representing the
files. When all data blocks are used, "No space left on device" is expected.
But in this strange scenario, files are created at random!?
Also when executing gfs_tool reclaim, new files are createable again. But
gfs_tool reclaim only should increase the number of already available free
data blocks by cleaning unused metadata blocks.
In my understanding, it should not be necessary to reclaim blocks, if there
are still free data blocks left.
Has anyone an explanation for this?
Best regards,
Reiner Rottmann
--%<---------------------------------------------------------------------------
(Filesystem filled with 256 byte files.)
# for i in $(seq 1 1000); do touch waste.$i; done
touch: cannot touch `waste.3': No space left on device
touch: cannot touch `waste.6': No space left on device
touch: cannot touch `waste.12': No space left on device
touch: cannot touch `waste.13': No space left on device
touch: cannot touch `waste.15': No space left on device
touch: cannot touch `waste.16': No space left on device
touch: cannot touch `waste.20': No space left on device
touch: cannot touch `waste.25': No space left on device
touch: cannot touch `waste.28': No space left on device
touch: cannot touch `waste.29': No space left on device
touch: cannot touch `waste.32': No space left on device
touch: cannot touch `waste.37': No space left on device
touch: cannot touch `waste.38': No space left on device
touch: cannot touch `waste.39': No space left on device
touch: cannot touch `waste.48': No space left on device
touch: cannot touch `waste.55': No space left on device
touch: cannot touch `waste.56': No space left on device
touch: cannot touch `waste.59': No space left on device
touch: cannot touch `waste.60': No space left on device
touch: cannot touch `waste.63': No space left on device
^C
# for i in $(seq 1 1000); do touch waste2.$i; done
touch: cannot touch `waste2.1': No space left on device
touch: cannot touch `waste2.8': No space left on device
touch: cannot touch `waste2.10': No space left on device
touch: cannot touch `waste2.11': No space left on device
touch: cannot touch `waste2.12': No space left on device
touch: cannot touch `waste2.14': No space left on device
touch: cannot touch `waste2.17': No space left on device
touch: cannot touch `waste2.19': No space left on device
touch: cannot touch `waste2.21': No space left on device
touch: cannot touch `waste2.24': No space left on device
touch: cannot touch `waste2.28': No space left on device
touch: cannot touch `waste2.31': No space left on device
touch: cannot touch `waste2.32': No space left on device
touch: cannot touch `waste2.33': No space left on device
touch: cannot touch `waste2.40': No space left on device
touch: cannot touch `waste2.43': No space left on device
touch: cannot touch `waste2.44': No space left on device
touch: cannot touch `waste2.49': No space left on device
touch: cannot touch `waste2.54': No space left on device
touch: cannot touch `waste2.55': No space left on device
touch: cannot touch `waste2.57': No space left on device
touch: cannot touch `waste2.58': No space left on device
touch: cannot touch `waste2.61': No space left on device
^C
# gfs_tool df .
/mnt/gfstest:
SB lock proto = "lock_dlm"
SB lock table = "axqa01:gfstest"
SB ondisk format = 1309
SB multihost format = 1401
Block size = 1024
Journals = 3
Resource Groups = 78
Mounted lock proto = "lock_dlm"
Mounted lock table = "axqa01:gfstest"
Mounted host data = ""
Journal number = 0
Lock module flags =
Local flocks = FALSE
Local caching = FALSE
Oopses OK = FALSE
Type Total Used Free use%
------------------------------------------------------------------------
inodes 18343309 18343309 0 100%
metadata 1690156 1687524 2632 100%
data 43931 0 43931 0%
# rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort
GFS-6.1.15-1
GFS-kernel-2.6.9-60.9
GFS-kernel-2.6.9-75.11
GFS-kernel-smp-2.6.9-60.9
GFS-kernel-smp-2.6.9-75.11
ccs-1.0.11-1
cman-1.0.17-0.el4_6.3
cman-kernel-smp-2.6.9-45.15
cman-kernel-smp-2.6.9-53.8
magma-1.0.8-1
magma-devel-1.0.8-1
magma-plugins-1.0.12-0
# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)
# uname -a
Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008
x86_64 x86_64 x86_64 GNU/Linux
--%<---------------------------------------------------------------------------
--
Gruss / Regards,
Dipl.-Ing. (FH) Reiner Rottmann
Phone: +49-89 452 3538-12
http://www.atix.de/
http://open-sharedroot.org/
PGP Key ID: 0xCA67C5A6
PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6
**
ATIX Informationstechnologie und Consulting AG
Einsteinstr. 10
85716 Unterschleissheim
Deutschland/Germany
Phone: +49-89 452 3538-0
Fax: +49-89 990 1766-0
Registergericht: Amtsgericht Muenchen
Registernummer: HRB 168930
USt.-Id.: DE209485962
Vorstand:
Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.)
Vorsitzender des Aufsichtsrats:
Dr. Martin Buss
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20080715/f8d516ef/attachment.sig>
More information about the Linux-cluster
mailing list