<div dir="ltr">I just wanted to confirm that I saw just this exact same issue today on a stock RHEL 4 Update 5. Running gfs_tool reclaim made my file system "work" again. # rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort ccs-1.0.10-0 ccs-devel-1.0.10-0 cman-1.0.17-0 cman-devel-1.0.17-0 cman-kernel-xenU-2.6.9-50.2 cman-kernheaders-2.6.9-50.2 GFS-6.1.14-0 GFS-kernel-xenU-2.6.9-72.2 magma-1.0.7-1 magma-plugins-1.0.12-0 # uname -rm 2.6.9-55.ELxenU x86_64 <div class="gmail_quote">2008/7/15 Reiner Rottmann <<a href="mailto:rottmann@atix.de">rottmann@atix.de</a>> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Hello everyone, I've experienced strange behavior on a 20 GB GFS formatted volume (although same behaviour applies to smaller and larger sizes) when reaching the max available disk space by writing lots of 256 byte files in a nested directory structure (~15k files in one dir). The expected behaviour would be that all free data blocks are transformed to inodes and metadata as required but although there are still plenty datablocks free, new 256 byte files cannot be created due to "No space left on device". After that, when creating sequential files via touch, it is expected that they are created till all data blocks are transformed in inodes representing the files. When all data blocks are used, "No space left on device" is expected. But in this strange scenario, files are created at random!? Also when executing gfs_tool reclaim, new files are createable again. But gfs_tool reclaim only should increase the number of already available free data blocks by cleaning unused metadata blocks. In my understanding, it should not be necessary to reclaim blocks, if there are still free data blocks left. Has anyone an explanation for this? Best regards, Reiner Rottmann --%<--------------------------------------------------------------------------- (Filesystem filled with 256 byte files.) # for i in $(seq 1 1000); do touch waste.$i; done touch: cannot touch `waste.3': No space left on device touch: cannot touch `waste.6': No space left on device touch: cannot touch `waste.12': No space left on device touch: cannot touch `waste.13': No space left on device touch: cannot touch `waste.15': No space left on device touch: cannot touch `waste.16': No space left on device touch: cannot touch `waste.20': No space left on device touch: cannot touch `waste.25': No space left on device touch: cannot touch `waste.28': No space left on device touch: cannot touch `waste.29': No space left on device touch: cannot touch `waste.32': No space left on device touch: cannot touch `waste.37': No space left on device touch: cannot touch `waste.38': No space left on device touch: cannot touch `waste.39': No space left on device touch: cannot touch `waste.48': No space left on device touch: cannot touch `waste.55': No space left on device touch: cannot touch `waste.56': No space left on device touch: cannot touch `waste.59': No space left on device touch: cannot touch `waste.60': No space left on device touch: cannot touch `waste.63': No space left on device ^C # for i in $(seq 1 1000); do touch waste2.$i; done touch: cannot touch `waste2.1': No space left on device touch: cannot touch `waste2.8': No space left on device touch: cannot touch `waste2.10': No space left on device touch: cannot touch `waste2.11': No space left on device touch: cannot touch `waste2.12': No space left on device touch: cannot touch `waste2.14': No space left on device touch: cannot touch `waste2.17': No space left on device touch: cannot touch `waste2.19': No space left on device touch: cannot touch `waste2.21': No space left on device touch: cannot touch `waste2.24': No space left on device touch: cannot touch `waste2.28': No space left on device touch: cannot touch `waste2.31': No space left on device touch: cannot touch `waste2.32': No space left on device touch: cannot touch `waste2.33': No space left on device touch: cannot touch `waste2.40': No space left on device touch: cannot touch `waste2.43': No space left on device touch: cannot touch `waste2.44': No space left on device touch: cannot touch `waste2.49': No space left on device touch: cannot touch `waste2.54': No space left on device touch: cannot touch `waste2.55': No space left on device touch: cannot touch `waste2.57': No space left on device touch: cannot touch `waste2.58': No space left on device touch: cannot touch `waste2.61': No space left on device ^C # gfs_tool df . /mnt/gfstest: SB lock proto = "lock_dlm" SB lock table = "axqa01:gfstest" SB ondisk format = 1309 SB multihost format = 1401 Block size = 1024 Journals = 3 Resource Groups = 78 Mounted lock proto = "lock_dlm" Mounted lock table = "axqa01:gfstest" Mounted host data = "" Journal number = 0 Lock module flags = Local flocks = FALSE Local caching = FALSE Oopses OK = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 18343309 18343309 0 100% metadata 1690156 1687524 2632 100% data 43931 0 43931 0% # rpm -qa | grep -e 'GFS\|cman\|magma\|ccs'|sort GFS-6.1.15-1 GFS-kernel-2.6.9-60.9 GFS-kernel-2.6.9-75.11 GFS-kernel-smp-2.6.9-60.9 GFS-kernel-smp-2.6.9-75.11 ccs-1.0.11-1 cman-1.0.17-0.el4_6.3 cman-kernel-smp-2.6.9-45.15 cman-kernel-smp-2.6.9-53.8 magma-1.0.8-1 magma-devel-1.0.8-1 magma-plugins-1.0.12-0 # cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 6) # uname -a Linux realserver10 2.6.9-67.0.4.ELsmp #1 SMP Fri Jan 18 05:00:00 EST 2008 x86_64 x86_64 x86_64 GNU/Linux --%<--------------------------------------------------------------------------- -- Gruss / Regards, Dipl.-Ing. (FH) Reiner Rottmann Phone: +49-89 452 3538-12 <a href="http://www.atix.de/" target="_blank">http://www.atix.de/</a> <a href="http://open-sharedroot.org/" target="_blank">http://open-sharedroot.org/</a> PGP Key ID: 0xCA67C5A6 PGP Key Fingerprint = BF59FF006360B6E8D48F26B10D9F5A84CA67C5A6 ** ATIX Informationstechnologie und Consulting AG Einsteinstr. 10 85716 Unterschleissheim Deutschland/Germany Phone: +49-89 452 3538-0 Fax: +49-89 990 1766-0 Registergericht: Amtsgericht Muenchen Registernummer: HRB 168930 USt.-Id.: DE209485962 Vorstand: Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.) Vorsitzender des Aufsichtsrats: Dr. Martin Buss -- Linux-cluster mailing list <a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/linux-cluster" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a> </blockquote></div> -- Dave Costakos mailto:<a href="mailto:david.costakos@gmail.com">david.costakos@gmail.com</a> </div>