<div dir="ltr">Try to check the number of inodes using "df -i", that can be 100%. If it's true, you need to change the max number of inodes, or remove some files.</div> <div class="gmail_quote"><div dir="ltr">On Wed, Sep 9, 2015 at 11:29 AM Vladislav Bogdanov <<a href="mailto:bubble@hoster-ok.com">bubble@hoster-ok.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Bob, 02.09.2015 15:36, Bob Peterson wrote: > ----- Original Message ----- >> Hi, >> >> I've got weird state on GFS2 (activated only on one node from the very >> beginning, but with dlm locking), when I'm unable to write with 'No >> space left on device' error, but df -m reports: >> /dev/mapper/vg_shared-lv_shared 570875 569622 1254 100% /storage/staging >> >> Umount/mount doesn't help, umount/fsck/rmmod/mount also does nothing. >> >> That is centos6 with 2.6.32-504.30.3.el6.x86_64 kernel. >> >> What could be the reason for such desync? >> Is there a way to fix that? >> >> Any help is appreciated, >> >> thank you, >> Vladislav > > Hi Vladislav, > > It sounds like maybe your system statfs file has gotten out of sync with > the actual free space. We've seen this before, and have bugzilla records > open to fix it. <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1191219" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1191219</a> > Is there something I can do to help solving this issue? Best, Vladislav > Ordinarily that should not prevent blocks from being allocated, because > unlinked dinodes should be automatically reclaimed as needed. > It could be a fragmentation issue: Maybe there's enough free space, but > the free space is too fragmented to allow for a required block allocation. > > So it is difficult to say what exactly is going on. If you want to send > me your file system metadata, I'd be happy to examine it and let you > know what I find. This can be saved with: gfs2_edit savemeta <device> <output file> > The resulting files are often too big to email, so you may need to put > it on an ftp server or something instead. > > Also, bear in mind that GFS2 has a severe performance penalty when your > file system is nearly full. The less free space available, the more time > it takes to find free space. So you'll probably get much better performance > if you make the file system bigger (lvextend the volume then gfs2_grow). > > Regards, > > Bob Peterson > Red Hat File Systems > -- Linux-cluster mailing list <a href="mailto:Linux-cluster@redhat.com" target="_blank">Linux-cluster@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/linux-cluster" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/linux-cluster</a> </blockquote></div>