[Linux-cluster] GFS2 'No space left on device' while df reports 1gb is free

[Franciscon]

Try to check the number of inodes using "df -i", that can be 100%. If it's
true, you need to change the max number of inodes, or remove some files.

On Wed, Sep 9, 2015 at 11:29 AM Vladislav Bogdanov <bubble at hoster-ok.com>
wrote:

> Hi Bob,
>
> 02.09.2015 15:36, Bob Peterson wrote:
> > ----- Original Message -----
> >> Hi,
> >>
> >> I've got weird state on GFS2 (activated only on one node from the very
> >> beginning, but with dlm locking), when I'm unable to write with 'No
> >> space left on device' error, but df -m reports:
> >> /dev/mapper/vg_shared-lv_shared 570875 569622 1254 100% /storage/staging
> >>
> >> Umount/mount doesn't help, umount/fsck/rmmod/mount also does nothing.
> >>
> >> That is centos6 with 2.6.32-504.30.3.el6.x86_64 kernel.
> >>
> >> What could be the reason for such desync?
> >> Is there a way to fix that?
> >>
> >> Any help is appreciated,
> >>
> >> thank you,
> >> Vladislav
> >
> > Hi Vladislav,
> >
> > It sounds like maybe your system statfs file has gotten out of sync with
> > the actual free space. We've seen this before, and have bugzilla records
> > open to fix it. https://bugzilla.redhat.com/show_bug.cgi?id=1191219
> >
>
> Is there something I can do to help solving this issue?
>
> Best,
> Vladislav
>
> > Ordinarily that should not prevent blocks from being allocated, because
> > unlinked dinodes should be automatically reclaimed as needed.
> > It could be a fragmentation issue: Maybe there's enough free space, but
> > the free space is too fragmented to allow for a required block
> allocation.
> >
> > So it is difficult to say what exactly is going on. If you want to send
> > me your file system metadata, I'd be happy to examine it and let you
> > know what I find. This can be saved with: gfs2_edit savemeta <device>
> <output file>
> > The resulting files are often too big to email, so you may need to put
> > it on an ftp server or something instead.
> >
> > Also, bear in mind that GFS2 has a severe performance penalty when your
> > file system is nearly full. The less free space available, the more time
> > it takes to find free space. So you'll probably get much better
> performance
> > if you make the file system bigger (lvextend the volume then gfs2_grow).
> >
>
>
>
> > Regards,
> >
> > Bob Peterson
> > Red Hat File Systems
> >
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20150909/bf572580/attachment.htm>