[Linux-cluster] GFS2 'No space left on device' while df reports 1gb is free

[Vladislav Bogdanov]

02.09.2015 15:36, Bob Peterson wrote:
> ----- Original Message -----
>> Hi,
>>
>> I've got weird state on GFS2 (activated only on one node from the very
>> beginning, but with dlm locking), when I'm unable to write with 'No
>> space left on device' error, but df -m reports:
>> /dev/mapper/vg_shared-lv_shared 570875 569622 1254 100% /storage/staging
>>
>> Umount/mount doesn't help, umount/fsck/rmmod/mount also does nothing.
>>
>> That is centos6 with 2.6.32-504.30.3.el6.x86_64 kernel.
>>
>> What could be the reason for such desync?
>> Is there a way to fix that?
>>
>> Any help is appreciated,
>>
>> thank you,
>> Vladislav
>
> Hi Vladislav,

Thank you Bob for a rapid answer!

>
> It sounds like maybe your system statfs file has gotten out of sync with
> the actual free space. We've seen this before, and have bugzilla records
> open to fix it. https://bugzilla.redhat.com/show_bug.cgi?id=1191219

That is not public unfortunately. Anyways, I'm willing to help to fix 
that as the project I'm currently work on is relying on GFS2 to be fast, 
stable and free from such defects.

>
> Ordinarily that should not prevent blocks from being allocated, because
> unlinked dinodes should be automatically reclaimed as needed.
> It could be a fragmentation issue: Maybe there's enough free space, but
> the free space is too fragmented to allow for a required block allocation.
>
> So it is difficult to say what exactly is going on. If you want to send
> me your file system metadata, I'd be happy to examine it and let you
> know what I find. This can be saved with: gfs2_edit savemeta <device> <output file>
> The resulting files are often too big to email, so you may need to put
> it on an ftp server or something instead.

Yes, that is a filesystem used only for tests so I can provide its 
metadata. I'll sent a link to you when it is ready.
Should fs be unmounted btw?

>
> Also, bear in mind that GFS2 has a severe performance penalty when your
> file system is nearly full. The less free space available, the more time
> it takes to find free space. So you'll probably get much better performance
> if you make the file system bigger (lvextend the volume then gfs2_grow).

That is actually a desired behavior, more, I use controlled IO 
throttling when it becomes full. That is temporary tier-0 area, all 
files go to another storage asynchronously.

Thank you,

Vladislav