[Linux-cluster] gfs2 mount: No space left on device

Mon Aug 27 19:00:23 UTC 2012

Hi Steven,

Even though i stumbled upon this issue multiple times already, I can't 
really fully reproduce it yet. It always happens to me when i'm 
live-migrating ( and libvirt create a bleh.xml.new file inthere ).

What gfs2_edit command output should i provide to you when it happens 
again ( and i'm sure it will )? I'll make sure to post it here on the 
list as soon as it occurs. Some other
commands,logs,outputs you think can be handy to figure this one out? I 
feel like an Ubuntu cluster guinnea pig :)

Thanks for your help so far!

Kind regards,

Bart

Steven Whitehouse schreef op 27.08.2012 14:53:
> Hi,
>
> On Thu, 2012-08-23 at 22:35 +0200, Bart Verwilst wrote:
>> Umounting and remounting made the filesystem writeable again.
>>
>> I've then ran a gfs2_fsck on the device, which gave me
>>
>
> The output from fsck doesn't really give any clues as to the cause.
>
> The reclaiming of unlinked inodes is a fairly normal thing to see,
> particularly if there has been some kind of crash just before running
> fsck and it is nothing to worry about.
>
> The real issue is why you got this out of space error in the first
> place, when there appears to be plenty of free blocks left. It would 
> be
> worth checking with gfs2_edit just to be sure that the allocation
> bitmaps are not full, even if the summary information says otherwise.
>
> Can you easily reproduce this issue, or is this something that has 
> just
> occurred once?
>
> Steve.
>
>
>> root at vm01-test:~# gfs2_fsck /dev/mapper/iscsi_cluster_qemu
>> Initializing fsck
>> Validating Resource Group index.
>> Level 1 rgrp check: Checking if all rgrp and rindex values are good.
>> (level 1 passed)
>> Okay to reclaim unlinked inodes in resource group 131090 (0x20012)?
>> (y/n)y
>> Error: resource group 131090 (0x20012): free space (65527) does not
>> match bitmap (65528)
>> (1 blocks were reclaimed)
>> Fix the rgrp free blocks count? (y/n)y
>> The rgrp was fixed.
>> RGs: Consistent: 7   Inconsistent: 1   Fixed: 1   Total: 8
>> Starting pass1
>> Pass1 complete
>> Starting pass1b
>> Pass1b complete
>> Starting pass1c
>> Pass1c complete
>> Starting pass2
>> Pass2 complete
>> Starting pass3
>> Pass3 complete
>> Starting pass4
>> Pass4 complete
>> Starting pass5
>> RG #131090 (0x20012) Inode count inconsistent: is 1 should be 0
>> Update resource group counts? (y/n) y
>> Resource group counts updated
>> Pass5 complete
>> The statfs file is wrong:
>>
>> Current statfs values:
>> blocks:  524228 (0x7ffc4)
>> free:    424937 (0x67be9)
>> dinodes: 24 (0x18)
>>
>> Calculated statfs values:
>> blocks:  524228 (0x7ffc4)
>> free:    424938 (0x67bea)
>> dinodes: 23 (0x17)
>> Okay to fix the master statfs file? (y/n)y
>> The statfs file was fixed.
>> Writing changes to disk
>> gfs2_fsck complete
>>
>>
>> root at vm01-test:~# gfs2_fsck /dev/mapper/iscsi_cluster_qemu
>> Initializing fsck
>> Validating Resource Group index.
>> Level 1 rgrp check: Checking if all rgrp and rindex values are good.
>> (level 1 passed)
>> Okay to reclaim unlinked inodes in resource group 131090 (0x20012)?
>> (y/n)y
>> Error: resource group 131090 (0x20012): free space (65527) does not
>> match bitmap (65528)
>> (1 blocks were reclaimed)
>> Fix the rgrp free blocks count? (y/n)y
>> The rgrp was fixed.
>> RGs: Consistent: 7   Inconsistent: 1   Fixed: 1   Total: 8
>> Starting pass1
>> Pass1 complete
>> Starting pass1b
>> Pass1b complete
>> Starting pass1c
>> Pass1c complete
>> Starting pass2
>> Pass2 complete
>> Starting pass3
>> Pass3 complete
>> Starting pass4
>> Pass4 complete
>> Starting pass5
>> RG #131090 (0x20012) Inode count inconsistent: is 1 should be 0
>> Update resource group counts? (y/n) y
>> Resource group counts updated
>> Pass5 complete
>> The statfs file is wrong:
>>
>> Current statfs values:
>> blocks:  524228 (0x7ffc4)
>> free:    424937 (0x67be9)
>> dinodes: 24 (0x18)
>>
>> Calculated statfs values:
>> blocks:  524228 (0x7ffc4)
>> free:    424938 (0x67bea)
>> dinodes: 23 (0x17)
>> Okay to fix the master statfs file? (y/n)y
>> The statfs file was fixed.
>> Writing changes to disk
>> gfs2_fsck complete
>>
>> Could it be that it looks like bug
>> https://bugzilla.redhat.com/show_bug.cgi?id=666080 ?
>>
>> Bart
>>
>> Bart Verwilst schreef op 23.08.2012 22:16:
>> > Hello,
>> >
>> > One problem fixed, up to the next one :) While everything seemed 
>> to
>> > work fine for a while, now I'm seeing this:
>> >
>> > root at vm02-test:~# df -h | grep libvirt
>> > /dev/mapper/iscsi_cluster_qemu     2.0G  388M  1.7G  19%
>> > /etc/libvirt/qemu
>> > /dev/mapper/iscsi_cluster_sanlock  5.0G  393M  4.7G   8%
>> > /var/lib/libvirt/sanlock
>> >
>> > root at vm02-test:~# ls -al /etc/libvirt/qemu
>> > total 16
>> > drwxr-xr-x 2 root root 3864 Aug 23 13:54 .
>> > drwxr-xr-x 6 root root 4096 Aug 14 15:09 ..
>> > -rw------- 1 root root 2566 Aug 23 13:51 firewall.xml
>> > -rw------- 1 root root 2390 Aug 23 13:54 zabbix.xml
>> >
>> > root at vm02-test:~# gfs2_tool journals /etc/libvirt/qemu
>> > journal2 - 128MB
>> > journal1 - 128MB
>> > journal0 - 128MB
>> > 3 journal(s) found.
>> >
>> >
>> > root at vm02-test:~# touch /etc/libvirt/qemu/test
>> > touch: cannot touch `/etc/libvirt/qemu/test': No space left on 
>> device
>> >
>> >
>> >
>> > Anything I can do to debug this further?
>> >
>> > Kind regards,
>> >
>> > Bart Verwilst
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster