[Linux-cluster] gfs2 mount: No space left on device

[Bart Verwilst]

Umounting and remounting made the filesystem writeable again.

I've then ran a gfs2_fsck on the device, which gave me

root at vm01-test:~# gfs2_fsck /dev/mapper/iscsi_cluster_qemu
Initializing fsck
Validating Resource Group index.
Level 1 rgrp check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Okay to reclaim unlinked inodes in resource group 131090 (0x20012)? 
(y/n)y
Error: resource group 131090 (0x20012): free space (65527) does not 
match bitmap (65528)
(1 blocks were reclaimed)
Fix the rgrp free blocks count? (y/n)y
The rgrp was fixed.
RGs: Consistent: 7   Inconsistent: 1   Fixed: 1   Total: 8
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c complete
Starting pass2
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Pass4 complete
Starting pass5
RG #131090 (0x20012) Inode count inconsistent: is 1 should be 0
Update resource group counts? (y/n) y
Resource group counts updated
Pass5 complete
The statfs file is wrong:

Current statfs values:
blocks:  524228 (0x7ffc4)
free:    424937 (0x67be9)
dinodes: 24 (0x18)

Calculated statfs values:
blocks:  524228 (0x7ffc4)
free:    424938 (0x67bea)
dinodes: 23 (0x17)
Okay to fix the master statfs file? (y/n)y
The statfs file was fixed.
Writing changes to disk
gfs2_fsck complete


root at vm01-test:~# gfs2_fsck /dev/mapper/iscsi_cluster_qemu
Initializing fsck
Validating Resource Group index.
Level 1 rgrp check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Okay to reclaim unlinked inodes in resource group 131090 (0x20012)? 
(y/n)y
Error: resource group 131090 (0x20012): free space (65527) does not 
match bitmap (65528)
(1 blocks were reclaimed)
Fix the rgrp free blocks count? (y/n)y
The rgrp was fixed.
RGs: Consistent: 7   Inconsistent: 1   Fixed: 1   Total: 8
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c complete
Starting pass2
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Pass4 complete
Starting pass5
RG #131090 (0x20012) Inode count inconsistent: is 1 should be 0
Update resource group counts? (y/n) y
Resource group counts updated
Pass5 complete
The statfs file is wrong:

Current statfs values:
blocks:  524228 (0x7ffc4)
free:    424937 (0x67be9)
dinodes: 24 (0x18)

Calculated statfs values:
blocks:  524228 (0x7ffc4)
free:    424938 (0x67bea)
dinodes: 23 (0x17)
Okay to fix the master statfs file? (y/n)y
The statfs file was fixed.
Writing changes to disk
gfs2_fsck complete

Could it be that it looks like bug 
https://bugzilla.redhat.com/show_bug.cgi?id=666080 ?

Bart

Bart Verwilst schreef op 23.08.2012 22:16:
> Hello,
>
> One problem fixed, up to the next one :) While everything seemed to
> work fine for a while, now I'm seeing this:
>
> root at vm02-test:~# df -h | grep libvirt
> /dev/mapper/iscsi_cluster_qemu     2.0G  388M  1.7G  19% 
> /etc/libvirt/qemu
> /dev/mapper/iscsi_cluster_sanlock  5.0G  393M  4.7G   8%
> /var/lib/libvirt/sanlock
>
> root at vm02-test:~# ls -al /etc/libvirt/qemu
> total 16
> drwxr-xr-x 2 root root 3864 Aug 23 13:54 .
> drwxr-xr-x 6 root root 4096 Aug 14 15:09 ..
> -rw------- 1 root root 2566 Aug 23 13:51 firewall.xml
> -rw------- 1 root root 2390 Aug 23 13:54 zabbix.xml
>
> root at vm02-test:~# gfs2_tool journals /etc/libvirt/qemu
> journal2 - 128MB
> journal1 - 128MB
> journal0 - 128MB
> 3 journal(s) found.
>
>
> root at vm02-test:~# touch /etc/libvirt/qemu/test
> touch: cannot touch `/etc/libvirt/qemu/test': No space left on device
>
>
>
> Anything I can do to debug this further?
>
> Kind regards,
>
> Bart Verwilst