Filesystem remounted read-only WAS: RE: Recent unexplained quota problems

[Chris St. Pierre]

On Wed, 12 Sep 2007, Mertens, Bram wrote:
> As a matter of fact I just noticed very similar behavior on one of our
> servers just now.  /var was remounted read-only while mount indicated
> nothing special.
>
> Dmesg however showed the following errors:
[...snip...]
> Info fld=0x0, Current sdb: sense key Aborted Command
[...snip...]
> EXT3-fs error (device dm-5) in start_transaction: Journal has aborted
>
> This is on a virtual server (VMWare ESX) with the datafiles on a SAN, so
> it's likely to have been an issue with the SAN-connection.
>
> I was unable to unmount /var even after going to runlevel 1.  After a
> reboot everything appeared to be fine, just like in th OP's message. But
> I rebooted to runlevel 1 and unmounted /var to run a file system check.
>
> E2fsck -p /dev/logvg/var indicates the filesystem is clean, as does
> e2fsck -b 32768 -p /dev/logvg/var.
>
> Can we trust this filesystem?

I'm not sure.  The "sense key Aborted Command" error means that the
target aborted the write; I don't think that's the error you'd see if
you simply dropped connectivity.  OTOH, the fact that the fs is clean
suggests that it was in a consistent state when the write was aborted,
so it's not a problem that's caused any data corruption.

It could still be due to a issue on the SAN -- perhaps some dying
media? -- so I'd see if you can get any logs off your SAN; if you can
find proof of a dropped connection, then you're good, but you might
want to look into it further.

Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University