[HELP] kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 9761

[Michael Bibby]

Hi, all.

I found something wrong with my server yesterday, and it cause the
database service stopped about 1 minute. I don't know what's this
exactly mean:

kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already
cleared for block 9761

and the detail message(from /var/log/messages) is pasted below. Thanks
for your help.
Best Regards.

Jul 16 15:43:52 SERVER1 clurgmgrd: [4403]: <info> Executing
/etc/init.d/db.sh status
Jul 16 15:44:20 SERVER1 remote(pam_unix)[26444]: session opened for user
sybase by (uid=0)
Jul 16 15:44:20 SERVER1 -- sybase[26444]: LOGIN ON pts/0 BY sybase FROM
192.168.88.205
Jul 16 15:44:22 SERVER1 clurgmgrd: [4403]: <info> Executing
/etc/init.d/db.sh status
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1):
ext3_free_blocks_sb: bit already cleared for block 9761
Jul 16 15:44:27 SERVER1 kernel: Aborting journal on device sda1.
Jul 16 15:44:27 SERVER1 kernel: ext3_abort called.
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1):
ext3_journal_start_sb: Detected aborted journal
Jul 16 15:44:27 SERVER1 kernel: Remounting filesystem read-only
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1) in
ext3_reserve_inode_write: Journal has aborted
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1) in
ext3_reserve_inode_write: Journal has aborted
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1) in
ext3_orphan_del: Journal has aborted
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1) in
ext3_truncate: Journal has aborted
Jul 16 15:44:27 SERVER1 kernel: __journal_remove_journal_head: freeing
b_committed_data
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1) in
start_transaction: Journal has aborted
Jul 16 15:44:27 SERVER1 remote(pam_unix)[26444]: session closed for user
sybase
Jul 16 15:44:27 SERVER1 kernel: EXT3-fs error (device sda1) in
start_transaction: Journal has aborted
Jul 16 15:44:44 SERVER1 last message repeated 327 times
Jul 16 15:44:52 SERVER1 clurgmgrd[4403]: <notice> status on fs "sybase"
returned 1 (generic error)
Jul 16 15:44:52 SERVER1 clurgmgrd[4403]: <notice> Stopping service sybase
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <info> unmounting
/opt/sybase-12.5
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <notice> Forcefully
unmounting /opt/sybase-12.5
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5387 (sybase RUN_SYBAS /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5412 (sybase RUN_SYB_B /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5413 (sybase backupser /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
17124 (sybase bash /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
9756 (sybase bash /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5388 (sybase dataserve /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 kernel: EXT3-fs error (device sda1) in
start_transaction: Journal has aborted
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5827 (sybase dataserve /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5828 (sybase dataserve /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5829 (sybase dataserve /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5830 (sybase dataserve /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
5831 (sybase dataserve /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 clurgmgrd: [4403]: <warning> killing process
9757 (sybase linux_kss /opt/sybase-12.5)
Jul 16 15:44:53 SERVER1 kernel: EXT3-fs error (device sda1) in
start_transaction: Journal has aborted
Jul 16 15:44:53 SERVER1 kernel: EXT3-fs error (device sda1) in
start_transaction: Journal has aborted
Jul 16 15:44:54 SERVER1 login(pam_unix)[17481]: session closed for user
sybase
Jul 16 15:45:04 SERVER1 clurgmgrd: [4403]: <info> unmounting
/opt/sybase-12.5
Jul 16 15:45:06 SERVER1 clurgmgrd: [4403]: <info> unmounting /backup
Jul 16 15:45:06 SERVER1 clurgmgrd: [4403]: <notice> Forcefully
unmounting /backup
Jul 16 15:45:06 SERVER1 clurgmgrd: [4403]: <warning> killing process
21668 (sybase bash /backup)
Jul 16 15:45:06 SERVER1 login(pam_unix)[21375]: session closed for user
sybase
Jul 16 15:45:16 SERVER1 clurgmgrd: [4403]: <info> unmounting /backup
Jul 16 15:45:17 SERVER1 clurgmgrd: [4403]: <info> Executing
/etc/init.d/db.sh stop
Jul 16 15:45:17 SERVER1 clurgmgrd: [4403]: <info> Removing IPv4 address
192.168.88.100 from bond0
Jul 16 15:45:27 SERVER1 clurgmgrd[4403]: <notice> Service sybase is
recovering
Jul 16 15:45:27 SERVER1 clurgmgrd[4403]: <notice> Recovering failed
service sybase
Jul 16 15:45:27 SERVER1 clurgmgrd: [4403]: <info> Adding IPv4 address
192.168.88.100 to bond0
Jul 16 15:45:28 SERVER1 clurgmgrd: [4403]: <info> mounting /dev/sda1 on
/opt/sybase-12.5
Jul 16 15:45:28 SERVER1 kernel: kjournald starting. Commit interval 5
seconds
Jul 16 15:45:28 SERVER1 kernel: EXT3-fs warning (device sda1):
ext3_clear_journal_err: Filesystem error recorded from previous mount:
IO failure
Jul 16 15:45:28 SERVER1 kernel: EXT3-fs warning (device sda1):
ext3_clear_journal_err: Marking fs in need of filesystem check.
Jul 16 15:45:28 SERVER1 kernel: EXT3-fs warning: mounting fs with
errors, running e2fsck is recommended
Jul 16 15:45:28 SERVER1 kernel: EXT3 FS on sda1, internal journal
Jul 16 15:45:28 SERVER1 kernel: EXT3-fs: recovery complete.
Jul 16 15:45:28 SERVER1 kernel: EXT3-fs: mounted filesystem with ordered
data mode.
Jul 16 15:45:29 SERVER1 clurgmgrd: [4403]: <info> mounting /dev/sdb1 on
/backup
Jul 16 15:45:29 SERVER1 kernel: kjournald starting. Commit interval 5
seconds
Jul 16 15:45:29 SERVER1 kernel: EXT3-fs warning: maximal mount count
reached, running e2fsck is recommended
Jul 16 15:45:29 SERVER1 kernel: EXT3 FS on sdb1, internal journal
Jul 16 15:45:29 SERVER1 kernel: EXT3-fs: mounted filesystem with ordered
data mode.
Jul 16 15:45:29 SERVER1 clurgmgrd: [4403]: <info> Executing
/etc/init.d/db.sh start
Jul 16 15:45:29 SERVER1 su(pam_unix)[27499]: session opened for user
sybase by (uid=0)
Jul 16 15:45:29 SERVER1 su(pam_unix)[27499]: session closed for user sybase
Jul 16 15:45:32 SERVER1 su(pam_unix)[27525]: session opened for user
sybase by (uid=0)
Jul 16 15:45:32 SERVER1 su(pam_unix)[27525]: session closed for user sybase
Jul 16 15:45:32 SERVER1 clurgmgrd[4403]: <notice> Service sybase started
Jul 16 15:45:32 SERVER1 clurgmgrd: [4403]: <info> Executing
/etc/init.d/db.sh status
Jul 16 15:46:02 SERVER1 clurgmgrd: [4403]: <info> Executing
/etc/init.d/db.sh status
Jul 16 15:46:17 SERVER1 login(pam_unix)[26873]: session opened for user
sybase by LOGIN(uid=0)