<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)"> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"/> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"/>  <style>  </style>  </head> <body lang=IT link=blue vlink=purple> <div class=Section1> <o:p> </o:p> <o:p> </o:p> Hi,<o:p></o:p> i’m reached a strange situation over my servers SUN X4200M2 running with Linux Advanced Platform 5.1 Linux fea.localdomain 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux.. This happen on both internal and external disks (Hitachi AMS 200 storage , emulex HBA , and HDLM sw <st1:City w:st="on"><st1:place w:st="on">Hitachi</st1:place></st1:City> for multipath)<o:p></o:p> <o:p> </o:p> After problem happening I’m not able to use the server due to root corruption files :<o:p></o:p> -rwxr-xr-x 1 root root 14096 Sep 5 2007 rmmod<o:p></o:p> -rwxr-xr-x 1 root root 521552 Aug 7 2006 rmt<o:p></o:p> -rwxr-xr-x 1 root root 14648 Jul 13 2006 rngd<o:p></o:p> -rwxr-xr-x 1 root root 57920 Aug 7 2006 route<o:p></o:p> -rwxr-xr-x 1 root root 5904 Sep 25 2007 rpc.lockd<o:p></o:p> -rwxr-xr-x 1 root root 49352 Sep 25 2007 rpc.statd<o:p></o:p> ?--------- ? ? ? ? ? rrestore<o:p></o:p> ?--------- ? ? ? ? ? rrestore.static<o:p></o:p> -rwxr-xr-x 1 root root 29976 Jan 9 2007 rtmon<o:p></o:p> -rwxr-xr-x 1 root root 7736 Oct 13 2006 runlevel<o:p></o:p> -rwxr-xr-x 1 root root 30840 Nov 27 2006 runuser<o:p></o:p> -rwxr-xr-x 1 root root 10376 Aug 17 2007 salsa<o:p></o:p> [root@fea sbin]#<o:p></o:p> <o:p> </o:p> And also file system are mounted in read-only mode<o:p></o:p> <o:p> </o:p> The following is a parts of messages file:<o:p></o:p> Jul 11 16:11:15 fea clurgmgrd[4739]: <notice> Service service:appl-dfdd is disabled<o:p></o:p> Jul 11 16:29:56 fea clurgmgrd[4739]: <notice> Stopping service service:db-dfdd<o:p></o:p> Jul 11 16:30:00 fea avahi-daemon[4622]: Withdrawing address record for 10.40.3.40 on eth1.<o:p></o:p> Jul 11 16:30:11 fea dlm_controld[4281]: uevent message has 3 args<o:p></o:p> Jul 11 16:30:11 fea clurgmgrd[4739]: <notice> Service service:db-dfdd is disabled<o:p></o:p> Jul 11 16:31:44 fea clurgmgrd[4739]: <notice> Starting disabled service service:db-dfdd<o:p></o:p> Jul 11 16:31:44 fea kernel: kjournald starting. Commit interval 5 seconds<o:p></o:p> Jul 11 16:31:44 fea kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended<o:p></o:p> Jul 11 16:31:44 fea kernel: EXT3 FS on sddlmab, internal journal<o:p></o:p> Jul 11 16:31:44 fea kernel: EXT3-fs: mounted filesystem with ordered data mode.<o:p></o:p> Jul 11 16:31:44 fea dlm_controld[4281]: uevent message has 3 args<o:p></o:p> Jul 11 16:31:44 fea avahi-daemon[4622]: Registering new address record for 10.40.3.40 on eth1.<o:p></o:p> Jul 11 16:31:48 fea clurgmgrd[4739]: <notice> Service service:db-dfdd started<o:p></o:p> Jul 11 16:40:23 fea clurgmgrd[4739]: <notice> Stopping service service:db-dfdd<o:p></o:p> Jul 11 16:40:25 fea avahi-daemon[4622]: Withdrawing address record for 10.40.3.40 on eth1.<o:p></o:p> Jul 11 16:40:35 fea dlm_controld[4281]: uevent message has 3 args<o:p></o:p> Jul 11 16:40:35 fea clurgmgrd[4739]: <notice> Service service:db-dfdd is disabled<o:p></o:p> Jul 11 17:13:01 fea kernel: EXT3-fs error (device dm-0): ext3_free_blocks_sb: bit already cleared for block 382976<o:p></o:p> Jul 11 17:13:01 fea kernel: Aborting journal on device dm-0.<o:p></o:p> Jul 11 17:13:01 fea kernel: EXT3-fs error (device dm-0): ext3_free_blocks_sb: bit already cleared for block 382977<o:p></o:p> Jul 11 17:13:01 fea kernel: EXT3-fs error (device dm-0): ext3_free_blocks_sb: bit already cleared for block 382978<o:p></o:p> Jul 11 17:13:01 fea kernel: EXT3-fs error (device dm-0): ext3_free_blocks_sb: bit already cleared for block 382979<o:p></o:p> Jul 11 17:13:01 fea kernel: EXT3-fs error (device dm-0): ext3_free_blocks_sb: bit already cleared for block 382980<o:p></o:p> Jul 11 17:13:02 fea kernel: EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted<o:p></o:p> Jul 11 17:13:02 fea kernel: EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted<o:p></o:p> Jul 11 17:13:02 fea kernel: EXT3-fs error (device dm-0) in ext3_orphan_del: Journal has aborted<o:p></o:p> Jul 11 17:13:02 fea kernel: EXT3-fs error (device dm-0) in ext3_truncate: Journal has aborted<o:p></o:p> Jul 11 17:13:02 fea kernel: ext3_abort called.<o:p></o:p> Jul 11 17:13:02 fea kernel: EXT3-fs error (device dm-0): ext3_journal_start_sb: Detected aborted journal<o:p></o:p> Jul 11 17:13:02 fea kernel: Remounting filesystem read-only<o:p></o:p> Jul 11 17:27:30 fea clurgmgrd[4739]: <info> State change: feb.iride DOWN<o:p></o:p> Jul 11 17:27:30 fea clurgmgrd[4739]: <info> State change: /dev/sddlmac UP<o:p></o:p> Jul 11 17:27:30 fea clurgmgrd[4739]: <info> Waiting for node #2 to be fenced<o:p></o:p> Jul 11 17:28:50 fea qdiskd[4191]: <info> Node 2 shutdown<o:p></o:p> <o:p> </o:p> And also a kernel bug as:<o:p></o:p> Jul 9 16:57:13 fea syslogd 1.4.1: restart.<o:p></o:p> /trace<o:p></o:p> Jul 10 17:41:09 fea kernel: EXT3-fs warning (device sddlmaa): ext3_unlink: Deleting nonexistent file (13353077), 0<o:p></o:p> Jul 10 18:20:04 fea dlm_controld[4260]: uevent message has 3 args<o:p></o:p> Jul 10 18:20:04 fea kernel: sb orphan head is 13353077<o:p></o:p> Jul 10 18:20:04 fea kernel: sb_info orphan list:<o:p></o:p> Jul 10 18:20:04 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 59479 times<o:p></o:p> Jul 10 18:20:13 fea kernel: BUG: soft lockup detected on CPU#1!<o:p></o:p> Jul 10 18:20:13 fea kernel:<o:p></o:p> Jul 10 18:20:13 fea kernel: Call Trace:<o:p></o:p> Jul 10 18:20:13 fea kernel: <IRQ> [<ffffffff800b50fa>] softlockup_tick+0xd5/0xe7<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800930e2>] update_process_times+0x42/0x68<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800746e3>] smp_local_timer_interrupt+0x23/0x47<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff80074da5>] smp_apic_timer_interrupt+0x41/0x47<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff8005bc8e>] apic_timer_interrupt+0x66/0x6c<o:p></o:p> Jul 10 18:20:13 fea kernel: <EOI> [<ffffffff8008d4b6>] vprintk+0x29e/0x2ea<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff8008d554>] printk+0x52/0xbd<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff80061a3f>] out_of_line_wait_on_bit+0x6c/0x78<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff880564f4>] :ext3:ext3_put_super+0x13e/0x1e0<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800d8e1e>] generic_shutdown_super+0x79/0xfb<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800d8ec6>] kill_block_super+0x26/0x3a<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800d8f94>] deactivate_super+0x6a/0x82<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800e1d13>] sys_umount+0x245/0x27b<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff800b27ae>] audit_syscall_entry+0x14d/0x180<o:p></o:p> Jul 10 18:20:13 fea kernel: [<ffffffff8005b28d>] tracesys+0xd5/0xe0<o:p></o:p> Jul 10 18:20:13 fea kernel:<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 50 times<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink , nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 54 times<o:p></o:p> Jul 10 18:20:13 fea kernel: in, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 54 times<o:p></o:p> Jul 10 18:20:13 fea kernel: in, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 54 times<o:p></o:p> Jul 10 18:20:13 fea kernel: in, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 54 times<o:p></o:p> Jul 10 18:20:13 fea kernel: in, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea kernel: inode dm-0:1010899 at ffff8100df1f3448: mode 100555, nlink 1, next 0<o:p></o:p> Jul 10 18:20:13 fea last message repeated 54 times<o:p></o:p> <o:p> </o:p> I’m planning to reinstall the server …<o:p></o:p> <o:p> </o:p> Some body can help me ?<o:p></o:p> Thanks a lot<o:p></o:p> Fabio<o:p></o:p> </div> <DIV><HR> -------------------------------------------- INFORMATIVA SULLA PRIVACY Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni contenute in questo messaggio e nei suoi eventuali allegati sono riservate e per uso esclusivo del destinatario. Nessuno, all'infuori dello stesso, può copiare o distribuire il messaggio, o parte di esso, a terzi. Chiunque riceva questo messaggio per errore è pregato di distruggerlo e di informare il mittente. PRIVACY NOTICE According to the D.Lgs. 196/2003 this document and its attachments are confidential and intended for the named addressee(s) only. If you are not the intended recipient of this message, any use or dissemination of this message is prohibited. If you have received this document by mistake, please notify the sender and destroy all physical and/or electronic copies. </DIV> <DIV><HR> -------------------------------------------- INFORMATIVA SULLA PRIVACY Ai sensi del D.Lgs. 196/2003 si precisa che le informazioni contenute in questo messaggio e nei suoi eventuali allegati sono riservate e per uso esclusivo del destinatario. Nessuno, all'infuori dello stesso, può copiare o distribuire il messaggio, o parte di esso, a terzi. Chiunque riceva questo messaggio per errore è pregato di distruggerlo e di informare il mittente. PRIVACY NOTICE According to the D.Lgs. 196/2003 this document and its attachments are confidential and intended for the named addressee(s) only. If you are not the intended recipient of this message, any use or dissemination of this message is prohibited. If you have received this document by mistake, please notify the sender and destroy all physical and/or electronic copies. </DIV> </body> </html>