<div dir="auto">I'd just do a rescue, doesn't even need to be EL-6, and do the fsck in rw mode<div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Dec 21, 2017 7:47 AM, "francis picabia" <<a href="mailto:fpicabia@gmail.com">fpicabia@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div>Thanks for the replies...<br></div><div><br>OK, I was expecting there must be some sort of false positive going on.<br></div>For the system I listed here, those are not persistent errors.<br><br></div>However there is one which does show the same orphaned inode numbers<br></div>on each run, so this is likely real.<br><br># fsck -n /var<br>fsck from util-linux-ng 2.17.2<br>e2fsck 1.41.12 (17-May-2010)<br>Warning! /dev/sda2 is mounted.<br>Warning: skipping journal recovery because doing a read-only filesystem check.<br>/dev/sda2 contains a file system with errors, check forced.<br>Pass 1: Checking inodes, blocks, and sizes<br>Deleted inode 1059654 has zero dtime. Fix? no<br><br>Inodes that were part of a corrupted orphan linked list found. Fix? no<br><br>Inode 1061014 was part of the orphaned inode list. IGNORED.<br>Inode 1061275 was part of the orphaned inode list. IGNORED.<br>Pass 2: Checking directory structure<br>Pass 3: Checking directory connectivity<br>Pass 4: Checking reference counts<br>Pass 5: Checking group summary information<br>Block bitmap differences: -124293 -130887 -4244999 -4285460 -4979711 -4984408 -4989489 -7052754 -7052847 -7053693 -7069384 -7069539 -7069657 -7069788 -7074507 -(7095835--7095839) -7096847 -7097195 -9626336<br>Fix? no<br><br>Free blocks count wrong (6918236, counted=5214069).<br>Fix? no<br><br>Inode bitmap differences: -1059654 -1061014 -1061275<br>Fix? no<br><br>Free inodes count wrong (1966010, counted=1878618).<br>Fix? no<br><br><br>/dev/sda2: ********** WARNING: Filesystem still has errors **********<br><br>/dev/sda2: 598086/2564096 files (1.5% non-contiguous), 3321764/10240000 blocks<br><br></div>dmesg shows it had some scsi issues. I suspect the scsi error<br></div>is triggered by operation of VDP backup, which freezes the system<br></div>for a second when completing the backup snapshot.<br><br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e618c0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e61ac0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e614c0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e61cc0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e61dc0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e617c0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e616c0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e615c0<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036e613c0<br>INFO: task jbd2/sda2-8:752 blocked for more than 120 seconds.<br> Not tainted 2.6.32-696.3.2.el6.x86_64 #1<br>"echo 0 > /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this message.<br>jbd2/sda2-8 D 0000000000000000 0 752 2 0x00000000<br> ffff880037ac7c20 0000000000000046 ffff880037ac7bd0 ffffffff813a27eb<br> ffff880037ac7b80 ffffffff81014b39 ffff880037ac7bd0 ffffffff810b2a4f<br> ffff880036c44138 0000000000000000 ffff880037a69068 ffff880037ac7fd8<br>Call Trace:<br> [<ffffffff813a27eb>] ? scsi_request_fn+0xdb/0x750<br> [<ffffffff81014b39>] ? read_tsc+0x9/0x20<br> [<ffffffff810b2a4f>] ? ktime_get_ts+0xbf/0x100<br> [<ffffffff811d1400>] ? sync_buffer+0x0/0x50<br> [<ffffffff8154b0e3>] io_schedule+0x73/0xc0<br> [<ffffffff811d1440>] sync_buffer+0x40/0x50<br> [<ffffffff8154bbcf>] __wait_on_bit+0x5f/0x90<br> [<ffffffff811d1400>] ? sync_buffer+0x0/0x50<br> [<ffffffff8154bc78>] out_of_line_wait_on_bit+0x78/<wbr>0x90<br> [<ffffffff810a69b0>] ? wake_bit_function+0x0/0x50<br> [<ffffffff810a67b7>] ? bit_waitqueue+0x17/0xd0<br> [<ffffffff811d13f6>] __wait_on_buffer+0x26/0x30<br> [<ffffffffa0180146>] jbd2_journal_commit_<wbr>transaction+0xaa6/0x14f0 [jbd2]<br> [<ffffffff8108fbdb>] ? try_to_del_timer_sync+0x7b/<wbr>0xe0<br> [<ffffffffa0185a68>] kjournald2+0xb8/0x220 [jbd2]<br> [<ffffffff810a6930>] ? autoremove_wake_function+0x0/<wbr>0x40<br> [<ffffffffa01859b0>] ? kjournald2+0x0/0x220 [jbd2]<br> [<ffffffff810a649e>] kthread+0x9e/0xc0<br> [<ffffffff8100c28a>] child_rip+0xa/0x20<br> [<ffffffff810a6400>] ? kthread+0x0/0xc0<br> [<ffffffff8100c280>] ? child_rip+0x0/0x20<br>INFO: task master:1778 blocked for more than 120 seconds.<br> Not tainted 2.6.32-696.3.2.el6.x86_64 #1<br>"echo 0 > /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this message.<br>master D 0000000000000000 0 1778 1 0x00000080<br> ffff8800ba0cb948 0000000000000082 0000000000000000 ffff88000003e460<br> 00000037ffffffc8 0000004100000000 001744a7cc279bbf 0000000000000001<br> ffff8800ba0c8000 00000002863b16d4 ffff880037a55068 ffff8800ba0cbfd8<br>Call Trace:<br> [<ffffffff811d1400>] ? sync_buffer+0x0/0x50<br> [<ffffffff8154b0e3>] io_schedule+0x73/0xc0<br> [<ffffffff811d1440>] sync_buffer+0x40/0x50<br> [<ffffffff8154b99a>] __wait_on_bit_lock+0x5a/0xc0<br> [<ffffffff811d1400>] ? sync_buffer+0x0/0x50<br> [<ffffffff8154ba78>] out_of_line_wait_on_bit_lock+<wbr>0x78/0x90<br> [<ffffffff810a69b0>] ? wake_bit_function+0x0/0x50<br> [<ffffffff811d0999>] ? __find_get_block+0xa9/0x200<br> [<ffffffff811d15e6>] __lock_buffer+0x36/0x40<br> [<ffffffffa017f2bb>] do_get_write_access+0x48b/<wbr>0x520 [jbd2]<br> [<ffffffffa017f4a1>] jbd2_journal_get_write_access+<wbr>0x31/0x50 [jbd2]<br> [<ffffffffa01cd4a8>] __ext4_journal_get_write_<wbr>access+0x38/0x80 [ext4]<br> [<ffffffffa01a6d63>] ext4_reserve_inode_write+0x73/<wbr>0xa0 [ext4]<br> [<ffffffffa01a6ddc>] ext4_mark_inode_dirty+0x4c/<wbr>0x1d0 [ext4]<br> [<ffffffffa017e3d5>] ? jbd2_journal_start+0xb5/0x100 [jbd2]<br> [<ffffffffa01a70d0>] ext4_dirty_inode+0x40/0x60 [ext4]<br> [<ffffffff811c69db>] __mark_inode_dirty+0x3b/0x1c0<br> [<ffffffff811b7102>] file_update_time+0xf2/0x170<br> [<ffffffff811a4f02>] pipe_write+0x312/0x6b0<br> [<ffffffff81199c2a>] do_sync_write+0xfa/0x140<br> [<ffffffff810a6930>] ? autoremove_wake_function+0x0/<wbr>0x40<br> [<ffffffff8119f964>] ? cp_new_stat+0xe4/0x100<br> [<ffffffff81014b39>] ? read_tsc+0x9/0x20<br> [<ffffffff810b2a4f>] ? ktime_get_ts+0xbf/0x100<br> [<ffffffff8123ae06>] ? security_file_permission+0x16/<wbr>0x20<br> [<ffffffff81199f28>] vfs_write+0xb8/0x1a0<br> [<ffffffff8119b416>] ? fget_light_pos+0x16/0x50<br> [<ffffffff8119aa61>] sys_write+0x51/0xb0<br> [<ffffffff810ee4ce>] ? __audit_syscall_exit+0x25e/<wbr>0x290<br> [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b<br>INFO: task pickup:1236 blocked for more than 120 seconds.<br> Not tainted 2.6.32-696.3.2.el6.x86_64 #1<br>"echo 0 > /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this message.<br>pickup D 0000000000000001 0 1236 1778 0x00000080<br> ffff880024c6f968 0000000000000086 0000000000000000 ffffea00019e4120<br> ffff880024c6f8e8 ffffffff811456e0 001744a7cc27fe9e ffffea00019e4120<br> ffff8800117ab4a8 00000002863b1637 ffff88003738d068 ffff880024c6ffd8<br>Call Trace:<br> [<ffffffff811456e0>] ? __lru_cache_add+0x40/0x90<br> [<ffffffff811d1400>] ? sync_buffer+0x0/0x50<br> [<ffffffff8154b0e3>] io_schedule+0x73/0xc0<br> [<ffffffff811d1440>] sync_buffer+0x40/0x50<br> [<ffffffff8154b99a>] __wait_on_bit_lock+0x5a/0xc0<br> [<ffffffff811d1400>] ? sync_buffer+0x0/0x50<br> [<ffffffff8154ba78>] out_of_line_wait_on_bit_lock+<wbr>0x78/0x90<br> [<ffffffff810a69b0>] ? wake_bit_function+0x0/0x50<br> [<ffffffff811d0999>] ? __find_get_block+0xa9/0x200<br> [<ffffffff811d15e6>] __lock_buffer+0x36/0x40<br> [<ffffffffa017f2bb>] do_get_write_access+0x48b/<wbr>0x520 [jbd2]<br> [<ffffffffa017f4a1>] jbd2_journal_get_write_access+<wbr>0x31/0x50 [jbd2]<br> [<ffffffffa01cd4a8>] __ext4_journal_get_write_<wbr>access+0x38/0x80 [ext4]<br> [<ffffffffa01a6d63>] ext4_reserve_inode_write+0x73/<wbr>0xa0 [ext4]<br> [<ffffffffa01a6ddc>] ext4_mark_inode_dirty+0x4c/<wbr>0x1d0 [ext4]<br> [<ffffffffa017e3d5>] ? jbd2_journal_start+0xb5/0x100 [jbd2]<br> [<ffffffffa01a70d0>] ext4_dirty_inode+0x40/0x60 [ext4]<br> [<ffffffff811c69db>] __mark_inode_dirty+0x3b/0x1c0<br> [<ffffffff811b7315>] touch_atime+0x195/0x1a0<br> [<ffffffff811a5684>] pipe_read+0x3e4/0x4d0<br> [<ffffffff81199d6a>] do_sync_read+0xfa/0x140<br> [<ffffffff811e2e80>] ? ep_send_events_proc+0x0/0x110<br> [<ffffffff810a6930>] ? autoremove_wake_function+0x0/<wbr>0x40<br> [<ffffffff8123ae06>] ? security_file_permission+0x16/<wbr>0x20<br> [<ffffffff8119a665>] vfs_read+0xb5/0x1a0<br> [<ffffffff8119b416>] ? fget_light_pos+0x16/0x50<br> [<ffffffff8119a9b1>] sys_read+0x51/0xb0<br> [<ffffffff810ee4ce>] ? __audit_syscall_exit+0x25e/<wbr>0x290<br> [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b<br>sd 2:0:0:0: [sda] task abort on host 2, ffff880036d7a680<br>sd 2:0:0:0: [sda] Failed to get completion for aborted cmd ffff880036d7a680<br>sd 2:0:0:0: [sda] SCSI device reset on scsi2:0<br><br></div>If I just repair systems with that in their runtime history I should be on target<br> for any concerns.<br><br><div><div><div><div>Thanks for the responses...<br></div><div><div><div><br><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 21, 2017 at 9:09 AM, Cale Fairchild <span dir="ltr"><<a href="mailto:cfairchild@brocku.ca" target="_blank">cfairchild@brocku.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div lang="EN-CA"> <div class="m_-5078953942600925576gmail-m_-5467416883976973565WordSection1"> <p class="m_-5078953942600925576gmail-MsoNormal"><span lang="EN-US">Have you checked the filesystem from a rescue disk or does the fsck on reboot report that it is fixing errors each time? As far as I understand running `fsck -n /` on the active root filesystem will most always return some errors as the blocks in the filesystem are changing while the fsck is running it’s passes. Thus the warning at the beginning of the process about the filesystem being mounted. Sorry if I am misunderstanding your process, but if you have not tried checking the filesystem after booting into rescue mode that would be a good step.<u></u><u></u></span></p> <p class="m_-5078953942600925576gmail-MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <p class="m_-5078953942600925576gmail-MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> <a href="mailto:rhelv6-list-bounces@redhat.com" target="_blank">rhelv6-list-bounces@redhat.com</a> [mailto:<a href="mailto:rhelv6-list-bounces@redhat.com" target="_blank">rhelv6-list-bounces@re<wbr>dhat.com</a>] <b>On Behalf Of </b>francis picabia<br> <b>Sent:</b> December 21, 2017 07:21<br> <b>To:</b> Red Hat Enterprise Linux 6 (Santiago) discussion mailing-list <<a href="mailto:rhelv6-list@redhat.com" target="_blank">rhelv6-list@redhat.com</a>><br> <b>Subject:</b> Re: [rhelv6-list] fsck -n always showing errors<u></u><u></u></span></p><div><div class="m_-5078953942600925576gmail-h5"> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">fsck -n is used to verify only.<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal">The touch on /forcefsck will force a regular fsck on unmounted<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">partitions on boot up.<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">So what I've done is:<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal">fsck -n<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal">touch /forcefsck<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">reboot<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">times three.<u></u><u></u></p> </div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">It should be actually fixing the problems on reboot.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">I can find there are at least some fsck errors on every Redhat 6 machine,<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">whether virtual or physical. I mean I've tested the fsck -n status on about<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">twelve systems which have some errors. Only 2 showed a history<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal" style="margin-bottom:12pt">of SCSI errors, both happening to be VMware.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Maybe some other people can test this on their Redhat 6 systems<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">and see if fsck -n /var or similar comes back clean. You might<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">be surprised to see the same state I've noticed. There is<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">no issue like read-only file system. Everything is functional.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> <div> <p class="m_-5078953942600925576gmail-MsoNormal">On Wed, Dec 20, 2017 at 5:57 PM, Gianluca Cecchi <<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>> wrote:<u></u><u></u></p> <blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentcolor currentcolor currentcolor rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> <div> <p class="m_-5078953942600925576gmail-MsoNormal">On Wed, Dec 20, 2017 at 9:27 PM, francis picabia <<a href="mailto:fpicabia@gmail.com" target="_blank">fpicabia@gmail.com</a>> wrote:<u></u><u></u></p> <blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentcolor currentcolor currentcolor rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <div> <div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> <div> <p class="m_-5078953942600925576gmail-MsoNormal">With one development box I did touch /forcefsck and rebooted.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Retested fsck and still issues. Repeated this cycle 3 times<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">and no improvement.<u></u><u></u></p> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Hi,<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">not going into the reasons of the problem, but into your "cycle".<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">if I have understood correctly your sentence, you run fsck and use "-n" option that automatically answers "no" to all the questions related to problems and suggestions to fix them.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">So, as you didn't fix anything, the next run the fsck command exposes the same problems again....<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Sometimes I have seen in vSphere environments storage problems causing linux VMs problems and so kernel to automatically put one or more filesystems in read-only mode: typically the filesystems where there were writes in action during the problem occurrence.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">So in your case it could be something similar with impact to all the VMs insisting on the problematic storage / datastore<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">If you have no monitoring in place, such as Nagios and a monitor like this:<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><a href="https://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/check_ro_mounts/details" target="_blank">https://exchange.nagios.org/di<wbr>rectory/Plugins/Operating-Syst<wbr>ems/Linux/check_ro_mounts/deta<wbr>ils</a><u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">you can go ahead also some days before realizing that you had a problem<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Analyzing /var/log/messages you should see when it happened<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Take in mind that if the filesystem went in read-only mode due to a SCSI error (action taken by the kernel to prevent further errors and data corruption), you will not be able to remount it read-write, but you have to reboot the server.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Just a guess.<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">HIH,<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal">Gianluca<u></u><u></u></p> </div> <div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> </div> <p class="m_-5078953942600925576gmail-MsoNormal"><br> ______________________________<wbr>_________________<br> rhelv6-list mailing list<br> <a href="mailto:rhelv6-list@redhat.com" target="_blank">rhelv6-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/rhelv6-list" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/rhelv6-list</a><u></u><u></u></p> </blockquote> </div> <p class="m_-5078953942600925576gmail-MsoNormal"><u></u> <u></u></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div></div></div> </div> <br>______________________________<wbr>_________________<br> rhelv6-list mailing list<br> <a href="mailto:rhelv6-list@redhat.com" target="_blank">rhelv6-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/rhelv6-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/rhelv6-list</a><br></blockquote></div><br></div></div></div></div></div></div></div></div></div></div> <br>______________________________<wbr>_________________<br> rhelv6-list mailing list<br> <a href="mailto:rhelv6-list@redhat.com">rhelv6-list@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/rhelv6-list" rel="noreferrer" target="_blank">https://www.redhat.com/<wbr>mailman/listinfo/rhelv6-list</a><br></blockquote></div></div>