[linux-lvm] directory # contains a hole
Kearnan_Keith at emc.com
Kearnan_Keith at emc.com
Mon Dec 11 21:35:21 UTC 2006
Hi Matt,
Please see bugzilla 213921. We are working on this one.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=213921
Keith R. Kearnan
Principal Engineer
E-Lab
EMC² Engineering
Phone: (508) 249-1315
Kearnan_Keith at emc.com
-----Original Message-----
From: linux-lvm-bounces at redhat.com [mailto:linux-lvm-bounces at redhat.com] On Behalf Of Matt P
Sent: Monday, December 11, 2006 4:24 PM
To: linux-lvm at redhat.com
Subject: [linux-lvm] directory # contains a hole
I'm not certain this is LVM related but at this point I'm grasping at
straws. Every so often, ranging from 14 days to a little over a month,
the System will through the following error to messages:
kernel: EXT3-fs error (device dm-6): ext3_readdir: directory #30933792
contains a hole at offset 28672
After that message shows up we get numerous "Journal Aborted" errors
(messages clip at the bottom). The File system is marked Read-Only and
that in turn initiates a fail-over of the application to the standby
node. Then on the standby node, the filesystem is fsck'ed and is
easily cleaned up without requiring any special intervention.
Out SAN admin isn't seeing any errors on his end, and I don't see
anything that jumps out at me in out messages. Every time this has
happened the inode mentioned in the above error is different. We have
rebuilt the VG and the subsequent filesystem from scratch once. we've
turned the firmware parameters on the QLogic card to match the
recommendations for EMC for our HW config and OS version. The failure
occurs on both nodes (primary and standby) in the cluster. We've got a
ticket open with EMC and another open with RedHat.
Any help and/or ideas would be much appreciated....
Here's the system config:
Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
Kernel: 2.6.9-22.ELsmp
Qlogic HBA's Connected to an EMC Clariion
Using QLogic's drivers and EMC Powerpath
About 350Gig presented to the system
LVM version: 2.01.14 (2005-08-04)
Library version: 1.01.04 (2005-08-02)
Driver version: 4.4.0
-------------------------------------------------------------------------------------------
# vgdisplay -v vg01
Using volume group(s) on command line
Finding volume group "vg01"
--- Volume group ---
VG Name vg01
System ID xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Format lvm2
Metadata Areas 7
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 256
Cur LV 1
Open LV 1
Max PV 256
Cur PV 7
Act PV 7
VG Size 349.56 GB
PE Size 32.00 MB
Total PE 11186
Alloc PE / Size 11186 / 349.56 GB
Free PE / Size 0 / 0
VG UUID 74biJE-Jn9V-i7jD-4Mng-2Pje-01YH-piE51u
--- Logical volume ---
LV Name /dev/vg01/data
VG Name vg01
LV UUID 000000-0000-0000-0000-0000-0000-000000
LV Write Access read/write
LV Status available
# open 1
LV Size 349.56 GB
Current LE 11186
Segments 7
Allocation normal
Read ahead sectors 1024
Block device 253:6
--- Physical volumes ---
PV Name /dev/emcpowera1
PV UUID DcRtrM-NtY6-ispA-uPwD-sJO1-SF0A-7iZlmm
PV Status allocatable
Total PE / Free PE 1598 / 0
PV Name /dev/emcpowerb1
PV UUID VIrhHw-LqEt-KNyZ-W86T-3oyY-9F25-Zu7zMA
PV Status allocatable
Total PE / Free PE 1598 / 0
PV Name /dev/emcpowerc1
PV UUID ySMNRg-m3XQ-ifF0-fWZx-ztAe-lhOX-seqJ9G
PV Status allocatable
Total PE / Free PE 1598 / 0
PV Name /dev/emcpowerd1
PV UUID puTa3Z-ZRrv-mRRS-orL0-Ndpz-dW4T-E1x46s
PV Status allocatable
Total PE / Free PE 1598 / 0
PV Name /dev/emcpowere1
PV UUID tsmFWq-YSVA-gHbI-ZGKG-4V57-766T-G8R6R2
PV Status allocatable
Total PE / Free PE 1598 / 0
PV Name /dev/emcpowerf1
PV UUID WipA0D-GQXE-fDWL-OPNK-tRbx-7BUO-bAAJHY
PV Status allocatable
Total PE / Free PE 1598 / 0
PV Name /dev/emcpowerg1
PV UUID jbcB7h-qh8O-LkAB-wcJz-CxKy-GIIm-xwylk2
PV Status allocatable
Total PE / Free PE 1598 / 0
-------------------------------------------------------------------------------------------
Messages Clip Below:
-------------------------------------------------------------------------------------------
Dec 8 17:50:01 localhost logger: 17:50:01 up 17 days, 4:17, 2 users,
load average: 3.07, 4.30, 4.54
Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6):
ext3_readdir: directory #30933792 contains a hole at offset 28672
Dec 8 17:50:04 localhost kernel: Aborting journal on device dm-6.
Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Readonly filesystem
Dec 8 17:50:04 localhost kernel: Aborting journal on device dm-6.
Dec 8 17:50:04 localhost kernel: ext3_abort called.
Dec 8 17:50:04 localhost kernel: ext3_abort called.
Dec 8 17:50:04 localhost kernel: ext3_abort called.
Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6):
ext3_journal_start_sb: Detected aborted journal
Dec 8 17:50:04 localhost kernel: Remounting filesystem read-only
Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6):
ext3_journal_start_sb: <2>EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6):
ext3_journal_start_sb: Detected aborted journalDetected aborted
journal
Dec 8 17:50:05 localhost kernel:
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost last message repeated 2 times
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost last message repeated 5 times
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_reserve_inode_write: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost last message repeated 7 times
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost last message repeated 2 times
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_reserve_inode_write: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in
ext3_dirty_inode: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 11 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 7 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_reserve_inode_write: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_dirty_inode: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 5 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 6 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
add_dirent_to_buf: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 37 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 44 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 4 times
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_inode: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
ext3_create: Journal has aborted
Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:06 localhost last message repeated 3 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 8 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_ordered_commit_write: IO failure
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 3 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 16 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
add_dirent_to_buf: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_rename: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 6 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_new_block: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
ext3_prepare_write: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 7 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 3 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal hasEXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 3 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:07 localhost last message repeated 3 times
Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost last message repeated 2 times
Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost last message repeated 2 times
Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost last message repeated 2 times
Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:08 localhost last message repeated 33 times
Dec 8 17:50:08 localhost kernel: EXT3-fs freeing b_committed_data
Dec 8 17:50:08 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec 8 17:50:09 localhost last message repeated 26 times
Dec 8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:09 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec 8 17:50:09 localhost last message repeated 4 times
Dec 8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:09 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec 8 17:50:09 localhost last message repeated 34 times
Dec 8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:10 localhost last message repeated 1779 times
Dec 8 17:50:11 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:25 localhost last message repeated 51 times
Dec 8 17:50:25 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:26 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:27 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:36 localhost last message repeated 11 times
Dec 8 17:50:38 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:50:56 localhost last message repeated 16 times
Dec 8 17:51:01 localhost logger: 17:51:01 up 17 days, 4:18, 2 users,
load average: 3.81, 4.62, 4.65
Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in
start_transaction: Journal has aborted
Dec 8 17:51:17 localhost kernel: __journal_remove_journal_head:
freeing b_committed_data
Dec 8 17:51:17 localhost last message repeated 8 times
Dec 8 17:52:01 localhost logger: 17:52:01 up 17 days, 4:19, 0 users,
load average: 1.77, 3.92, 4.41
_______________________________________________
linux-lvm mailing list
linux-lvm at redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
More information about the linux-lvm
mailing list