<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-7">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
..MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EL link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span lang=EN-US>Hello all<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I have a two node cluster on 5.2 with
2.6.18-92.1.6.el5.centos.plus running for some time with drbd 8.2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I also have gfs2-utils-0.1.44-1.el5_2.1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Suddenly when trying to mount, I started
getting for my gfs2 (running on LV, over VG, over PV over DRBD):<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=: Trying to join cluster
"lock_dlm", "tweety:gfs2-00"<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0: Joined
cluster. Now mounting FS...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0: jid=0, already
locked for use<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0: jid=0: Looking
at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0: fatal:
filesystem consistency error<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0:
inode = 4 25<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0:
function = jhead_scan, file = fs/gfs2/recovery.c, line = 239<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0: about to
withdraw this file system<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>GFS2: fsid=tweety:gfs2-00.0: telling LM to
withdraw<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>dlm: closing connection to node 2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span lang=EN-US>Trying to
mount again the fs I get the same error.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Trying to gfs2_fsck –vy
/dev/mapper/vg0-data0 gives:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Initializing fsck<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Initializing lists...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Recovering journals (this may take a
while)jid=0: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=0: <b>Failed</b><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=1: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=1: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=2: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=2: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=3: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=3: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=4: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=4: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=5: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=5: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=6: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=6: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=7: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=7: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=8: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=8: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=9: Looking at journal...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>jid=9: Journal is clean.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Journal recovery complete.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Initializing special inodes...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Validating Resource Group index.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Level 1 RG check.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>(level 1 passed)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>1392 resource groups found.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Setting block ranges...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking metadata in Resource Group #0<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking metadata in Resource Group #1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking metadata in Resource Group #2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>...................<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking metadata in Resource Group #1391<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass1 complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'master'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'master' is located at
block 23 (0x17)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'root'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'root' is located at block
22 (0x16)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'inum'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'inum' is located at block
330990 (0x50cee)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'statfs'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'statfs' is located at
block 330991 (0x50cef)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'jindex'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'jindex' is located at
block 24 (0x18)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'rindex'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'rindex' is located at
block 330992 (0x50cf0)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'quota'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'quota' is located at
block 331026 (0x50d12)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system inode 'per_node'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>System inode for 'per_node' is located at
block 328392 (0x502c8)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass1b<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Looking for duplicate blocks...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>No duplicate blocks found<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass1b complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass1c<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Looking for inodes containing ea blocks...<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass1c complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system directory inode 'jindex'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system directory inode 'per_node'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system directory inode 'master'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking system directory inode 'root'<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking directory inodes.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass2 complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass3<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Marking root inode connected<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Marking master directory inode connected<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking directory linkage.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass3 complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Checking inode reference counts.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass4 complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Starting pass5<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #0<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #3<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>..............<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #1388<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #1389<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #1390<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Verifying Resource Group #1391<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Pass5 complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Writing changes to disk<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Syncing the device.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Freeing buffers.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>gfs2_fsck complete<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>My questions are:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>What it really means for GFS2 to have journal
0 locked.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>How to get out of this situation and make
the fs mountable again?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Should I try to write some garbage on
journal 0 with gfs2_edit so to force gfs2_fsck to recover it?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Thank you all for your time.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Theophanis Kontogiannis<o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>