<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:Z="urn:schemas-microsoft-com:" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-7">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
..MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:590621770;
mso-list-type:hybrid;
mso-list-template-ids:-2131832522 67633153 67633155 67633157 67633153 67633155 67633157 67633153 67633155 67633157;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1
{mso-list-id:886992398;
mso-list-type:hybrid;
mso-list-template-ids:-498560214 67633153 67633155 67633157 67633153 67633155 67633157 67633153 67633155 67633157;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EL link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Hello again,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Becoming queries why
only once service fails, I tried to encircle the root cause.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>I ended up that files
in only one directory (were the failing service keeps its files), are corrupted.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Trying to ls –l
in the directory gives the following output:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>ls: reading directory .: Input/output error<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>total 192<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
account_boinc.bakerlab.org_rosetta.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
account_climateprediction.net.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
account_predictor.chem.lsa.umich.edu.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
all_projects_list.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>-rw-r--r-- 1 boinc boinc 159796 Jun 22 22:47
client_state_prev.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
client_state.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>-rw-r--r-- 1 boinc boinc 5141 Jun 13 23:21
get_current_version.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
get_project_config.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>-rw-r--r-- 1 boinc boinc 899 Apr 4 17:06
global_prefs.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
gui_rpc_auth.cfg<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
job_log_boinc.bakerlab.org_rosetta.txt<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
job_log_predictor.chem.lsa.umich.edu.txt<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ? lockfile<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
lookup_account.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
lookup_website.html<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
master_boinc.bakerlab.org_rosetta.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
master_climateprediction.net.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
master_predictor.chem.lsa.umich.edu.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ? projects<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
sched_reply_boinc.bakerlab.org_rosetta.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
sched_reply_climateprediction.net.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
sched_reply_predictor.chem.lsa.umich.edu.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
sched_request_boinc.bakerlab.org_rosetta.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>-rw-r--r-- 1 boinc boinc 6766 Jun 22 21:27
sched_request_climateprediction.net.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
sched_request_predictor.chem.lsa.umich.edu.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ? slots<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
statistics_boinc.bakerlab.org_rosetta.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
statistics_climateprediction.net.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
statistics_predictor.chem.lsa.umich.edu.xml<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
stderrdae.txt<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
stdoutdae.txt<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>?--------- ? ? ? ? ?
time_stats_log<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>At the same moment
the kernel reports what is following below (attached the previous e-mail).<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Trying to rm –rf
the directory fails with the same kernel message.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Any ideas on how to
erase the problematic directory?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Also the other node
(the one on which I do not try to make any actions on the file system in
question, gives the following message:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>GFS2: fsid=tweety:gfs2-00.0: jid=1: Trying to acquire
journal lock...<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US
style='color:#1F497D'>GFS2: fsid=tweety:gfs2-00.0: jid=1: Busy<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>And the file system
becomes inaccessible forever. Any one knows why is that?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>Thank you all for
your time<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'>T. Kontogiannis<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> linux-cluster-bounces@redhat.com
[mailto:linux-cluster-bounces@redhat.com] <b>On Behalf Of </b>Theophanis
Kontogiannis<br>
<b>Sent:</b> Monday, June 30, 2008 5:52 PM<br>
<b>To:</b> 'linux clustering'<br>
<b>Subject:</b> [Linux-cluster] Problem with GFS2 - Kernel Panic<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=EN-US>Hello all,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I have a two node cluster with DRBD running
in Primary/Primary.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Both nodes are running:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
lang=EN-US style='font-family:Symbol'><span style='mso-list:Ignore'>·<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span lang=EN-US>Kernel 2.6.18-92.1.6.el5.centos.plus<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
lang=EN-US style='font-family:Symbol'><span style='mso-list:Ignore'>·<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span lang=EN-US>GFS2 fsck 0.1.44<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
lang=EN-US style='font-family:Symbol'><span style='mso-list:Ignore'>·<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span lang=EN-US>cman_tool 2.0.84<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span
lang=EN-US style='font-family:Symbol'><span style='mso-list:Ignore'>·<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span lang=EN-US>Cluster LVM daemon version:
2.02.32-RHEL5 (2008-03-04)<o:p></o:p></span></p>
<p class=MsoListParagraph><span lang=EN-US>Protocol
version: 0.2.1<o:p></o:p></span></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l1 level1 lfo4'><![if !supportLists]><span
lang=EN-US style='font-family:Symbol'><span style='mso-list:Ignore'>·<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span lang=EN-US>DRBD Version: 8.2.6 (api:88)<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:18.0pt'><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoListParagraph><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>After a corruption (which was the result of
combining updating and rebooting with the FS mounted, network interruption
during the reboot and like issues, I keep on getting the following on one node:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:40 tweety1 clurgmgrd[5283]:
<notice> stop on script "BOINC" returned 1 (generic error)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:40 tweety1 clurgmgrd[5283]:
<info> Services Initialized<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:40 tweety1 clurgmgrd[5283]:
<info> State change: Local UP<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:45 tweety1 clurgmgrd[5283]:
<notice> Starting stopped service service:BOINC-t1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:45 tweety1 kernel: GFS2:
fsid=tweety:gfs2-00.0: fatal: invalid metadata block<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:45 tweety1 kernel: GFS2:
fsid=tweety:gfs2-00.0: bh = 21879736 (magic number)<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:45 tweety1 kernel: GFS2:
fsid=tweety:gfs2-00.0: function = gfs2_meta_indirect_buffer, file =
fs/gfs2/meta_io.c, line = 332<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:45 tweety1 kernel: GFS2:
fsid=tweety:gfs2-00.0: about to withdraw this file system<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:45 tweety1 kernel: GFS2:
fsid=tweety:gfs2-00.0: telling LM to withdraw<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 clurgmgrd[5283]:
<notice> Service service:BOINC-t1 started<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel: GFS2:
fsid=tweety:gfs2-00.0: withdrawn<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel: Call Trace:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff88629146>] :gfs2:gfs2_lm_withdraw+0xc1/0xd0<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff800639de>] __wait_on_bit+0x60/0x6e<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff80014eec>] sync_buffer+0x0/0x3f<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff80063a58>] out_of_line_wait_on_bit+0x6c/0x78<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff8009d1bb>] wake_bit_function+0x0/0x23<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff8863af7f>] :gfs2:gfs2_meta_check_ii+0x2c/0x38<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff8862ca06>] :gfs2:gfs2_meta_indirect_buffer+0x104/0x15e<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff8862795a>] :gfs2:gfs2_inode_refresh+0x22/0x2ca<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff8009d1bb>] wake_bit_function+0x0/0x23<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:46 tweety1 kernel:
[<ffffffff88626d9c>] :gfs2:inode_go_lock+0x29/0x57<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff88625f04>] :gfs2:glock_wait_internal+0x1d4/0x23f<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff8862611d>] :gfs2:gfs2_glock_nq+0x1ae/0x1d4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff88632053>] :gfs2:gfs2_lookup+0x58/0xa7<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff8863204b>] :gfs2:gfs2_lookup+0x50/0xa7<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff80022663>] d_alloc+0x174/0x1a9<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff8000cbb4>] do_lookup+0xd3/0x1d4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel: [<ffffffff80009f73>]
__link_path_walk+0xa01/0xf42<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff8861fd37>] :gfs2:compare_dents+0x0/0x57<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff8000e782>] link_path_walk+0x5c/0xe5<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Jun 30 00:13:47 tweety1 kernel:
[<ffffffff88624d6f>] :gfs2:gfs2_glock_put+0x26/0x133<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>After that, the machine freezes completely.
The only way to recover is to power-cycle / reset.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>“gfs2-fsck –vy
/dev/mapper/vg0-data0” ends (not terminates, it just look like it
finishes) with: <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US>Pass5 complete<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><span lang=EN-US>Writing changes
to disk<o:p></o:p></span></p>
<p class=MsoNormal style='margin-left:36.0pt'><b><span lang=EN-US>gfs2_fsck:
buffer still held for block: 21875415 (0x14dcad7)<o:p></o:p></span></b></p>
<p class=MsoNormal><b><span lang=EN-US><o:p> </o:p></span></b></p>
<p class=MsoNormal><span lang=EN-US>After remounting the file system and having
a service start (that has its files on this gfs2 filesystem), the kernel again
crasses with the same message and the node freezes up.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Unfortunately due to bad handling, I failed
to DRBD invalidate the problematic node, and instead of making it sync target
(which theoretically would solve the problem, since the good node, would sync
the bad node).<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Instead I made the bad node, sync source
and now both nodes have the same issue </span><span lang=EN-US
style='font-family:Wingdings'>L</span><span lang=EN-US><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Any ideas of how can I resolve this issue?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Sincerely,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Theophanis Kontogiannis<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
</div>
</body>
</html>