<div dir="ltr">FYI - i talked to our network folks and it looks like they were doing some testing last night with port failover which may or may not have caused this issue. However, I was able to correct it by fencing the problem nodes.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 3, 2015 at 10:31 AM, Megan . <span dir="ltr"><<a href="mailto:nagemnna@gmail.com" target="_blank">nagemnna@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Anybody ever seen "<span style="color:rgb(41,249,20);font-family:'Andale Mono';font-size:12px;background-color:rgb(0,0,0)">Error: ClientSocket(String): connect() failed: No such file or directory"</span> when doing a start all? Something seems to have broken with our closer. Our UAT setup works as expected. I looked at tcpdumps the best that i could (i'm not a network person though) and i didn't see anything obvious. I shutdown iptables on all nodes.<div><br></div><div>We are running Centos 6,6, ccs-0.16.2-75.el6_6.1.x86_64<div>cman-3.0.12.1-68.el6.x86_64. We have a 12 node cluster in production that allows us to share gfs2 iscsi mounts. no other services are used. clvmd -R runs fine at this time. ccs -h node --sync --activate also runs fine.</div><div><blockquote type="cite" style="color:rgb(0,0,0);font-family:Calibri;font-size:medium"><div style="font-family:Calibri,sans-serif;font-size:14px"><br></div><div style="font-family:Calibri,sans-serif;font-size:14px"><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">[root@admin1 ~]# ccs -h admin1-ops --startall</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Unable to start map1-ops, possibly due to lack of quorum, try --startall</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Error: ClientSocket(String): connect() failed: No such file or directory</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started cache2-ops</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Unable to start data1-ops, possibly due to lack of quorum, try --startall</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Error: ClientSocket(String): connect() failed: No such file or directory</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started map2-ops</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Unable to start archive1-ops, possibly due to lack of quorum, try --startall</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Error: ClientSocket(String): connect() failed: No such file or directory</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started data3-ops</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started mgmt1-ops</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Unable to start admin1-ops, possibly due to lack of quorum, try --startall</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Error: ClientSocket(String): connect() failed: No such file or directory</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started data2-ops</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started cache1-ops</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">[root@admin1 ~]# </p></div><div><div style="font-family:Calibri,sans-serif;font-size:14px"><span style="font-family:'Lucida Grande'"><span style="font-size:11pt"><br></span></span></div><div><font face="Lucida Grande"><span style="font-size:15px">I</span></font><span style="font-family:'Lucida Grande'"><span style="font-size:11pt"> have quorum:</span></span></div><div><span style="font-family:'Lucida Grande'"><span style="font-size:11pt"><br></span></span></div><div><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">[root@admin1 ~]# clustat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Cluster Status for bitsops @ Wed Jun 3 02:13:08 2015</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Member Status: Quorate</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0);min-height:14px"><br></p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> Member Name ID Status</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> ------ ---- ---- ------</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> admin1-ops 1 Online, Local</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> mgmt1-ops 2 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> archive1-ops 3 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> map1-ops 4 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> map2-ops 5 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> cache1-ops 6 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> cache2-ops 7 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> data1-ops 8 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> data2-ops 9 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"> data3-ops 10 Online</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0);min-height:14px"><br></p></div><div style="font-family:Calibri,sans-serif;font-size:14px"><span style="font-family:'Lucida Grande'"><span style="font-size:11pt"><br></span></span></div><div style="font-family:Calibri,sans-serif;font-size:14px"><span style="font-family:'Lucida Grande'"><span style="font-size:11pt"><br></span></span></div><div><font face="Lucida Grande" size="3">Here is what </font><font face="Lucida Grande"><span style="font-size:15px">I</span></font><font face="Lucida Grande" size="3"> expect, and what UAT gives me:</font></div><div><font face="Lucida Grande" size="3"><br></font></div><div><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">[root@admin1-uat ~]# ccs -h admin1-uat --startall</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started mgmt1-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started data1-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started data2-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started admin1-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started tools-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started map1-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started archive1-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started cache2-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started cache1-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">Started map2-uat</p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">[root@admin1-uat ~]# </p><p style="margin:0px;font-size:12px;font-family:'Andale Mono';color:rgb(41,249,20);background-color:rgb(0,0,0);min-height:14px"><br></p></div></div></blockquote></div></div></div>
</blockquote></div><br></div>