<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><DIV><DIV>On Apr 20, 2007, at 3:49 AM, Kadlecsik Jozsi wrote:</DIV><BLOCKQUOTE type="cite"><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I have little information about how resilient the protocol is, however,<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">in a unit we have with a bad disk, I've had the cec connection<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">spontaneously drop mid-command.<SPAN class="Apple-converted-space"> </SPAN>I'm sure they're working to fix this,<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">but it doesn't bode well for something as critical as fencing.<SPAN class="Apple-converted-space"> </SPAN>I'm also<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">unclear on whether a dropped connection generates a non-zero exit code<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">(i.e. is even detectable).</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">The fence_coraid script I wrote uses expect in perl. So if the cec<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">connection fails (at any point) it is detected and reported by the script.</DIV></BLOCKQUOTE>Also of interest is whether these masks are saved over reboot. I think they are, but its probably worth checking.</DIV><DIV><BR><BLOCKQUOTE type="cite"><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Also, on APCs, the fence_apc script has the benefit that the APC<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">switches do not allow more than one concurrent telnet connection, which<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">effectively serializes fence requests. With the cec, not so much.</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">This is problematic: the requests are not serialized at all, two<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">concurrent cec sessions are totally mixed: command issued in one cec</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">appears in the other (letter by letter). Yes, this is a real issue.</DIV></BLOCKQUOTE>In our current setup, we utilize over 25 lblades on four shelves. We are adding a fifth this weekend, with an additional six lblades. As you can imaging, fencing in this situation becomes complex. Additionally, if you desire to dynamically detect which lblades to fence, this becomes fairly complex quickly. I will leave to the reader envisioning the alternative of manually updating fencing scripts across the cluster for each lblade addition.</DIV><DIV><BR><BLOCKQUOTE type="cite"><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Also, this fences the entire Coraid device in a way that must be manually</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">cleared if it gets left masked.<SPAN class="Apple-converted-space"> </SPAN>This is a real possibility where multiple</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">nodes are racing to fence each other--especially on multiple Coraid shelfs (as</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">it must be done per shelf).</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Since we use our Coraids for non-GFS boot volumes as well, this is also</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">problematic for us, since a stale mask entry keeps us from booting.</DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">The masking disallows the access to the logical blades only. The host<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">still able to connect to the Coraid box over cec and re-enable it's<SPAN class="Apple-converted-space"> </SPAN></DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">access rights to the lblades.</DIV></BLOCKQUOTE>This certainly complicates setups that attempt to use the Coraid as a root device. I don't like the idea of having to include cec and expect in an initrd.</DIV><BR><DIV> <SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><DIV>-- </DIV><DIV>Jayson Vantuyl</DIV><DIV><FONT class="Apple-style-span" size="2"><SPAN class="Apple-style-span" style="font-size: 10px;"><SPAN class="Apple-style-span" style="font-size: 10px; "><SPAN class="Apple-style-span" style="font-size: 10px; ">Systems Architect</SPAN></SPAN></SPAN></FONT></DIV><DIV><B style="font-weight: bold; "><A href="http://www.engineyard.com"><SPAN class="Apple-style-span" style="font-weight: normal;; color: rgb(0, 0, 238); -khtml-text-decorations-in-effect: underline; "><SPAN class="Apple-style-span" style="color: rgb(0, 0, 238); -khtml-text-decorations-in-effect: underline; ">Engine Yard</SPAN></SPAN></A></B></DIV><DIV><FONT class="Apple-style-span" size="2"><SPAN class="Apple-style-span" style="font-size: 10px;"><SPAN class="Apple-style-span" style="font-size: 10px; "><SPAN class="Apple-style-span" style="font-size: 10px; "><A href="mailto:jvantuyl@engineyard.com">jvantuyl@engineyard.com</A></SPAN></SPAN></SPAN></FONT></DIV></SPAN><BR class="Apple-interchange-newline"></SPAN></SPAN> </DIV><BR></BODY></HTML>