<div dir="ltr">We saw the following happen on a system today, and wanted to follow up:<div><br></div><div>System became unresponsive to ssh logins with the error:</div><div><br></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">ssh -v incentives01</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Reading configuration data /etc/ssh/ssh_config</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: /etc/ssh/ssh_config line 4: Applying options for *</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Connecting to incentives01 [172.31.9.16] port 22.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Connection established.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_rsa-cert type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_dsa-cert type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_ecdsa-cert type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519 type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: identity file /home/jeff.hallyburton/.ssh/id_ed25519-cert type -1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Enabling compatibility mode for protocol 2.0</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Local version string SSH-2.0-OpenSSH_6.6.1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: SSH2_MSG_KEXINIT sent</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: SSH2_MSG_KEXINIT received</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: kex: server->client aes128-ctr <a href="mailto:hmac-md5-etm@openssh.com">hmac-md5-etm@openssh.com</a> none</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: kex: client->server aes128-ctr <a href="mailto:hmac-md5-etm@openssh.com">hmac-md5-etm@openssh.com</a> none</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: kex: <a href="mailto:curve25519-sha256@libssh.org">curve25519-sha256@libssh.org</a> need=16 dh_need=16</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: kex: <a href="mailto:curve25519-sha256@libssh.org">curve25519-sha256@libssh.org</a> need=16 dh_need=16</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: sending SSH2_MSG_KEX_ECDH_INIT</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: expecting SSH2_MSG_KEX_ECDH_REPLY</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Server host key: ECDSA 89:e0:f8:25:21:db:c9:46:67:14:38:0c:c1:f4:f7:51</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Host 'incentives01' is known and matches the ECDSA host key.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Found key in /home/jeff.hallyburton/.ssh/known_hosts:7</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: ssh_ecdsa_verify: signature correct</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: SSH2_MSG_NEWKEYS sent</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: expecting SSH2_MSG_NEWKEYS</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: SSH2_MSG_NEWKEYS received</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: SSH2_MSG_SERVICE_REQUEST sent</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: SSH2_MSG_SERVICE_ACCEPT received</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">This is a private computer system which is restricted to authorized individuals.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0);min-height:14px"><br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Actual or attempted unauthorized use of this computer system will result in</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">criminal and/or civil prosecution.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0);min-height:14px"><br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">We reserve the right to view, monitor and record activity on the system without</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">notice or permission. Any information obtained by monitoring, reviewing or</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">recording is subject to review by law enforcement organizations in connection</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">with the investigation or prosecution of possible criminal activity on this system.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0);min-height:14px"><br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">If you are not an authorized user of this system or do not consent to continued</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">monitoring, disconnect at this time.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Next authentication method: gssapi-keyex</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: No valid Key exchange context</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Next authentication method: gssapi-with-mic</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Received disconnect from <a href="http://172.31.9.16">172.31.9.16</a>: 2: Too many authentication failures for jeff.hallyburton</p><div><div class="gmail_signature"><div dir="ltr"><span style="font-family:Arial,sans-serif"><br></span></div><div><span style="font-family:Arial,sans-serif">Ultimately we rebooted the node to restore connectivity.  After we were back in, we're seeing that sssd crashed due what looks like a memory allocation error:</span></div><div><span style="font-family:Arial,sans-serif"><br></span></div><div><span style="font-family:Arial,sans-serif">/var/log/sssd/sssd.log</span></div><div><span style="font-family:Arial,sans-serif"><br></span></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)"># cat /var/log/sssd/sssd.log </p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">(Thu Jan 28 20:15:56 2016) [sssd] [mt_svc_sigkill] (0x0010): [<a href="http://enervee.com">enervee.com</a>][620] is not responding to SIGTERM. Sending SIGKILL.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">(Thu Jan 28 20:16:27 2016) [sssd] [talloc_log_fn] (0x0010): talloc: access after free error - first free may be at src/monitor/monitor.c:2760</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0);min-height:14px"><br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">(Thu Jan 28 20:16:27 2016) [sssd] [talloc_log_fn] (0x0010): Bad talloc magic value - access after free</p></div><div><br></div><div>/var/log/secure</div><div><br></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:05:48 incentives01 sshd[26145]: Timeout, client not responding.<br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:05:48 incentives01 sshd[26142]: pam_unix(sshd:session): session closed for user </p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:16:28 incentives01 sshd[14504]: Timeout, client not responding.</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:16:28 incentives01 sshd[14501]: pam_systemd(sshd:session): Failed to release session: Connection timed out</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:16:28 incentives01 sshd[14501]: pam_unix(sshd:session): session closed for user </p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:16:28 incentives01 sshd[14501]: pam_sss(sshd:session): Request to sssd failed. Bad address</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:16:29 incentives01 sshd[14501]: fatal: login_init_entry: Cannot find user</p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'Andale Mono';color:rgb(249,254,236);background-color:rgb(0,0,0)">Jan 28 20:21:40 incentives01 sshd[26882]: Invalid user from 172.31.8.34</p></div><div><br></div><div>The system may have simply run out of ram, but wanted to check to see if there were any known or contributing issues.</div><div><br></div><div>Thanks,</div><div><br></div><div>Jeff</div><div><br></div><div dir="ltr"><span style="font-family:Arial,sans-serif">Jeff Hallyburton</span><span style="font-size:10pt;font-family:Arial,sans-serif"><br></span><span style="font-size:10pt;font-family:Arial,sans-serif">Strategic Systems Engineer<br><span style="">Bloomip Inc.</span></span><span><span style="font-size:10pt;font-family:Arial,sans-serif"><br><span style="">Web: </span></span><a href="http://www.bloomip.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif">http://www.bloomip.com</span></a><span style="font-size:10pt;font-family:Arial,sans-serif"><br><br><span style="">Engineering Support: </span></span><a href="mailto:support@bloomip.com" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif">support@bloomip.com</span></a><span style="font-size:10pt;font-family:Arial,sans-serif"><br><span style="">Billing Support: </span></span><a href="mailto:billing@bloomip.com" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif">billing@bloomip.com</span></a><span style="font-size:10pt;font-family:Arial,sans-serif"><br><span style="">Customer Support Portal:  </span></span><a href="http://my.bloomip.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-size:10pt;font-family:Arial,sans-serif">https://my.bloomip.com</span></a></span><span style="font-size:10pt;font-family:Arial,sans-serif"><br></span></div></div></div>
</div></div>