<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-GB link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hello All,</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am investigating strange behaviour described here</p><p class=MsoNormal>https://github.com/coreos/bugs/issues/2357 and would like to ask for</p><p class=MsoNormal>help/advice to diagnose it further.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>It all boils down to kworker spinning 100% of CPU per iSCSI portal,</p><p class=MsoNormal>when logging out from iSCSI sessions. It happens only if multipathd is</p><p class=MsoNormal>running or had run and created multipath devices. No problem occur if</p><p class=MsoNormal>I mask multipathd, reboot, wait for all scsi devices to be discovered</p><p class=MsoNormal>and attempt to logout.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> 1. start server, login to all iscsi portals, no need to mount anything</p><p class=MsoNormal> 2. logout from all or one session. logging out from one sometimes</p><p class=MsoNormal>work, but more often does not. Logging out from all simultaneously</p><p class=MsoNormal>triggers problem 100% of the time.</p><p class=MsoNormal> 3. every iSCSI session we had befor makes kworker to spin. In my</p><p class=MsoNormal>case there are 4 portals to NetApp. `l > /proc/sysrq-trigger` shows</p><p class=MsoNormal>following stack traces per kworker spinning:</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>[19806.968333] Call Trace:</p><p class=MsoNormal>[19806.968347] scsi_remove_device+0x19/0x60 [scsi_mod]</p><p class=MsoNormal>[19806.968354] scsi_remove_target+0x167/0x1b0 [scsi_mod]</p><p class=MsoNormal>[19806.968355] iscsi_free_session+0x383/0x430 [scsi_transport_iscsi]</p><p class=MsoNormal>[19806.968366] process_one_work+0x144/0x350</p><p class=MsoNormal>[19806.968367] worker_thread+0x4d/0x3e0</p><p class=MsoNormal>[19806.968369] kthread+0xfc/0x130</p><p class=MsoNormal>[19806.968370] ? rescuer_thread+0x310/0x310</p><p class=MsoNormal>[19806.968371] ? kthread_park+0x60/0x60</p><p class=MsoNormal>[19806.968372] ? do_syscall_64+0xe9/0x1c0</p><p class=MsoNormal>[19806.968374] ret_from_fork+0x35/0x40</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> 4. if I do `multipath -F; for d in $(iscsiadm -m session -P3 |awk</p><p class=MsoNormal>'/scsi disk/ {print $4}'); do echo 1 > /sys/block/$d/device/delete;</p><p class=MsoNormal>done` before logging out, it works just fine</p><p class=MsoNormal> 5. it is reproducible 100% of the time, so I you want me to collect</p><p class=MsoNormal>some data, I can run commands for you :)</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any help is greatly appreciated.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Software versions:</p><p class=MsoNormal>Kernel version is 4.14.16-coreos (CoreOS 1632.2.1)</p><p class=MsoNormal>iscsid version 2.0-873</p><p class=MsoNormal>multipath-tools 0.6.4</p><p class=MsoNormal>no iSCSI offloading</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Quick check shows that problem occurred between 4.13.16 and 4.14.0-rc8</p></div></body></html>