<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div>On Mon, 2021-01-18 at 09:31 -0500, Steve Grubb wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>On Monday, January 18, 2021 8:54:30 AM EST Paul Moore wrote:</pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>I like the N of M concept but there would be a LOT of change -</pre><pre>especially</pre><pre>for all the non-kernel event sources. The EOE would be the most</pre><pre>seamless, but at a cost. My preference is to allow the 2 second 'timer'</pre><pre>to be configurable.</pre></blockquote><pre><br></pre><pre>Agree with Burn, numbering the records coming up from the kernel is</pre><pre>going to be a real nightmare, and not something to consider lightly.</pre><pre>Especially when it sounds like we don't yet have a root cause for the</pre><pre>issue.</pre></blockquote><pre><br></pre><pre>A very long time ago, we had numbered records. But it was decided that</pre><pre>there's no real point in it and we'd rather just save disk space.</pre></blockquote><pre><br></pre><pre>With the current kernel code, adding numbered records is not something to</pre><pre>take lightly.</pre></blockquote><pre><br></pre><pre>That's why I'm saying we had it and it was removed. I could imagine that if </pre><pre>you had auditing of the kill syscall enabled and a whole process group was </pre><pre>being killed, you could have hundreds of records that need numbering. No good </pre><pre>way to know in advance how many records make up the event.</pre><pre><br></pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>I know that the kernel does not serialize the events headed for user</pre><pre>space. But I'm curious how an event gets stuck and others can jump ahead</pre><pre>while one that's already inflight can get hung for 4 seconds before it's</pre><pre>next record goes out?</pre></blockquote><pre><br></pre><pre>Have you determined that the problem is the kernel? </pre></blockquote><pre><br></pre><pre>I assume so because the kernel adds the timestamp and choses what hits the </pre><pre>socket next. Auditd does no ordering of events. It just looks up the text </pre><pre>event ID, some minor translation if the enriched format is being used, and </pre><pre>writes it to disk. It can handle well over 100k records per second.</pre><pre><br></pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Initially it was looking like it was a userspace issue, is that no longer</pre><pre>the general thought?</pre></blockquote><pre><br></pre><pre>I don't see how user space could cause this. Even if auditd was slow, it </pre><pre>shouldn't take 4 seconds to write to disk and then come back to read another </pre><pre>record. And even it did, why would the newest record go out before completing </pre><pre>one that's in progress? Something in the kernel chooses what's next. I </pre><pre>suspect that might need looking at.</pre><pre><br></pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Also, is there a reliable reproducer yet?</pre></blockquote><pre><br></pre><pre>I don't know of one. But, I suppose we could modify ausearch to look for </pre><pre>examples of this.</pre></blockquote><div><br></div><div>Happy to run this where I can. I have also added the auditd.conf and audit.rules files to my github issue (<a href="https://github.com/linux-audit/audit-userspace/issues/148">https://github.com/linux-audit/audit-userspace/issues/148</a>) that makes this activity more likely to occur if that helps.</div><div><br></div><div>Also, to meet the issue of existing ausearch and the auparse library failing to process audit.log files with such issues, are we happy for a configuration item in auditd.conf?</div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre><br></pre><pre>-Steve</pre><pre><br></pre><pre><br></pre></blockquote></body></html>