Systemd Journald and audit logging causing journal issues

[Brad Zynda]

Hey Steve,

Just wanted to follow up on this and say we are still seeing services
across the board that have:

Warning: Journal has been rotated since unit was started. Log output is
incomplete or unavailable

basically created a script to check all unit file services/targets and
grep status -l for Journal, it is effecting a large range of
service.target's, service.service's and service.socket's

If we restart the service or reboot we no longer see those messages
about journal and everything appears to run as expected.

Thanks,
Brad

On 10/19/2017 04:13 PM, Steve Grubb wrote:
> On Thursday, October 19, 2017 1:08:22 PM EDT Brad Zynda wrote:
>>>> grep perm_mod /etc/audit/audit.rules
>>>> -a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=1000
>>>> -F auid!=4294967295 -k perm_mod
>>>> -a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F auid>=1000
>>>> -F auid!=4294967295 -k perm_mod
>>>> -a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F
>>>> auid>=1000 -F auid!=4294967295 -k perm_mod
>>>> -a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F
>>>> auid>=1000 -F auid!=4294967295 -k perm_mod
>>>> -a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S
>>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
>>>> auid!=4294967295 -k perm_mod
>>>> -a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S
>>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
>>>> auid!=4294967295 -k perm_mod
>>>>
>>>> grep delete /etc/audit/audit.rules
>>>> -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat
>>>> -F auid>=1000 -F auid!=4294967295 -k delete
>>>> -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat
>>>> -F auid>=1000 -F auid!=4294967295 -k delete
>>>> -a always,exit -F arch=b64 -S init_module -S delete_module -k modules
>>>> -a always,exit -F arch=b32 -S init_module -S delete_module -k modules
>>>
>>> These rules are well formed. Meaning no obvious holes that would cause
>>> unintended events. The other ausearch/aureport commands I gave you should
>>> show you what is causing the events and to which files. This information
>>> may be used to create some kind of "never" rule that limits what gets
>>> audited. If you do create some exclusion rule, it needs to be above the
>>> perm_mod rules because audit is a "first match wins" system.
>>>
>>> -Steve
>>>
>>> -Steve
>>
>> Here is a peak report (at least in the last 24 hours) didnt include the
>> file summaries as that would make this email HUGE:
> 
> Well, the idea is to look for something that's getting hit a lot. What it 
> sounds like is that things are getting deleted and modified quite a bit all 
> over the place. Does the executable report show a pattern such as one 
> application doing a lot? For example, with the rule you have, doing a yum 
> update will delete a whole lot of stuff and modify a whole lot of stuff.
> 
> Its possible to put exceptions in the rules so that one program does not flood 
> the logs. But looking at the quantities below, the audit system can easily 
> handle that.
> 
> Its also possible to exclude directories from auditing if the pattern is that 
> you have a daemon receiving and modifying files and then deleting them. You 
> should look at the executables & files and see if you can do with auditing 
> what they are doing because its not interesting.
> 
> If this is causing you problems on the journald side where its causing other 
> tasks to fail. Then I'd break the link between auditd and journald. The amount 
> of event you show is highish but well within range of what auditd can do. Just 
> make sure flush is set to incremental_async and freq is 100 or 200. You should 
> be OK.
> 
> -Steve
> 
>> Key Summary Report
>> ===========================
>> total  key
>> ===========================
>> 8170  perm_mod
>> 5390  delete
>> 1073  access
>> 56  time-change
>> 26  session
>> 12  privileged
>> 7  logins
>>
>>
>> Syscall Summary Report
>> ==========================
>> total  syscall
>> ==========================
>> 4250  fchmodat
>> 1613  chmod
>> 831  fchmod
>> 521  fchown
>> 97  chown
>> 12  setxattr
>>
>>
>> Syscall Summary Report
>> ==========================
>> total  syscall
>> ==========================
>> 2887  unlink
>> 2189  rename
>> 186  unlinkat
>>
>>
>> so from the list the top 2 in each perm_mod and delete from the above
>> list seem to be the culprits..
> 
> 
>