[libvirt-users] Transient permission denied errors when sending audit logs

[Daniel P. Berrangé]

On Thu, Oct 17, 2019 at 11:26:12AM +0200, Roman Mohr wrote:
> Hi,
> 
> In kubevirt we are running into a strange permission problem on
> libvirt-5.0. We see transient "Permission Denied" errors when "virAuditSend"
> wants to send an audit log. [1] shows the logs of one of these containers.
> Here an example:
> 
> {"component":"virt-launcher","level":"warning","msg":"Failed to send audit
> message virt=kvm
> vm=\"kubevirt-test-default_testvmit2pqrkrlrwbhptcjcs4n67jn6pjqvmtd7pkrpdmkrl5sldzs4rxr9zdg8m45jxz\"
> uuid=56a33283-f6d7-4002-b188-1fed83186545 vm-ctx=+107:+107
> img-ctx=+107:+107 model=dac: Permission
> denied","pos":"virAuditSend:141","subcomponent":"libvirt","thread":"30","timestamp":"2019-10-08T23:58:40.651000Z"}
> 
> We recently switched in kubevirt to a dedicated selinux policy and remove
> the general "privileged" flag from the containers where we run libvirt in.
> This is very likely related to it, but we can't make sense out of it,
> because:
> 
>  * It randomly affects one out of a few hundred containers which we start
>  * It is not bound to a specific node
>  * It is only transient on that container. After a few denials libvirt can
> just continue.

The transient nature makes this really bizarre.

The error message above comes from the audit_log_user_message() call
that libvirt makes. This is implemented in libaudit.so using the
sendto() syscall on a netlink socket opened with

  socket(PF_NETLINK, SOCK_RAW, NETLINK_AUDIT);

So in terms of code being run there, there's very little - the EPERM
is coming back from the kernel when sending the message.

I'm not sure what scenario could cause this - perhaps the audit log
buffer in the kernel is full or something like that ?



>  * Sometimes it is accompanied with a transient "Permission denied" on
> /dev/null from our code in that container (so not from something which
> libvirt tries to do).
> 
> Has someone seen something like this before in different environments?

Never seen anything like this reported before.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|