[libvirt-users] Transient permission denied errors when sending audit logs

[Roman Mohr]

On Thu, Oct 17, 2019 at 11:34 AM Daniel P. Berrangé <berrange at redhat.com>
wrote:

> On Thu, Oct 17, 2019 at 11:26:12AM +0200, Roman Mohr wrote:
> > Hi,
> >
> > In kubevirt we are running into a strange permission problem on
> > libvirt-5.0. We see transient "Permission Denied" errors when
> "virAuditSend"
> > wants to send an audit log. [1] shows the logs of one of these
> containers.
> > Here an example:
> >
> > {"component":"virt-launcher","level":"warning","msg":"Failed to send
> audit
> > message virt=kvm
> >
> vm=\"kubevirt-test-default_testvmit2pqrkrlrwbhptcjcs4n67jn6pjqvmtd7pkrpdmkrl5sldzs4rxr9zdg8m45jxz\"
> > uuid=56a33283-f6d7-4002-b188-1fed83186545 vm-ctx=+107:+107
> > img-ctx=+107:+107 model=dac: Permission
> >
> denied","pos":"virAuditSend:141","subcomponent":"libvirt","thread":"30","timestamp":"2019-10-08T23:58:40.651000Z"}
> >
> > We recently switched in kubevirt to a dedicated selinux policy and remove
> > the general "privileged" flag from the containers where we run libvirt
> in.
> > This is very likely related to it, but we can't make sense out of it,
> > because:
> >
> >  * It randomly affects one out of a few hundred containers which we start
> >  * It is not bound to a specific node
> >  * It is only transient on that container. After a few denials libvirt
> can
> > just continue.
>
> The transient nature makes this really bizarre.
>
> The error message above comes from the audit_log_user_message() call
> that libvirt makes. This is implemented in libaudit.so using the
> sendto() syscall on a netlink socket opened with
>
>   socket(PF_NETLINK, SOCK_RAW, NETLINK_AUDIT);
>
> So in terms of code being run there, there's very little - the EPERM
> is coming back from the kernel when sending the message.
>
> I'm not sure what scenario could cause this - perhaps the audit log
> buffer in the kernel is full or something like that ?
>

I will adjust our collectors to collect the selinux auditlog and dmesg.
Maybe I can see more there. Will update the thread when I have more
information.


>
>
> >  * Sometimes it is accompanied with a transient "Permission denied" on
> > /dev/null from our code in that container (so not from something which
> > libvirt tries to do).
> >
> > Has someone seen something like this before in different environments?
>
> Never seen anything like this reported before.
>

Thanks Daniel.

Roman


>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20191017/b33efb54/attachment.htm>