<div dir="ltr"><div>Hi Daniel,</div><div><br></div><div>just wanted to ask if you had a chance to check the shared file.</div><div><br></div><div>Let me know if you need more info.</div><div><br></div><div>Best regards,</div><div>Roman</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 13, 2022 at 5:31 PM Roman Mohr <<a href="mailto:rmohr@google.com">rmohr@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Sep 8, 2022 at 4:32 PM Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, Sep 08, 2022 at 04:22:17PM +0200, Roman Mohr wrote:<br> > On Thu, Sep 8, 2022 at 4:04 PM Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>><br> > wrote:<br> > <br> > > On Thu, Sep 08, 2022 at 02:23:31PM +0100, Daniel P. Berrangé wrote:<br> > > > On Thu, Sep 08, 2022 at 03:10:09PM +0200, Roman Mohr wrote:<br> > > > > On Thu, Sep 8, 2022 at 2:56 PM Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a><br> > > ><br> > > > > wrote:<br> > > > ><br> > > > > > On Thu, Sep 08, 2022 at 02:24:00PM +0200, Roman Mohr wrote:<br> > > > > > > Hi,<br> > > > > > ><br> > > > > > ><br> > > > > > > I have a question regarding capability caching in the context of<br> > > > > > KubeVirt.<br> > > > > > > Since we start in KubeVirt one libvirt instance per VM, libvirt<br> > > has to<br> > > > > > > re-discover on every VM start the qemu capabilities which leads to<br> > > a<br> > > > > > 1-2s+<br> > > > > > > delay in startup.<br> > > > > > ><br> > > > > > > We already discover the features in a dedicated KubeVirt pod on<br> > > each<br> > > > > > node.<br> > > > > > > Therefore I tried to copy the capabilities over to see if that<br> > > would<br> > > > > > work.<br> > > > > > ><br> > > > > > > It looks like in general it could work, but libvirt seems to<br> > > detect a<br> > > > > > > mismatch in the exposed KVM CPU ID in every pod. Therefore it<br> > > invalidates<br> > > > > > > the cache. The recreated capability cache looks esctly like the<br> > > original<br> > > > > > > one though ...<br> > > > > > ><br> > > > > > > The check responsible for the invalidation is this:<br> > > > > > ><br> > > > > > > ```<br> > > > > > > Outdated capabilities for '%s': host cpuid changed<br> > > > > > > ```<br> > > > > > ><br> > > > > > > So the KVM_GET_SUPPORTED_CPUID call seems to return<br> > > > > > > slightly different values in different containers.<br> > > > > > ><br> > > > > > > After trying out the attached golang scripts in different<br> > > containers, I<br> > > > > > > could indeed see differences.<br> > > > > > ><br> > > > > > > I can however not really judge what the differences in these KVM<br> > > function<br> > > > > > > registers mean and I am curious if someone else knows. The files<br> > > are<br> > > > > > > attached too (as json for easy diffing).<br> > > > > ><br> > > > > > Can you confirm whether the two attached data files were captured<br> > > > > > by containers running on the same physical host, or could each<br> > > > > > container have run on a different host.<br> > > > > ><br> > > > ><br> > > > > They are coming from the same host, that is the most surprising bit<br> > > for me.<br> > > > > I am also very sure that this is the case, because I only had one k8s<br> > > node<br> > > > > from where I took these.<br> > > > > The containers however differ (obviously) on namespaces and on the<br> > > > > privilege level (less obvious). The handler dump is from a fully<br> > > privileged<br> > > > > container.<br> > > ><br> > > > The privilege level sounds like something that might be impactful,<br> > > > so I'll investigate that. I'd be pretty surprised for namespaces<br> > > > to have any impact thnough.<br> > ><br> > > The privilege level is a red herring. Peter reminded me that we have<br> > > to filter out some parts of CPUID because the APIC IDs vary depending<br> > > on what host CPU the task executes on.<br> > ><br> > ><br> > > <a href="https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virhostcpu.c#L1346" rel="noreferrer" target="_blank">https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virhostcpu.c#L1346</a><br> > ><br> > > In the 2 jSON files you provide, the differences i see should already<br> > > be matched by<br> > ><br> > > /* filter out local apic id */<br> > > if (entry->function == 0x01 && entry->index == 0x00)<br> > > entry->ebx &= 0x00ffffff;<br> > > if (entry->function == 0x0b)<br> > > entry->edx &= 0xffffff00;<br> > ><br> > > so those differences ought not to be causing the cache to be<br> > > invalidated.<br> > ><br> > <br> > Hm, maybe I misinterpreted the logs then. The snipped I looked at was this:<br> > <br> > <br> > ```<br> > {"component":"virt-launcher","level":"info","msg":"/dev/kvm has changed<br> > (1661786802 vs<br> > 0)","pos":"virQEMUCapsKVMUsable:4850","subcomponent":"libvirt","thread":"25","timestamp":"2022-08-29T15:26:42.936000Z"}<br> > {"component":"virt-launcher","level":"info","msg":"a=0x7f8138153ba0,<br> > b=0x7f818001c480","pos":"virCPUDataIsIdentical:1178","subcomponent":"libvirt","thread":"25","timestamp":"2022-08-29T15:26:42.939000Z"}<br> > {"component":"virt-launcher","level":"info","msg":"Outdated capabilities<br> > for '/usr/bin/qemu-system-x86_64': host cpuid<br> > changed","pos":"virQEMUCapsIsValid:4993","subcomponent":"libvirt","thread":"25","timestamp":"2022-08-29T15:26:42.939000Z"}<br> > {"component":"virt-launcher","level":"info","msg":"Outdated cached<br> > capabilities<br> > '/var/cache/libvirt/qemu/capabilities/926803a9278e445ec919c2b6cbd8c1c449c75b26dcb1686b774314180376c725.xml'<br> > for<br> > '/usr/bin/qemu-system-x86_64'","pos":"virFileCacheLoad:163","subcomponent":"libvirt","thread":"25","timestamp":"2022-08-29T15:26:42.939000Z"}<br> > ```<br> <br> Can you capture the <br> <br> /var/cache/libvirt/qemu/capabilities/926803a9278e445ec919c2b6cbd8c1c449c75b26dcb1686b774314180376c725.xml' </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> from the virt-handler and virt-launcher pods. It contains a <cpuid><br> block that will show us the differences libvirt recored, /after/,<br> libvirt has done its filtering. This will show if there si more<br> we need to filter.<br></blockquote><div><br></div><div>Done. Only attached the one from handler since it is 100% identical to the launcher.</div><div><br></div><div>Let me know if you need more information.</div><div><br></div><div>Thanks and best regards,</div><div>Roman</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> > I had the impression from the code that the `/dev/kvm` change (because the<br> > containers are not created at the same time) does not invalidate it either.<br> > <br> > I added the whole debug log, maybe I missed something obvious.<br> > <br> > Does it make a difference if the cache is created via `virsh<br> > domcapabilities` and `virsh capabilities` or via defining the first domain?<br> <br> They'll all end up at the same caching scode so should not make any<br> difference.<br> <br> > <br> > Best regards,<br> > Roman<br> > <br> > <br> > ><br> > > With regards,<br> > > Daniel<br> > > --<br> > > |: <a href="https://berrange.com" rel="noreferrer" target="_blank">https://berrange.com</a> -o-<br> > > <a href="https://www.flickr.com/photos/dberrange" rel="noreferrer" target="_blank">https://www.flickr.com/photos/dberrange</a> :|<br> > > |: <a href="https://libvirt.org" rel="noreferrer" target="_blank">https://libvirt.org</a> -o-<br> > > <a href="https://fstop138.berrange.com" rel="noreferrer" target="_blank">https://fstop138.berrange.com</a> :|<br> > > |: <a href="https://entangle-photo.org" rel="noreferrer" target="_blank">https://entangle-photo.org</a> -o-<br> > > <a href="https://www.instagram.com/dberrange" rel="noreferrer" target="_blank">https://www.instagram.com/dberrange</a> :|<br> > ><br> > ><br> <br> <br> <br> With regards,<br> Daniel<br> -- <br> |: <a href="https://berrange.com" rel="noreferrer" target="_blank">https://berrange.com</a> -o- <a href="https://www.flickr.com/photos/dberrange" rel="noreferrer" target="_blank">https://www.flickr.com/photos/dberrange</a> :|<br> |: <a href="https://libvirt.org" rel="noreferrer" target="_blank">https://libvirt.org</a> -o- <a href="https://fstop138.berrange.com" rel="noreferrer" target="_blank">https://fstop138.berrange.com</a> :|<br> |: <a href="https://entangle-photo.org" rel="noreferrer" target="_blank">https://entangle-photo.org</a> -o- <a href="https://www.instagram.com/dberrange" rel="noreferrer" target="_blank">https://www.instagram.com/dberrange</a> :|<br> <br> </blockquote></div></div> </blockquote></div></div>