[libvirt] [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology (CMT)

John Ferlan jferlan at redhat.com
Tue Nov 27 16:05:53 UTC 2018 


On 11/27/18 8:28 AM, Wang, Huaqiang wrote:
> Hi John,
> 
> The issue mentioned is generated by a -1 returned by qemuDomainGetIOThreadsMon. 
> 
> Further, it is caused by a -1 returned by virQEMUCapsGet:
>  6174         if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_OBJECT_IOTHREAD)) {
>  6175             virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
>  6176                            _("IOThreads not supported with this binary"));
>  6177             goto endjob;
>  6178         }
> 
> By checking qemuCaps->flags, it seems the bit of QEMU_CAPS_OBJECT_IOTHREAD(176) is not set. 
> (gdb) call virBitmapFormat(qemuCaps->flags)
> $4 = 0x7f08a4004cb0 "13,33,46,50,54-55,58,61-64,66-70,72-73,77-78,80,87-88,92-93,95-97,99,102-110,112,114,119-121,128-131,141-142,146,148-149,151-156,158-161,165,167,174-175,180,182,188,193-198,210,214,221-222,225,250,255"...
> 

Hmm... Thanks for the details...

This is an "interesting" failure.  There's a couple of bugs I suppose.
One simple to fix - the qemuConnectGetAllDomainStats should have saved
the error, but the call to virDomainStatsRecordListFree wipes out the
error when calling virDomainFree (via virResetLastError) when there's
more than one domain's worth of stats being collect and "some" domains
had already succeeded. So saving that error across the failure is
possible via a couple of means.

The other part of this would be a change to qemuDomainGetStatsIOThread
to not fail when the cap isn't available, but yet still allow a failure
if the cap was there, but the mon call failed.

As an aside, your QEMU must be fairly "old" to fail on the lack of the
capability QEMU_CAPS_OBJECT_IOTHREAD... Perhaps older than 2.0 - which
doesn't quite make sense given everything else.

I'll post a couple of patches shortly...

Tks,

John


> Breakpoint 1 at 0x7f088ef00880: file qemu/qemu_driver.c, line 5492.
> (gdb) c
> Continuing.
> [Switching to Thread 0x7f08bcdcb700 (LWP 24711)]
> 
> Breakpoint 1, qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492
> 5492    {
> Unprocessed breakpoint [1]
> (gdb) finish
> Run till exit from #0  qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492
> 0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885
> 20885       if ((niothreads = qemuDomainGetIOThreadsMon(driver, dom, &iothreads)) < 0)
> Value returned is $1 = -1
> (gdb) bt
> #0  0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885
> #1  0x00007f088eee70a9 in qemuDomainGetStats (flags=1, record=<synthetic pointer>, stats=255, dom=0x7f08740cd840, conn=0x7f08ac000c50) at qemu/qemu_driver.c:21062
> #2  qemuConnectGetAllDomainStats (conn=0x7f08ac000c50, doms=<optimized out>, ndoms=<optimized out>, stats=255, retStats=0x7f08bcdcab10, flags=<optimized out>) at qemu/qemu_driver.c:21162
> #3  0x00007f08cd147646 in virConnectGetAllDomainStats (conn=0x7f08ac000c50, stats=0, retStats=retStats at entry=0x7f08bcdcab10, flags=0) at libvirt-domain.c:11685
> #4  0x0000562e2404d2c0 in remoteDispatchConnectGetAllDomainStats (server=0x562e24ee74a0, msg=<optimized out>, ret=0x7f08ac001ec0, args=0x7f08ac001f30, rerr=0x7f08bcdcac50, client=0x562e24ef5200) at remote/remote_daemon_dispatch.c:6665
> #5  remoteDispatchConnectGetAllDomainStatsHelper (server=0x562e24ee74a0, client=0x562e24ef5200, msg=<optimized out>, rerr=0x7f08bcdcac50, args=0x7f08ac001f30, ret=0x7f08ac001ec0) at remote/remote_daemon_dispatch_stubs.h:743
> #6  0x00007f08cd05dad5 in virNetServerProgramDispatchCall (msg=0x562e24ef5910, client=0x562e24ef5200, server=0x562e24ee74a0, prog=0x562e24ee6650) at rpc/virnetserverprogram.c:437
> #7  virNetServerProgramDispatch (prog=0x562e24ee6650, server=server at entry=0x562e24ee74a0, client=0x562e24ef5200, msg=0x562e24ef5910) at rpc/virnetserverprogram.c:304
> #8  0x00007f08cd063f7d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x562e24ee74a0) at rpc/virnetserver.c:144
> #9  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562e24ee74a0) at rpc/virnetserver.c:165
> #10 0x00007f08ccf95171 in virThreadPoolWorker (opaque=opaque at entry=0x562e24ee69a0) at util/virthreadpool.c:167
> #11 0x00007f08ccf944f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
> #12 0x00007f08cb213dc5 in start_thread (arg=0x7f08bcdcb700) at pthread_create.c:308
> #13 0x00007f08cab3b76d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> (gdb)
> 
> My domain configuration is：
> [root at dl-c200 libvirt]# virsh list
> v^H Id   Name               State
> ----------------------------------
>  1    ubuntu16.04-base   running
> 
> [root at dl-c200 libvirt]# virsh dumpxml 1
> <domain type='kvm' id='1'>
>   <name>ubuntu16.04-base</name>
>   <uuid>19d01bd1-ad54-4176-84bd-e510de08fe00</uuid>
>   <memory unit='KiB'>10485760</memory>
>   <currentMemory unit='KiB'>10485760</currentMemory>
>   <vcpu placement='static' current='4'>8</vcpu>
>   <cputune>
>     <cachetune vcpus='0' id='vcpus_0'>
>       <monitor level='3' vcpus='0'/>
>     </cachetune>
>   </cputune>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>   </features>
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Broadwell</model>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='require' name='xsaveopt'/>
>   </cpu>
>   <clock offset='utc'>
>     <timer name='rtc' tickpolicy='catchup'/>
>     <timer name='pit' tickpolicy='delay'/>
>     <timer name='hpet' present='no'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <pm>
>     <suspend-to-mem enabled='no'/>
>     <suspend-to-disk enabled='no'/>
>   </pm>
>   <devices>
>     <emulator>/usr/libexec/qemu-kvm</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/tmp/kvm-donot-delete/vm-ubuntu16.04-base'/>
>       <backingStore/>
>       <target dev='vda' bus='virtio'/>
>       <alias name='virtio-disk0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu'/>
>       <target dev='hda' bus='ide'/>
>       <readonly/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <controller type='usb' index='0' model='ich9-ehci1'>
>       <alias name='usb'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci1'>
>       <alias name='usb'/>
>       <master startport='0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci2'>
>       <alias name='usb'/>
>       <master startport='2'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
>     </controller>
>     <controller type='usb' index='0' model='ich9-uhci3'>
>       <alias name='usb'/>
>       <master startport='4'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pci-root'>
>       <alias name='pci.0'/>
>     </controller>
>     <controller type='ide' index='0'>
>       <alias name='ide'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:eb:24:d3'/>
>       <source bridge='virbr0'/>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <source path='/dev/pts/1'/>
>       <target type='isa-serial' port='0'>
>         <model name='isa-serial'/>
>       </target>
>       <alias name='serial0'/>
>     </serial>
>     <console type='pty' tty='/dev/pts/1'>
>       <source path='/dev/pts/1'/>
>       <target type='serial' port='0'/>
>       <alias name='serial0'/>
>     </console>
>     <input type='mouse' bus='ps2'>
>       <alias name='input0'/>
>     </input>
>     <input type='keyboard' bus='ps2'>
>       <alias name='input1'/>
>     </input>
>     <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1'>
>       <listen type='address' address='127.0.0.1'/>
>     </graphics>
>     <video>
>       <model type='cirrus' vram='16384' heads='1' primary='yes'/>
>       <alias name='video0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <alias name='balloon0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>     </memballoon>
>   </devices>
>   <seclabel type='dynamic' model='selinux' relabel='yes'>
>     <label>system_u:system_r:svirt_t:s0:c157,c912</label>
>     <imagelabel>system_u:object_r:svirt_image_t:s0:c157,c912</imagelabel>
>   </seclabel>
>   <seclabel type='dynamic' model='dac' relabel='yes'>
>     <label>+0:+0</label>
>     <imagelabel>+0:+0</imagelabel>
>   </seclabel>
> </domain>
> 
> I wonder if these message is enough for you to locate the root cause, I don’t have too much time in investigating this today, I'll try to create a fix tomorrow if you need.
> 
> Huaqiang
> 
>> -----Original Message-----
>> From: Wang, Huaqiang
>> Sent: Tuesday, November 27, 2018 11:21 AM
>> To: 'John Ferlan' <jferlan at redhat.com>; libvir-list at redhat.com
>> Subject: RE: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology
>> (CMT)
>>
>>
>>
>>> -----Original Message-----
>>> From: John Ferlan [mailto:jferlan at redhat.com]
>>> Sent: Tuesday, November 27, 2018 10:59 AM
>>> To: Wang, Huaqiang <huaqiang.wang at intel.com>; libvir-list at redhat.com
>>> Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology
>>> (CMT)
>>>
>>>
>>>
>>> On 11/26/18 9:39 PM, Wang, Huaqiang wrote:
>>>> Hi John,
>>>>
>>>> Really appreciate your hard work for the CMT series. Next I'll
>>>> working on
>>> the MBM.
>>>>
>>>> In testing the newly pushed code, I find a problem:
>>>>
>>>> <error message>
>>>> [david at dl-c200 ~]$ sudo virsh domstats
>>>> error: An error occurred, but the cause is unknown </error message>
>>>>
>>>
>>> I couldn't reproduced in a quick test here. Can you get a thread trace
>>> of the failure?
>>>
>>> What I usually do, build libvirt, then in a terminal session at the
>>> top of the git tree "./run gdb src/libvirtd"  (dbg> r)... THen in
>>> another terminal session run the virsh command and when the libvirtd
>> session stops do a "t a a bt"
>>> (thread apply all backtrace)...
>>>
>>> John
>>
>> I'll trace the error.
>>
>> Thanks.
>> Huaqiang
>>
>>>
>>> (done for the night)
>>>
>>>> seems it is caused by qemuDomainGetStatsIOThread not by the new
>>> series.
>>>> What I test is return 0 immediately at top of
>>>> qemuDomainGetStatsIOThread, the command 'virsh domstats' reports
>>> the cache statistics normally.
>>>>
>>>> BR
>>>> Huaqiang
>>>>
>>>>> -----Original Message-----
>>>>> From: John Ferlan [mailto:jferlan at redhat.com]
>>>>> Sent: Tuesday, November 27, 2018 9:49 AM
>>>>> To: Wang, Huaqiang <huaqiang.wang at intel.com>; libvir-
>>> list at redhat.com
>>>>> Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring
>>> Technology
>>>>> (CMT)
>>>>>
>>>>>
>>>>>
>>>>> On 11/26/18 12:56 PM, Wang Huaqiang wrote:
>>>>>> These patches are the remaining part for the CMT enabling series,
>>>>>> and most of the series have been merged.
>>>>>>
>>>>>> This series is addressing John's review comments and suggestions,
>>>>>> which are
>>>>>> https://www.redhat.com/archives/libvir-list/2018-
>>>>> November/msg00907.htm
>>>>>> l
>>>>>> https://www.redhat.com/archives/libvir-list/2018-
>>>>> November/msg00510.htm
>>>>>> l
>>>>>> https://www.redhat.com/archives/libvir-list/2018-
>>>>> November/msg00561.htm
>>>>>> l
>>>>>>
>>>>>> Change lists:
>>>>>> Changes in v10:
>>>>>> -. Add tag (virResctrlMonitorType) in
>> qemuDomainGetResctrlMonData,
>>>>> thus
>>>>>>    qemuDomainGetResctrlMonData could be reused for MBM.
>>>>>> -. Using VIR_APPEND_ELEMENT to append
>> virQEMUResctrlMonDataPtr
>>>>> list.
>>>>>> -. Add qemuDomainFreeResctrlMonData.
>>>>>> -. Add virResctrlMonitorFreeStats.
>>>>>> -. Return a list of virResctrlMonitorStatsPtr instead of
>>>>>>    a virResctrlMonitorStats array in virResctrlMonitorGetStats.
>>>>>>
>>>>>> Changes in V9:
>>>>>> -. Addressing code review comments form John.
>>>>>> -. Refined the names for new data structure and new functions.
>>>>>> -. Merged qemuDomainGetStatsCpuResMonitorPerTag and
>>>>> qemuDomainGetStatsCpuResMonitor,
>>>>>>    and refined new function name based on the fact that we only
>>>>>> support
>>>>> cache monitor now.
>>>>>> Wang Huaqiang (4):
>>>>>>   util: Return a list of pointer in virResctrlMonitorGetStats
>>>>>>   util: Add function to free monitor statistical data
>>>>>>   qemu: Report cache occupancy (CMT) with domstats
>>>>>>   docs: Updated news.xml for CMT
>>>>>>
>>>>>>  docs/news.xml            |  12 ++++
>>>>>>  src/libvirt-domain.c     |  12 ++++
>>>>>>  src/libvirt_private.syms |   1 +
>>>>>>  src/qemu/qemu_driver.c   | 183
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++-
>>>>>>  src/util/virresctrl.c    |  26 +++++--
>>>>>>  src/util/virresctrl.h    |   8 ++-
>>>>>>  tools/virsh.pod          |  14 ++++
>>>>>>  7 files changed, 248 insertions(+), 8 deletions(-)
>>>>>>
>>>>>
>>>>> Reviewed-by: John Ferlan <jferlan at redhat.com>
>>>>> (series)
>>>>>
>>>>> and pushed,
>>>>>
>>>>> John

More information about the libvir-list mailing list