<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="ZH-CN" link="blue" vlink="purple"> <div class="WordSection1"> Hi Yalan,<o:p></o:p> You are right about it. For other interface type, values in tc rules are calculated by multiply 8*1000 instead of 8*1024. I didn’t notice it. To make them uniform, I will fix it in next patch. Really thanks for your help.<o:p></o:p> <o:p> </o:p> Best Regards,<o:p></o:p> Jinsheng Zhang<o:p></o:p> 发件人: Yalan Zhang [mailto:yalzhang@redhat.com] 发送时间: 2021年10月29日 16:52 收件人: Jinsheng Zhang (张金生)-云服务集团 抄送: libvir-list@redhat.com; Norman Shen(申嘉童) 主题: Re: [PATCH v3 0/4] Add qemu support setting qos via ovs on ovs interface<o:p></o:p> <o:p> </o:p> <div> <div> Hi Jinsheng,<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> I asked as I have mixed the "K" as 1000. Thank you for the explanation, I'm clear now.<o:p></o:p> </div> <div> And I found the "inbound.peak : 200" was calculated to "ceil 1638Kbit", maybe 1600Kbit is more reasonable? as I found for other interface type like nat, it was 1600Kbit from tc output. Please help to confirm, Thank you!<o:p></o:p> </div> <div> # virsh domiftune rhel vnet5<o:p></o:p> inbound.average: 100<o:p></o:p> inbound.peak : 200<o:p></o:p> inbound.burst : 256<o:p></o:p> ...<o:p></o:p> # tc -d class show dev vnet5<o:p></o:p> class htb 1:1 parent 1:fffe prio 0 quantum 10240 rate 819200bit ceil 1638Kbit linklayer ethernet burst 256Kb/1 mpu 0b cburst 256Kb/1 mpu 0b level 0 <o:p></o:p> class htb 1:fffe root rate 1638Kbit ceil 1638Kbit linklayer ethernet burst 1499b/1 mpu 0b cburst 1499b/1 mpu 0b level 7 <o:p></o:p> </div> <div> <div> <div> <div> ------- Best Regards, Yalan Zhang IRC: yalzhang<o:p></o:p> </div> </div> </div> </div> <o:p> </o:p> </div> <o:p> </o:p> <div> <div> On Thu, Oct 28, 2021 at 4:20 PM Jinsheng Zhang (张金生)-云服务集团 <<a href="mailto:zhangjl02@inspur.com">zhangjl02@inspur.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> Hi Yalan,<o:p></o:p> It seems that there is no output error abount inbound settings from your statistics. 100KB is short for 100 kilobytes, and 1 byte is 8 bit, therefore 100 kilobytes is 800 kilobit and is also 1024*800 bit which is 819200 bit or 800 Kbit for short. Similarly, 200 KB is equal to 1600Kbit.<o:p></o:p> <o:p></o:p> From your test results, inbound.average is set to 400 KB which is 400 * 1024 * 8 bit(approximately 3.2*10^6 bits). outbound.average is set to 100 KB which is approximately 0.8*10^6 bits. Considering peek and burst is larger than average. The netperf test result is meaningful.<o:p></o:p> <o:p></o:p> For the second bug mentioned, after create the ovs-net, tc rules are created. But when attach an interface to an instance, qos settings is not add to port neither in xml or tc . It is a bug, I think. I will think about fixing this.<o:p></o:p> <o:p></o:p> -------<o:p></o:p> Best Regards,<o:p></o:p> Jinsheng Zhang<o:p></o:p> 发件人: Yalan Zhang [mailto:<a href="mailto:yalzhang@redhat.com" target="_blank">yalzhang@redhat.com</a>] 发送时间: 2021年10月27日 18:35 收件人: Jinsheng Zhang (张金生)-云服务集团 抄送: <a href="mailto:libvir-list@redhat.com" target="_blank"> libvir-list@redhat.com</a>; Norman Shen(申嘉童) 主题: Re: [PATCH v3 0/4] Add qemu support setting qos via ovs on ovs interface<o:p></o:p> <o:p></o:p> <div> <div> <o:p></o:p> </div> Hi Jinsheng, <o:p></o:p> <div> <o:p></o:p> </div> <div> Thank you for the explanation. From the statistics above, the tc outputs for outbound matches. But I'm confused about the inbound statistics:<o:p></o:p> </div> <div> <div> <div> <div> # virsh domiftune rhel vnet5<o:p></o:p> inbound. approximately 3.2*10^6 bits: 100<o:p></o:p> inbound.peak : 200<o:p></o:p> inbound.burst : 256<o:p></o:p> ...<o:p></o:p> # tc -d class show dev vnet5<o:p></o:p> class htb 1:1 parent 1:fffe prio 0 quantum 10240 rate 819200bit ceil 1638Kbit linklayer ethernet burst 256Kb/1 mpu 0b cburst 256Kb/1 mpu 0b level 0 <o:p></o:p> class htb 1:fffe root rate 1638Kbit ceil 1638Kbit linklayer ethernet burst 1499b/1 mpu 0b cburst 1499b/1 mpu 0b level 7 <o:p></o:p> <o:p></o:p> As the value in libvirt xml is KB, inbound.average: *100 KB* can not match with *"rate 819200bit"* in tc outputs, I supposed it should be 800Kbit. Please help to confirm.<o:p></o:p> And so does "ceil 1638Kbit" (may be it should be 1600Kbit as "inbound.peak : 200").<o:p></o:p> <o:p></o:p> I have run netperf to test the actual rate, the result is pass. 2 vm connected to the same bridge, set one vm with Qos, see test results below:<o:p></o:p> # virsh domiftune rhel vnet0 inbound.average: 400 inbound.peak : 500 inbound.burst : 125 inbound.floor : 0 outbound.average: 100 outbound.peak : 200 outbound.burst : 256<o:p></o:p> Throughput for inbound: 3.92 * 10^6bits/sec<o:p></o:p> Throughput for outbound: 0.93 * 10^6bits/sec<o:p></o:p> <div> <o:p></o:p> </div> <div> These patches fixed the bug [1] which closed with deferred resolution. Thank you!<o:p></o:p> </div> <div> And this reminds me of another ovs Qos related bug [2], which was about network. <o:p></o:p> </div> <div> And I tried with the scenarios in [2], there are no changes(not fixed). Just for information. :-)<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> [1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1510237" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1510237</a><o:p></o:p> </div> <div> [2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1826168" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1826168</a><o:p></o:p> </div> <div> <o:p></o:p> </div> ------- Best Regards, Yalan Zhang IRC: yalzhang<o:p></o:p> </div> </div> </div> <o:p></o:p> </div> </div> <o:p></o:p> <div> On Tue, Oct 26, 2021 at 3:23 PM Jinsheng Zhang (张金生)-云服务集团 <<a href="mailto:zhangjl02@inspur.com" target="_blank">zhangjl02@inspur.com</a>> wrote:<o:p></o:p> </div> Hi Yalan,<o:p></o:p> <o:p></o:p> 1) For inbound, we can use `ovs-vsctl list qos` and `ovs-vsctl list queue` to check them from the openvswitch side. Values can be found in other_config. Inbound is in kbyte when set qos with `virsh domiftune …`, well it is in bit in ovs, Therefore, when inbound.average is set to 100, the corresponding value will be set to 819200 in ovs. <o:p></o:p> 2) For outbound, it is in kbyte in libvirt and ingress_policing_XX in ovs interface is in kbit.<o:p></o:p> 3) Ovs use tc to set qos, so we can see output from tc command. <o:p></o:p> This patch is to unify the qos control and query on ovs ports. <o:p></o:p> The conversion explanation is added in this patch: <a href="https://listman.redhat.com/archives/libvir-list/2021-August/msg00422.html" target="_blank"> https://listman.redhat.com/archives/libvir-list/2021-August/msg00422.html</a><o:p></o:p> And there are 6 following patches to fix some bugs. See <a href="https://listman.redhat.com/archives/libvir-list/2021-August/msg00423.html" target="_blank"> https://listman.redhat.com/archives/libvir-list/2021-August/msg00423.html</a><o:p></o:p> <o:p></o:p> -------<o:p></o:p> Best Regards,<o:p></o:p> Jinsheng Zhang<o:p></o:p> <o:p></o:p> 发件人: Yalan Zhang [mailto:<a href="mailto:yalzhang@redhat.com" target="_blank">yalzhang@redhat.com</a>] 发送时间: 2021年10月25日 17:54 收件人: Michal Prívozník; Jinsheng Zhang (张金生)-云服务集团 抄送: <a href="mailto:libvir-list@redhat.com" target="_blank"> libvir-list@redhat.com</a>; Norman Shen(申嘉童); zhangjl02 主题: Re: [PATCH v3 0/4] Add qemu support setting qos via ovs on ovs interface<o:p></o:p> <o:p></o:p> <div> <div> Hi Jinsheng,<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> I have tested the patch and have some questions, could you please help to confirm?<o:p></o:p> </div> <div> 1) For inbound, how to check it from the openvswitch side? tc will still show the statistics, is that expected?<o:p></o:p> </div> <div> 2) For outbound, the peak is ignored. I just can not understand the "ingress_policing_burst: 2048", how can it come from the setting "outbound.burst : 256"?<o:p></o:p> </div> <div> 3) Is the output from tc command expected? <o:p></o:p> </div> <div> <o:p></o:p> </div> <div> Test inbound:<o:p></o:p> </div> <div> 1. start vm with setting as below:<o:p></o:p> </div> <div> <interface type='bridge'> <source bridge='ovsbr0'/> <virtualport type='openvswitch'/><o:p></o:p> </div> <div> <bandwidth><o:p></o:p> <inbound average='100' peak='200' burst='256'/><o:p></o:p> </bandwidth><o:p></o:p> ...<o:p></o:p> </interface><o:p></o:p> 2. <o:p></o:p> </div> <div> # virsh domiftune rhel vnet5<o:p></o:p> inbound.average: 100<o:p></o:p> inbound.peak : 200<o:p></o:p> inbound.burst : 256<o:p></o:p> inbound.floor : 0<o:p></o:p> outbound.average: 0<o:p></o:p> outbound.peak : 0<o:p></o:p> outbound.burst : 0<o:p></o:p> # ip l<o:p></o:p> 17: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000<o:p></o:p> link/ether fe:54:00:4d:43:5a brd ff:ff:ff:ff:ff:ff<o:p></o:p> </div> <div> # ovs-vsctl show interface<o:p></o:p> …...<o:p></o:p> ingress_policing_burst: 0<o:p></o:p> ingress_policing_kpkts_burst: 0<o:p></o:p> ingress_policing_kpkts_rate: 0<o:p></o:p> ingress_policing_rate: 0<o:p></o:p> …...<o:p></o:p> name : vnet5<o:p></o:p> <o:p></o:p> </div> <div> # tc -d class show dev vnet5<o:p></o:p> class htb 1:1 parent 1:fffe prio 0 quantum 10240 rate 819200bit ceil 1638Kbit linklayer ethernet burst 256Kb/1 mpu 0b cburst 256Kb/1 mpu 0b level 0 <o:p></o:p> class htb 1:fffe root rate 1638Kbit ceil 1638Kbit linklayer ethernet burst 1499b/1 mpu 0b cburst 1499b/1 mpu 0b level 7 <o:p></o:p> # tc -d filter show dev vnet5 parent ffff:<o:p></o:p> (no outputs)<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> For outbound:<o:p></o:p> </div> <div> # virsh dumpxml rhel | grep /bandwidth -B2<o:p></o:p> <bandwidth><o:p></o:p> <outbound average='100' peak='200' burst='256'/><o:p></o:p> </bandwidth><o:p></o:p> </div> <div> # virsh domiftune rhel vnet9<o:p></o:p> inbound.average: 0<o:p></o:p> inbound.peak : 0<o:p></o:p> inbound.burst : 0<o:p></o:p> inbound.floor : 0<o:p></o:p> outbound.average: 100<o:p></o:p> outbound.peak : 200<o:p></o:p> outbound.burst : 256<o:p></o:p> # ovs-vsctl list interface<o:p></o:p> ingress_policing_burst: 2048<o:p></o:p> ingress_policing_kpkts_burst: 0<o:p></o:p> ingress_policing_kpkts_rate: 0<o:p></o:p> ingress_policing_rate: 800<o:p></o:p> ...<o:p></o:p> </div> <div> # tc -d filter show dev vnet9 parent ffff:<o:p></o:p> filter protocol all pref 49 basic chain 0 <o:p></o:p> filter protocol all pref 49 basic chain 0 handle 0x1 <o:p></o:p> action order 1: police 0x1 rate 800Kbit burst 256Kb mtu 64Kb action drop/pipe overhead 0b linklayer unspec <o:p></o:p> ref 1 bind 1<o:p></o:p> # tc -d class show dev vnet9 <o:p></o:p> (no outputs)<o:p></o:p> </div> <div> <div> <div> <div> ------- Best Regards, Yalan Zhang IRC: yalzhang<o:p></o:p> </div> </div> </div> </div> <o:p></o:p> </div> <o:p></o:p> <div> On Mon, Jul 12, 2021 at 3:43 PM Michal Prívozník <<a href="mailto:mprivozn@redhat.com" target="_blank">mprivozn@redhat.com</a>> wrote:<o:p></o:p> </div> On 7/9/21 3:31 PM, Jinsheng Zhang (张金生)-云服务集团 wrote: > Here is my signed-off-by line > > Signed-off-by: <a href="mailto:zhangjl02@inspur.com" target="_blank">zhangjl02@inspur.com</a> > > Thanks again for reminding:) . Perfect. Reviewed-by: Michal Privoznik <<a href="mailto:mprivozn@redhat.com" target="_blank">mprivozn@redhat.com</a>> and pushed. Congratulations on your first libvirt contribution! Michal<o:p></o:p> </div> </div> </blockquote> </div> </div> </body> </html>