[libvirt] [PATCH] Fix vm's outbound traffic control problem

[Michal Privoznik]

On 29.06.2012 08:09, Eiichi Tsukata wrote:
> Hello,
> 
> This is a patch to fix vm's outbound traffic control problem.
> 
> Currently, vm's outbound traffic control by libvirt doesn't go well.
> This problem was previously discussed at libvir-list ML, however
> it seems that there isn't still any answer to the problem.
> http://www.redhat.com/archives/libvir-list/2011-August/msg00333.html
> 
> I measured Guest(with virtio-net) to Host TCP throughput with the
> command "netperf -H".
> Here are the outbound QoS parameters and the results.
> 
> outbound average rate[kilobytes/s] : Guest to Host throughput[Mbit/s]
> ======================================================================
> 1024  (8Mbit/s)                    : 4.56
> 2048  (16Mbit/s)                   : 3.29
> 4096  (32Mbit/s)                   : 3.35
> 8192  (64Mbit/s)                   : 3.95
> 16384 (128Mbit/s)                  : 4.08
> 32768 (256Mbit/s)                  : 3.94
> 65536 (512Mbit/s)                  : 3.23
> 
> The outbound traffic goes down unreasonably and is even not controled.
> 
> The cause of this problem is too large mtu value in "tc filter" command run by
> libvirt. The command uses burst value to set mtu and the burst is equal to
> average rate value if it's not set. This value is too large. For example
> if the average rate is set to 1024 kilobytes/s, the mtu value is set to 1024
> kilobytes. That's too large compared to the size of network packets.
> Here libvirt applies tc ingress filter to Host's vnet(tun) device.
> Tc ingress filter is implemented with TBF(Token Buckets Filter) algorithm. TBF
> uses mtu value to calculate the amount of token consumed by each packet. With too
> large mtu value, the token consumption rate is set too large. This leads to
> token starvation and deterioration of TCP throughput.
> 
> Then, should we use the default mtu value 2 kilobytes?
> The anser is No, because Guest with virtio-net device uses 65536 bytes
> as mtu to transmit packets to Host, and the tc filter with the default mtu
> value 2k drops packets whose size is larger than 2k. So, the most packets
> is droped and again leads to deterioration of TCP throughput.
> 
> The appropriate mtu value is 65536 bytes which is equal to the maximum value
> of network interface device defined in <linux/netdevice.h>. The value is
> not so large that it causes token starvation and not so small that it
> drops most packets.
> Therefore this patch set the mtu value to 64kb(== 65535 bytes).
> 
> Again, here are the outbound QoS parameters and the TCP throughput with
> the libvirt patched.
> 
> outbound average rate[kilobytes/s] : Guest to Host throughput[Mbit/s]
> ======================================================================
> 1024  (8Mbit/s)                    : 8.22
> 2048  (16Mbit/s)                   : 16.42
> 4096  (32Mbit/s)                   : 32.93
> 8192  (64Mbit/s)                   : 66.85
> 16384 (128Mbit/s)                  : 133.88
> 32768 (256Mbit/s)                  : 271.01
> 65536 (512Mbit/s)                  : 547.32
> 
> The outbound traffic conforms to the given limit.
> 
> Thank you,
> 
> Signed-off-by: Eiichi Tsukata <eiichi.tsukata.xh at hitachi.com>
> ---
>  AUTHORS                       |    1 +
>  src/util/virnetdevbandwidth.c |    2 +-
>  2 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/AUTHORS b/AUTHORS
> index b876ee6..375db24 100644
> --- a/AUTHORS
> +++ b/AUTHORS
> @@ -246,6 +246,7 @@ Patches have also been contributed by:
>    Gerd Hoffmann        <kraxel at redhat.com>
>    Viktor Mihajlovski   <mihajlov at linux.vnet.ibm.com>
>    Thang Pham           <thang.pham at us.ibm.com>
> +  Eiichi Tsukata       <eiichi.tsukata.xh at hitachi.com>
> 
>    [....send patches to get your name here....]
> 
> diff --git a/src/util/virnetdevbandwidth.c b/src/util/virnetdevbandwidth.c
> index b9bd2e3..93df5c1 100644
> --- a/src/util/virnetdevbandwidth.c
> +++ b/src/util/virnetdevbandwidth.c
> @@ -136,7 +136,7 @@ virNetDevBandwidthSet(const char *ifname,
>          virCommandAddArgList(cmd, "filter", "add", "dev", ifname, "parent",
>                               "ffff:", "protocol", "ip", "u32", "match", "ip",
>                               "src", "0.0.0.0/0", "police", "rate", average,
> -                             "burst", burst, "mtu", burst, "drop", "flowid",
> +                             "burst", burst, "mtu", "64kb", "drop", "flowid",
>                               ":1", NULL);
> 
>          if (virCommandRun(cmd, NULL) < 0)
> 

Yeah, this tc magic was the hardest part when I've introduced QoS
feature. I tested it on small values where this has not fully showed.

ACK and pushed. Thanks!

Michal