[libvirt] [PATCH] Fix vm's outbound traffic control problem

[Daniel Veillard]

On Fri, Jun 29, 2012 at 11:08:18AM +0200, Michal Privoznik wrote:
> On 29.06.2012 08:09, Eiichi Tsukata wrote:
> > Hello,
> > 
> > This is a patch to fix vm's outbound traffic control problem.
> > 
> > Currently, vm's outbound traffic control by libvirt doesn't go well.
> > This problem was previously discussed at libvir-list ML, however
> > it seems that there isn't still any answer to the problem.
> > http://www.redhat.com/archives/libvir-list/2011-August/msg00333.html
> > 
> > I measured Guest(with virtio-net) to Host TCP throughput with the
> > command "netperf -H".
> > Here are the outbound QoS parameters and the results.
> > 
> > outbound average rate[kilobytes/s] : Guest to Host throughput[Mbit/s]
> > ======================================================================
> > 1024  (8Mbit/s)                    : 4.56
> > 2048  (16Mbit/s)                   : 3.29
> > 4096  (32Mbit/s)                   : 3.35
> > 8192  (64Mbit/s)                   : 3.95
> > 16384 (128Mbit/s)                  : 4.08
> > 32768 (256Mbit/s)                  : 3.94
> > 65536 (512Mbit/s)                  : 3.23
> > 
> > The outbound traffic goes down unreasonably and is even not controled.
> > 
> > The cause of this problem is too large mtu value in "tc filter" command run by
> > libvirt. The command uses burst value to set mtu and the burst is equal to
> > average rate value if it's not set. This value is too large. For example
> > if the average rate is set to 1024 kilobytes/s, the mtu value is set to 1024
> > kilobytes. That's too large compared to the size of network packets.
> > Here libvirt applies tc ingress filter to Host's vnet(tun) device.
> > Tc ingress filter is implemented with TBF(Token Buckets Filter) algorithm. TBF
> > uses mtu value to calculate the amount of token consumed by each packet. With too
> > large mtu value, the token consumption rate is set too large. This leads to
> > token starvation and deterioration of TCP throughput.
> > 
> > Then, should we use the default mtu value 2 kilobytes?
> > The anser is No, because Guest with virtio-net device uses 65536 bytes
> > as mtu to transmit packets to Host, and the tc filter with the default mtu
> > value 2k drops packets whose size is larger than 2k. So, the most packets
> > is droped and again leads to deterioration of TCP throughput.
> > 
> > The appropriate mtu value is 65536 bytes which is equal to the maximum value
> > of network interface device defined in <linux/netdevice.h>. The value is
> > not so large that it causes token starvation and not so small that it
> > drops most packets.
> > Therefore this patch set the mtu value to 64kb(== 65535 bytes).
> > 
> > Again, here are the outbound QoS parameters and the TCP throughput with
> > the libvirt patched.
> > 
> > outbound average rate[kilobytes/s] : Guest to Host throughput[Mbit/s]
> > ======================================================================
> > 1024  (8Mbit/s)                    : 8.22
> > 2048  (16Mbit/s)                   : 16.42
> > 4096  (32Mbit/s)                   : 32.93
> > 8192  (64Mbit/s)                   : 66.85
> > 16384 (128Mbit/s)                  : 133.88
> > 32768 (256Mbit/s)                  : 271.01
> > 65536 (512Mbit/s)                  : 547.32
> > 
> > The outbound traffic conforms to the given limit.
> > 
> > Thank you,
> > 
> > Signed-off-by: Eiichi Tsukata <eiichi.tsukata.xh at hitachi.com>
> > ---
> >  AUTHORS                       |    1 +
> >  src/util/virnetdevbandwidth.c |    2 +-
> >  2 files changed, 2 insertions(+), 1 deletions(-)
> > 
> > diff --git a/AUTHORS b/AUTHORS
> > index b876ee6..375db24 100644
> > --- a/AUTHORS
> > +++ b/AUTHORS
> > @@ -246,6 +246,7 @@ Patches have also been contributed by:
> >    Gerd Hoffmann        <kraxel at redhat.com>
> >    Viktor Mihajlovski   <mihajlov at linux.vnet.ibm.com>
> >    Thang Pham           <thang.pham at us.ibm.com>
> > +  Eiichi Tsukata       <eiichi.tsukata.xh at hitachi.com>
> > 
> >    [....send patches to get your name here....]
> > 
> > diff --git a/src/util/virnetdevbandwidth.c b/src/util/virnetdevbandwidth.c
> > index b9bd2e3..93df5c1 100644
> > --- a/src/util/virnetdevbandwidth.c
> > +++ b/src/util/virnetdevbandwidth.c
> > @@ -136,7 +136,7 @@ virNetDevBandwidthSet(const char *ifname,
> >          virCommandAddArgList(cmd, "filter", "add", "dev", ifname, "parent",
> >                               "ffff:", "protocol", "ip", "u32", "match", "ip",
> >                               "src", "0.0.0.0/0", "police", "rate", average,
> > -                             "burst", burst, "mtu", burst, "drop", "flowid",
> > +                             "burst", burst, "mtu", "64kb", "drop", "flowid",
> >                               ":1", NULL);
> > 
> >          if (virCommandRun(cmd, NULL) < 0)
> > 
> 
> Yeah, this tc magic was the hardest part when I've introduced QoS
> feature. I tested it on small values where this has not fully showed.
> 
> ACK and pushed. Thanks!

  And special thanks for a very detailed explanation :-), excellent !

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel at veillard.com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/