[libvirt] problem with outbound limiting

[Stefan Berger]

On 08/11/2011 08:07 AM, Upendra Moturi wrote:
> Thanks for your reply.
>
> The patch it appears still has problem.
> I have changed only mtu value as 2kb instead of burst value in the
> network.c file
>
> With average as 2048 , burst as 2048 and peak 2048
> scp copied at 30-40KB/s
All test I did were with the patch applied. I saw the throughput 
collapsing yesterday at 2Mb mtu.

I generated a file using

dd if=/dev/urandom of=testfile bs=1024 count=409600

I did the same test you did with 2048 without specifying a specific 
model of ethernet card and reach 1.9MB/s using scp'ing towards remote 
machine. When specifying <model type='virtio'/> I also get 1.9MB/s as 
result.
> With average as 4096 , burst as 4096 and peak 4096
> scp copied at same 30-40KB/s
>
With these numbers and using the virtio model I ended up getting a 
throughput of 3.8Mb/s.

The host is a 3.0.0 Linux kernel. Here's the command line output of the 
tc command inspecting the tap interface of the VM (vnet0).

# tc filter show dev vnet0 root
filter parent ffff: protocol ip pref 49152 u32
filter parent ffff: protocol ip pref 49152 u32 fh 800: ht divisor 1
filter parent ffff: protocol ip pref 49152 u32 fh 800::800 order 2048 
key ht 800 bkt 0 flowid :1
   match 00000000/00000000 at 12
  police 0x5b rate 32768Kbit burst 4Mb mtu 2Kb action drop overhead 0b
ref 1 bind 1


Maybe someone else can verify as well.

    Stefan

> where as earlier it was 1 MB/s.(without patch)
>
> On Wed, Aug 10, 2011 at 9:16 PM, Stefan Berger
> <stefanb at linux.vnet.ibm.com>  wrote:
>> On 08/10/2011 11:20 AM, Eric Blake wrote:
>>> On 08/10/2011 09:04 AM, Stefan Berger wrote:
>>>> On 08/10/2011 02:55 AM, Upendra Moturi wrote:
>>>>
>>>> Hi,
>>>>
>>>> does this patch solve your problem? I am setting the MTU to fixed 2kb.
>>>> Doing tests with scp seems to indicate that this improved the situation
>>>> -- at least for me.
>>>>
>>>> Stefan
>>>>
>>>> Signed-off-by: Stefan Berger<stefanb at linux.vnet.ibm.com>
>>>>
>>>> ---
>>>> src/util/network.c | 16 ++++++++--------
>>>> 1 file changed, 8 insertions(+), 8 deletions(-)
>>>>
>>>> Index: libvirt-acl/src/util/network.c
>>>> ===================================================================
>>>> --- libvirt-acl.orig/src/util/network.c
>>>> +++ libvirt-acl/src/util/network.c
>>>> @@ -1156,8 +1156,8 @@ virBandwidthEnable(virBandwidthPtr bandw
>>>>
>>>> virCommandFree(cmd);
>>>> cmd = virCommandNew(TC);
>>>> - virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent",
>>>> - "1:", "classid", "1:1", "htb", NULL);
>>>> + virCommandAddArgList(cmd,"class", "add", "dev", iface, "parent",
>>>> + "1:", "classid", "1:1", "htb", NULL);
>>>> virCommandAddArgList(cmd, "rate", average, NULL);
>>> This hunk was just indentation; you should push the whitespace changes as
>>> a separate commit (and can push that now, under the trivial rule, without
>>> needing further review).  [It doesn't help matters that thunderbird botched
>>> the whitespace in my reply]
>>>
>>>> @@ -1202,7 +1202,7 @@ virBandwidthEnable(virBandwidthPtr bandw
>>>> virCommandAddArgList(cmd, "filter", "add", "dev", iface, "parent",
>>>> "ffff:", "protocol", "ip", "u32", "match", "ip",
>>>> "src", "0.0.0.0/0", "police", "rate", average,
>>>> - "burst", burst, "mtu", burst, "drop", "flowid",
>>>> + "burst", burst, "mtu", "2kb", "drop", "flowid",
>>>> ":1", NULL);
>>> Here's the meat of the proposed patch.  Is 2kb always valid, or will we
>>> run into problems on NICs that insist on traditional limits near the 1518
>>> mark?  Should it be user-configurable?  I'm reluctant to ACK this without a
>>> bit more understanding of why 2kb works, as well as feedback on test results
>>> with the patch in place.
>>>
>> Yes, I am not an exper on tc. I have seen examples with the mtu being set to
>> '1' for dropping icmp traffic for example. So if a VM produces smaller than
>> 2kb packets, which on traditional networks it probably should, then the
>> packets should go through. We could also leave out the mtu parameter above
>> and it would go to 2kb by default. Check via
>>
>> tc filter show dev<ifname>  root
>>
>> The 2kb parameter doesn't explain why 2MB doesn't work, though.
>> Even if this wasn't the final fix, it at least 'improves the situation' (for
>> me).
>>
>>     Stefan
>>
>>
>>
>
>