[Rdo-list] Jumbo MTU to instances in Kilo?
Erich Weiler
weiler at soe.ucsc.edu
Wed Oct 7 23:35:49 UTC 2015
Actually I think I'm closer - on the compute nodes, I set this in nova.conf:
network_device_mtu=9000
even though there was a big note above it that said not to use it
because this option was deprecated. But after setting that option, and
restarting nova and openvswitch, br-int, my tap device and my qvb device
all got set to MTU=9000. So I'm closer! But still one item is blocking
me. I show this tracepath from my controller node direct to the VM
(which is on a compute node on the local network):
# tracepath 10.50.100.4
1?: [LOCALHOST] pmtu 9000
1: 10.50.100.4 0.682ms
1: 10.50.100.4 0.241ms
2: 10.50.100.4 0.297ms pmtu
1500
2: 10.50.100.4 1.664ms reached
10.50.100.4 is the VM. It looks like the path is jumbo clean up until
that third hop. But the thing is, I don't know what the third hop is. ;)
On my compute node I still see some stuff with MTU=1500, but I'm not
sure if one of those is blocking me:
# ifconfig
br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
RX packets 2401498 bytes 359284253 (342.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 30 bytes 1572 (1.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid 0x20<link>
ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet)
RX packets 133 bytes 12934 (12.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet)
RX packets 165957142 bytes 20333410092 (18.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23299881 bytes 5950708819 (5.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255
inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
RX packets 6014767 bytes 813880745 (776.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79301 bytes 19052451 (18.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 22462729 bytes 1202484822 (1.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22462729 bytes 1202484822 (1.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::4c1a:55ff:feba:14c3 prefixlen 64 scopeid 0x20<link>
ether 56:a6:a6:db:83:c4 txqueuelen 0 (Ethernet)
RX packets 16 bytes 1520 (1.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid 0x20<link>
ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet)
RX packets 15 bytes 1456 (1.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
inet6 fe80::54a6:a6ff:fedb:83c4 prefixlen 64 scopeid 0x20<link>
ether 56:a6:a6:db:83:c4 txqueuelen 1000 (Ethernet)
RX packets 86 bytes 9610 (9.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 133 bytes 12767 (12.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid 0x20<link>
ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet)
RX packets 377 bytes 57664 (56.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 333 bytes 38765 (37.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
inet6 fe80::b44a:bff:fe72:aaea prefixlen 64 scopeid 0x20<link>
ether b6:4a:0b:72:aa:ea txqueuelen 1000 (Ethernet)
RX packets 133 bytes 12767 (12.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 86 bytes 9610 (9.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid 0x20<link>
ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet)
RX packets 333 bytes 38765 (37.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 377 bytes 57664 (56.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::fc16:3eff:fefa:9945 prefixlen 64 scopeid 0x20<link>
ether fe:16:3e:fa:99:45 txqueuelen 500 (Ethernet)
RX packets 118 bytes 11561 (11.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 95 bytes 10316 (10.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast
192.168.122.255
ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
My network node has all interfaces set to MTU=9000. I thought maybe the
bottleneck might be there but I don't think it is. Here's ifconfig
from my network node:
# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 2042 bytes 238727 (233.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2042 bytes 238727 (233.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::207:43ff:fe10:deb8 prefixlen 64 scopeid 0x20<link>
ether 00:07:43:10:de:b8 txqueuelen 1000 (Ethernet)
RX packets 2156053308 bytes 325330839639 (302.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 223004 bytes 24769304 (23.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 72
p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.50.1.51 netmask 255.255.0.0 broadcast 10.50.255.255
inet6 fe80::260:ddff:fe44:2aea prefixlen 64 scopeid 0x20<link>
ether 00:60:dd:44:2a:ea txqueuelen 1000 (Ethernet)
RX packets 49352916 bytes 3501547231 (3.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18876911 bytes 3768900461 (3.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::260:ddff:fe44:2aeb prefixlen 64 scopeid 0x20<link>
ether 00:60:dd:44:2a:eb txqueuelen 1000 (Ethernet)
RX packets 2491224974 bytes 348058319500 (324.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1597 bytes 204525 (199.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Any way I can figure out what the third hop is from my tracepath?
Thanks as always for the sage advice!
-erich
On 10/07/2015 09:57 AM, Erich Weiler wrote:
> Yeah, I made the changes and then recreated all the networks. For some
> reason br-int and the individual virtual instance interfaces on the
> compute node still show 1500 byte frames.
>
> Has anyone else configured jumbo frames in a Kilo environment? Or maybe
> I'm just an outlier... ;)
>
> -erich
>
> On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote:
>> Hi Erich,
>>
>> did you recreate the neutron networks after the configuration changes?
>>
>> Pedro Navarro Pérez
>> OpenStack product specialist
>> Red Hat Iberia
>> Passeig de Gràcia 120,
>> 08008 Barcelona
>> Spain
>> M +34 639 642 379
>> E pnavarro at redhat.com
>>
>> ----- Original Message -----
>> From: "Erich Weiler" <weiler at soe.ucsc.edu>
>> To: rdo-list at redhat.com
>> Sent: Wednesday, 7 October, 2015 2:34:28 AM
>> Subject: [Rdo-list] Jumbo MTU to instances in Kilo?
>>
>> Hi Y'all,
>>
>> I know someone must have figured this one out, but I can't seem to get
>> 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes have
>> MTU=9000 on their interfaces, so does the network node. dnsmasq also is
>> configured to set MTU=9000 on instances, which works. But I still can't
>> ping with large packets to my instance:
>>
>> [weiler at stacker ~]$ ping 10.50.100.2
>> PING 10.50.100.2 (10.50.100.2) 56(84) bytes of data.
>> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms
>> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms
>> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms
>>
>> That works fine. This however doesn't work:
>>
>> [root at stacker ~]# ping -M do -s 8000 10.50.100.2
>> PING 10.50.100.2 (10.50.100.2) 8000(8028) bytes of data.
>> From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)
>> ping: local error: Message too long, mtu=1500
>> ping: local error: Message too long, mtu=1500
>> ping: local error: Message too long, mtu=1500
>> ping: local error: Message too long, mtu=1500
>>
>> It looks like somehow the br-int interface for OVS isn't set at 9000,
>> but I can't figure out how to do that...
>>
>> Here's ifconfig on my compute node:
>>
>> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid
>> 0x20<link>
>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
>> RX packets 2401432 bytes 359276713 (342.6 MiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 30 bytes 1572 (1.5 KiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid
>> 0x20<link>
>> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet)
>> RX packets 69 bytes 6866 (6.7 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 8 bytes 648 (648.0 B)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid
>> 0x20<link>
>> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet)
>> RX packets 130174458 bytes 15334807929 (14.2 GiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 22919305 bytes 5859090420 (5.4 GiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
>> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255
>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid
>> 0x20<link>
>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
>> RX packets 38429352 bytes 5152853436 (4.7 GiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 419842 bytes 101161981 (96.4 MiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
>> inet 127.0.0.1 netmask 255.0.0.0
>> inet6 ::1 prefixlen 128 scopeid 0x10<host>
>> loop txqueuelen 0 (Local Loopback)
>> RX packets 22141566 bytes 1185622090 (1.1 GiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 22141566 bytes 1185622090 (1.1 GiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>> inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid
>> 0x20<link>
>> ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet)
>> RX packets 16 bytes 1472 (1.4 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 8 bytes 648 (648.0 B)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid
>> 0x20<link>
>> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet)
>> RX packets 15 bytes 1456 (1.4 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 8 bytes 648 (648.0 B)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>> inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid
>> 0x20<link>
>> ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet)
>> RX packets 247 bytes 28323 (27.6 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 233 bytes 25355 (24.7 KiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid
>> 0x20<link>
>> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet)
>> RX packets 377 bytes 57664 (56.3 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 333 bytes 38765 (37.8 KiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>> inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid
>> 0x20<link>
>> ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet)
>> RX packets 233 bytes 25355 (24.7 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 247 bytes 28323 (27.6 KiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid
>> 0x20<link>
>> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet)
>> RX packets 333 bytes 38765 (37.8 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 377 bytes 57664 (56.3 KiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>> inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid
>> 0x20<link>
>> ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet)
>> RX packets 219 bytes 24239 (23.6 KiB)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 224 bytes 26661 (26.0 KiB)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
>> inet 192.168.122.1 netmask 255.255.255.0 broadcast
>> 192.168.122.255
>> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet)
>> RX packets 0 bytes 0 (0.0 B)
>> RX errors 0 dropped 0 overruns 0 frame 0
>> TX packets 0 bytes 0 (0.0 B)
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>> This is on RHEL 7.1. Any obvious way I can get all the intermediate
>> bridges to MTU=9000? I've RTFM'd and googled to no avail...
>>
>> Here's the ovs-vsctl outout:
>>
>> [root at node-136 ~]# ovs-vsctl show
>> 6f5a5f00-59e2-4420-aeaf-7ad464ead232
>> Bridge br-int
>> fail_mode: secure
>> Port br-int
>> Interface br-int
>> type: internal
>> Port "qvo247da3ed-a4"
>> tag: 1
>> Interface "qvo247da3ed-a4"
>> Port "int-br-eth1"
>> Interface "int-br-eth1"
>> Port "int-br-enp3s0f0"
>> Interface "int-br-enp3s0f0"
>> type: patch
>> options: {peer="phy-br-enp3s0f0"}
>> Bridge "br-enp3s0f0"
>> Port "enp3s0f0"
>> Interface "enp3s0f0"
>> Port "br-enp3s0f0"
>> Interface "br-enp3s0f0"
>> type: internal
>> Port "phy-br-enp3s0f0"
>> Interface "phy-br-enp3s0f0"
>> type: patch
>> options: {peer="int-br-enp3s0f0"}
>> ovs_version: "2.3.1"
>>
>> Many thanks if anyone has any information on this topic! Or can point
>> me to some documentation I missed...
>>
>> Thanks,
>> erich
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
More information about the rdo-list
mailing list