[libvirt-users] How can a bridge be optimized?

[Laine Stump]

On 06/11/2012 11:34 AM, Whit Blauvelt wrote:
> On Sat, Jun 02, 2012 at 11:43:34PM -0400, Andrew Cathrow wrote:
>
>>> On Jun 1, 2012 2:17 PM, "Whit Blauvelt" < whit.virt at transpect.com >
>>> wrote:
>>>> I recall the kernel recently adding a new alternative for handling VM
>>>> interfaces, but can't recall what it was called. Would that be a
>>>> marked improvement? Is it supported by libvirt?
>>> Linux Bridge < macvtap < SR-IOV
>>
>> or openvswitch
> Looking into background info on these, it looks like SR-IOV capability is
> specific to certain NICs and not documented for KVM/libvirt in any obvious
> place. Maybe I just didn't find it.

In a later message you indicate that your host has an Intel 82576 NIC.
That *is* one of the few that has SR-IOV capabilities - each of its
ports has up to 7 Virtual Functions (VF) which can each be directly
assigned to a guest. The problem with this in the version of libvirt you
use (0.8.3, which is very dated) is that you can only attach the device
using <hostdev>, which doesn't support setting the MAC address of the
device before assigning it to the guest; since SR-IOV VFs are each given
new random MAC addresses on each reboot of the host, this means your
guest will not be see a stable MAC address, and will thus believe that a
new network device has been added after each host reboot. You can solve
this with a short script to set the VFs' MAC addresses at host boot
time, or if you can upgrade to libvirt-0.9.11 or later, you can use
<interface type='hostdev'> to attach SR-IOV VFs to your guests:

 
http://wiki.libvirt.org/page/Networking#PCI_Passthrough_of_host_network_devices

(it also points to a page that explains how to use <hostdev> to attach
an SR-IOV (or non-SR-IOV) NIC to a guest.

> Openvswitch looks more widely useable and promising, but also lacking
> user-level documentation. Openvswitch.org's "Documentation" section has
> three brief notes, all presuming you've already got it in use. 


openvswitch itself is a fairly new project, and its support in libvirt
is even newer. At this point, the libvirt support requires that an
openvswitch bridge is already configured and running on the host. (I'm
sure openvswitch would be happy to receive updates/additions to their
documentation :-)


Of course, since you're using libvirt-0.8.3, you wouldn't be able to
easily use openvswitch with your guests anyway (and since you are using
kernel-2.6.35 you wouldn't have openvswitch support in the kernel
anyway, unless you add it yourself).


>
> Okay, so falling back to macvtap, a long post from over a year ago at 
>
>      http://ubuntuforums.org/showthread.php?t=1687750
>
> says that the host cannot communicate directly with the guests through the
> bridge when using it. Is this correct? That would rule it out for my use.

Yes, that is true. There is a way around it, though - add a 2nd
interface to each guest that is connected to an isolated bridge:

 
http://wiki.libvirt.org/page/Guest_can_reach_outside_network%2C_but_can%27t_reach_host_%28macvtap%29


I'm not really sure how much performance you would gain from switch to
macvtap though (I have heard of someone who improved lxc guest network
performance from 4Gb/sec to 9Gb/sec by using macvtap to connect to a
vlan interface (which I previously didn't even know was supported!), but
that situation is very different from yours, and frankly the low numbers
you give make it sound like you have some other basic problem that won't
be fixed by switching to a different connection type.

(Another thing to think about with macvtap is that it was only added in
kernel-2.6.34, so the version in your kernel probably still has bugs.)

> So now I fall back to the question of whether I should enable virtio where
> I've currently got (working, but slowish) bridges defined.

Yes. Any time the guest supports virtio-net, use it (unless you are
using PCI passthrough to directly use the physical device).

Note that the use of the guest virtio driver is completely orthogonal to
whether you use macvtap, a host bridge, or a libvirt-managed virtual
network

> (I've got virtio
> in use for memballoon, but not for the bridges.) I look at
> http://wiki.libvirt.org/page/Virtio and while helpful it underspecifies:
>
>     In the <interface> section, add a virtio model, like this:
>
>     <interface type='network'>
>       ...
>       <model type='virtio' />
>     </interface>
>
> That's all it says. So ... I've got entries that currently look about like:
>
>     <interface type='bridge'>
>       <mac address='52:54:00:25:0a:a2'/>
>       <source bridge='br0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>     </interface>
>
> What if anything should carry across to the elipsis in the doc?

That <interface> definition will use the default model for the
hypervisor, which in this case is rtl8139. To change an existing
interface definition to use a different model, just add the <model ... >
line and leave the rest of the <interface> section as it is. If you are
adding a new interface, you don't need to add a <mac address=.../> or
<adress type='pci' .../> line, as those are automatically generated and
added for you (in particular, don't add the <address> line, because each
guest device needs to have a unique address, and there are some odd
rules about placement of some devices that libvirt knows about and
accounts for).

> The doc at
> http://www.linux-kvm.org/page/Tuning_KVM is similarly terse, and puts it in
> terms of the qemu command line, rather than libvirt XML, so not completely
> helpful short of studying how libvirt translates the XML to that line. Ah,
> here's a more useful doc at
> http://wiki.libvirt.org/page/Networking#Guest_configuration_2:
>
>     Guest configuration
>
>     In order to let your virtual machines use this bridge, their configuration
>     should include the interface definition as described in Bridge to LAN. In
>     essence you are specifying the bridge name to connect to. Assuming a shared
>     physical device where the bridge is called "br0", the following guest XML
>     would be used:
>
>      <interface type='bridge'>
>         <source bridge='br0'/>
>         <mac address='00:16:3e:1a:b3:4a'/>
>         <model type='virtio'/>   # try this if you experience problems with VLANs
>      </interface>
>
> So the <model type='virtio'/> can just go under <interface type='bridge'>,
> that doesn't need to change to <interface type='network'>.

Right. Those two things are orthogonal. <model type='xxx'/> is defining
what kind of device to present to the guest. The type='nnn' attribute
directly in <interface> is defining how to connect the guest's network
data stream to the physical network.


>  Then should the
> current line like:
>
>     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>
> stay or go? 


If this line is already there, don't remove it. If there isn't one,
don't add it - an address appropriate for the situation will be
automatically added.


>
> Do others share my perception that it's a shame there's so little interest
> in writing good docs in this whole area?


As much documentation as exists, there could still be a lot more, and it
could stand to be better organized. We welcome any contributions to the
libvirt wiki.

Some specific resources that may be helpful to you:

 http://www.libvirt.org/formatdomain.html#elementsNICS

(This is (or should be) a full reference of every option supported in
<interface> definitions. Pay close attention to all of the "since x.x.x"
tags - many features have been added to libvirt since 0.8.3, so a lot of
what's there won't be supported by your host's libvirt anyway.)

  http://wiki.libvirt.org/page/Troubleshooting

(in particular, there are several items in the middle of the list
specifically dealing with networking problems)

  http://wiki.libvirt.org/page/Networking

(I think you've already found that one).