<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body bgcolor="white" lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> <o:p> </o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: sendmail [mailto:justsendmailnothingelse@gmail.com] On Behalf Of Laine Stump Sent: Saturday, September 03, 2016 10:50 PM To: Libvirt <libvir-list@redhat.com> Cc: Moshe Levi <moshele@mellanox.com>; Edan David <edand@mellanox.com> Subject: Re: [libvirt] <interface type='direct'><o:p></o:p> </div> </div> <o:p> </o:p> <div> On 09/03/2016 11:08 AM, Moshe Levi wrote:<o:p></o:p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt">-----Original Message-----<o:p></o:p></pre> <pre style="margin-left:15.75pt">From: sendmail [<a href="mailto:justsendmailnothingelse@gmail.com">mailto:justsendmailnothingelse@gmail.com</a>] On Behalf Of<o:p></o:p></pre> <pre style="margin-left:15.75pt">Laine Stump<o:p></o:p></pre> <pre style="margin-left:15.75pt">Sent: Thursday, September 01, 2016 5:59 PM<o:p></o:p></pre> <pre style="margin-left:15.75pt">To: Libvirt <a href="mailto:libvir-list@redhat.com"><libvir-list@redhat.com></a><o:p></o:p></pre> <pre style="margin-left:15.75pt">Cc: Moshe Levi <a href="mailto:moshele@mellanox.com"><moshele@mellanox.com></a>; Edan David<o:p></o:p></pre> <pre style="margin-left:15.75pt"><a href="mailto:edand@mellanox.com"><edand@mellanox.com></a><o:p></o:p></pre> <pre style="margin-left:15.75pt">Subject: Re: [libvirt] <interface type='direct'><o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">On 09/01/2016 04:05 AM, Moshe Levi wrote:<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt">Hi,<o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">In OpenStack we have a port type macvtap.<o:p></o:p></pre> <pre style="margin-left:15.75pt">Mavtap port is just a tap device connected to VF.<o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">In Libvirt the guest xml look like<o:p></o:p></pre> <pre style="margin-left:15.75pt"><interface type='direct'><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <mac address='fa:16:3e:b1:06:4e'/><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <source dev='p1p6' mode='passthrough'/><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <target dev='macvtap1'/><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <model type='virtio'/><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <driver name='vhost'/><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <alias name='net0'/><o:p></o:p></pre> <pre style="margin-left:15.75pt"> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'<o:p></o:p></pre> <pre style="margin-left:15.75pt">function='0x0'/> </interface><o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">In the hypervisor we can see that the mac of the VF which is<o:p></o:p></pre> <pre style="margin-left:15.75pt">fa:16:3e:f3:9b:e8 - is set by OpenStack see [1]<o:p></o:p></pre> <pre style="margin-left:15.75pt">9: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">master ovs-system state UP mode DEFAULT group default qlen 1000<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"> link/ether 7c:fe:90:29:24:4e brd ff:ff:ff:ff:ff:ff<o:p></o:p></pre> <pre style="margin-left:15.75pt"> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state disable<o:p></o:p></pre> <pre style="margin-left:15.75pt"> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state disable<o:p></o:p></pre> <pre style="margin-left:15.75pt"> vf 2 MAC fa:16:3e:f3:9b:e8, vlan 48, spoof checking on, link-state enable<o:p></o:p></pre> <pre style="margin-left:15.75pt"> vf 3 MAC fa:16:3e:f6:02:c8, vlan 48, spoof checking on,<o:p></o:p></pre> <pre style="margin-left:15.75pt">link-state enable<o:p></o:p></pre> <pre style="margin-left:15.75pt">41: ens3f4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">state UP mode DEFAULT group default qlen 1000<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"> link/ether fa:16:3e:f6:02:c8 brd ff:ff:ff:ff:ff:ff<o:p></o:p></pre> <pre style="margin-left:15.75pt">42: macvtap0@ens3f4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"> link/ether fa:16:3e:f6:02:c8, brd ff:ff:ff:ff:ff:ff<o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">The netdevice of the VF which is ens3f4 has also the same mac. This<o:p></o:p></pre> <pre style="margin-left:15.75pt">mac is set when using Libvirt 1.2.2 (Ubuntu 14.04), But when we tested<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">with new Libvirt versions >= 1.2.17 (Fedora 23/Ubuntu 16.04) the mac<o:p></o:p></pre> <pre style="margin-left:15.75pt">netdevice of the VF (ens3f4) is not set.<o:p></o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt">This change in Libvirt breaks the guest from getting DHCP in OpenStack.<o:p></o:p></pre> <pre style="margin-left:15.75pt">Do you know why the behavior change in newer releases?<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">The MAC address is now set with a netlink command to set the VFINFO of<o:p></o:p></pre> <pre style="margin-left:15.75pt">the particular VF# of the PF. This change was made in response to a bug<o:p></o:p></pre> <pre style="margin-left:15.75pt">report stating that once the MAC address had been set for a hostdev<o:p></o:p></pre> <pre style="margin-left:15.75pt">assignment of a VF (in which case this method is required), it was no longer<o:p></o:p></pre> <pre style="margin-left:15.75pt">possible to set the MAC address for macvtap passthrough (the VF driver<o:p></o:p></pre> <pre style="margin-left:15.75pt">would complain "MAC has been administratively set", on Intel igbvf at least). <o:p></o:p></pre> <pre style="margin-left:15.75pt">Unfortunately I recently found that when you set the MAC address in this<o:p></o:p></pre> <pre style="margin-left:15.75pt">manner, it doesn't take effect on the actual device<o:p></o:p></pre> <pre style="margin-left:15.75pt">- it's only saved in memory to be applied the *next time* the host driver is<o:p></o:p></pre> <pre style="margin-left:15.75pt">rebound to the VF.<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">Are saying that the change was to update the MAC of the VF? <o:p></o:p></pre> <pre style="margin-left:15.75pt">So I don’t understand how this effect the issue that VF netdevice MAC don't get set<o:p></o:p></pre> </blockquote> Look at the explanation in commit cb3fe38c and also <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1113474">https://bugzilla.redhat.com/show_bug.cgi?id=1113474</a><o:p></o:p> <o:p> </o:p> That commit switched from using a simple ioctl(SIOCSIFHWADDR) to the VF's netdev name, to using a netlink RTM_SETLINK message to the netdev of the *PF* for the given VF. This was done because the latter is the *only* way you can set the MAC address for a VF that you're going to assign to the guest with vfio device assignment, and once you've set the MAC address that way, future attempts to set the MAC address with ioctl(SIOCSIGHWADDR) result in failure and a kernel log like this: <o:p></o:p> <pre style="margin-left:15.75pt">kernel: igb 0000:0e:00.1: VF 1 attempted to override administratively set MAC address<o:p></o:p></pre> <pre style="margin-left:15.75pt">kernel: Reload the VF driver to resume operations<o:p></o:p></pre> Looking into the kernel, it appears that once the MAC address for a VF has been set via RTM_SETLINK, the igb driver (and I believe also the ixgbe driver, not sure about others) doesn't allow it to be changed via ioctl until the PF driver is reloaded (which can't realistically be done on an active system) But recently there was another report of the MAC address not getting set properly for macvtap passthrough mode when the device is an SRIOV VF (I can't find it in bugzilla, so it must have been an email to one of the lists) and when I tried it myself I found they were correct - in the output of "ip link show" the MAC address showed in the list of VFs under the PF is correctly modified, but it's not set properly in the VF's netdev instance - apparently the MAC addresses in the VF list aren't set in the VF's netdev immediately, they're just saved to be set *the next time the VF is re-bound to the VF netdev driver*. I think in the past the interface may have been in promiscuous mode so it didn't matter, but now it isn't? I'm not sure as I haven't had much time to investigate. Does that make any more sense now?<o:p></o:p> <o:p> </o:p> Yes Thanks J <o:p></o:p> <o:p> </o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt">Since I don't see a reasonably efficient way to get this to work, I need to<o:p></o:p></pre> <pre style="margin-left:15.75pt">make a patch to revert to the old behavior, and we'll then just have to tell<o:p></o:p></pre> <pre style="margin-left:15.75pt">people "If you do hostdev device assignment of VFs, then you can't later re-<o:p></o:p></pre> <pre style="margin-left:15.75pt">use the same device for macvtap passthrough mode".<o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">(actually, I *think* an alternative would be to unbind/rebind the host driver<o:p></o:p></pre> <pre style="margin-left:15.75pt">to the VF after setting the VF MAC address, but that seems a bit<o:p></o:p></pre> <pre style="margin-left:15.75pt">disruptive/extreme to work around a problem that is probably only seen in<o:p></o:p></pre> <pre style="margin-left:15.75pt">QE labs, but not in the real world (realistically, production systems likely use<o:p></o:p></pre> <pre style="margin-left:15.75pt">either hostdev or macvtap, and don't switch back and forth between them).<o:p></o:p></pre> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">A question - I notice you have the vlan set for the VF. Does *that* properly<o:p></o:p></pre> <pre style="margin-left:15.75pt">take effect? (it's set in the same manner as the MAC address, via a netlink<o:p></o:p></pre> <pre style="margin-left:15.75pt">command to set the VFINFO)<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">I am not sure what you mean, but we set the vlan in OpenStack after we create the guest xml.<o:p></o:p></pre> </blockquote> What command do you use to set it? Do you use "ip link set $PF vf $VF# vlan $VLANID" ? I think that's what it's showing here:<o:p></o:p> Yes we use ip link set $PF vlan $VLANID to set the vlan.<o:p></o:p> So in the guest xml we don’t put the vlan id for <interface type='direct'> only for interface type='hostdev' are you saying that is should be supported <o:p></o:p> In both? Should I open a bug for this as well? <o:p></o:p> <a href="https://review.openstack.org/#/c/364121/1/nova/network/linux_net.py">https://review.openstack.org/#/c/364121/1/nova/network/linux_net.py</a> (I don't know my way around openstack code, but arrived at that page via clicking on links from a google search) <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <pre style="margin-left:15.75pt">In OpenStack we put the MAC of the VF and the vlan using iproute2.<o:p></o:p></pre> <pre style="margin-left:15.75pt">I just want to know if that should be the part of Libvirt setting mac/vlan or <o:p></o:p></pre> <pre style="margin-left:15.75pt">Libvirt just create the macvtap interface and we should put the mac/vlan? <o:p></o:p></pre> </blockquote> libvirt *should* do it. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt"><o:p> </o:p></pre> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <pre style="margin-left:15.75pt">We have a WIP patch in OpenStack for setting also the mac for the<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">netdevice of the VF [2]. Just wanted to know that this is the correct<o:p></o:p></pre> <pre style="margin-left:15.75pt">approach.<o:p></o:p></pre> </blockquote> <pre style="margin-left:15.75pt">Can you confirm that setting the VF netdevice mac in OpenStack is a reasonable workaround for the newer Libvirt versions?<o:p></o:p></pre> </blockquote> If libvirt isn't getting the job done, and you can set it yourself, then that's a workaround. I don't know that I'd call it "reasaonable" though. If everybody puts in special code to workaround bugs in libvirt (which is apparently what's been done) rather than actually reporting the bug (what you're doing now - Thanks!) then we are tricked into thinking that either the code works, or that nobody is using it so it doesn't matter if it's broken.<o:p></o:p> Sure, I would like to get the fix in Libvirt so I opened this bug <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1372944">https://bugzilla.redhat.com/show_bug.cgi?id=1372944</a> <o:p></o:p> But then again I can’t assume that everyone will use the latest Libvirt so I will put a workaround in OpenStack with TODO for removal <o:p></o:p> The *best* way of overcoming this problem is to fix libvirt so it does what it's supposed to do. It's possible we can make it work by adding some operation after we send the RTM_SETLINK (maybe unbind the VF from its netdev driver, then re-bind, but that seems so drastic and time consuming!), or maybe we'll have to revert to using ioctl(SIOCSIFHWADDR), but of course that will fail if the interface has been used for hostdev assignment since the last host reboot. It's interesting that openstack is apparently using the RTM_SETLINK method to set the mac address (afaik, that's what is used by the "ip link set $pf_ifname vf $vf_num mac $mac_addr vlan $vlanid" command that's shown in the bit of code from nova/network/linux_net.py at the link I posted above).<o:p></o:p> </div> </div> </body> </html>