[libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete

Moshe Levi moshele at mellanox.com
Wed Dec 16 14:20:24 UTC 2015 


> -----Original Message-----
> From: sendmail [mailto:justsendmailnothingelse at gmail.com] On Behalf Of
> Laine Stump
> Sent: Wednesday, December 16, 2015 12:31 AM
> To: Libvirt <libvir-list at redhat.com>
> Cc: vyasevic at redhat.com; Moshe Levi <moshele at mellanox.com>
> Subject: Re: [libvirt] <interface type='hostdev'>vf configuration cleanup when
> VM is delete
> 
> On 12/15/2015 04:12 PM, Vlad Yasevich wrote:
> > On 12/15/2015 02:45 PM, Laine Stump wrote:
> >> On 12/15/2015 01:34 PM, Laine Stump wrote:
> >>> On 12/13/2015 10:51 AM, Moshe Levi wrote:
> >>>> Hi,
> >>>>
> >>>> I have a setup with libvirt 1.3.0 and OpenStack trunk.
> >>>>
> >>>> Before launched the VM ip link command show the following VF
> >>>> mac/vlan configuration [1]
> >>>>
> >>>> When I launch a VM with <interface type='hostdev'> via openstack
> >>>> api (OpenStack direct
> >>>> port)
> >>>>
> >>>> I can see that the VF get the mac/vlan according to libvrit xml [2]
> >>>> and ip link command  [3], but when I delete the VM the mac/vlan
> >>>> config are still shown as in [3] and not restored to [1]
> >>>>
> >>>> Shouldn’t  libvirt restore the mac/vlan to [1].
> >>>>
> >>>> The same problem exists when using <interface type='direct'>
> >>>> (OpenStack macvtap port) but just for the MAC configuration of the VF.
> >>>>
> >>> What libvirt does is to restore the MAC address to whatever it was
> >>> before we set it up for use with a guest. Although there are some
> >>> sriov net drivers that (for some unfathomable reason) think it's
> >>> cool to assign a random MAC address to each VF at boot time, the
> "normal" thing is for the VFs to have a MAC address of all 0's to start with.
> >>> So libvirt should be saving 00:00:00:00:00:00 (it will be in the
> >>> file
> >>> /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC
> >>> to use; when done, libvirt will read the 00:00:00:00:00:00 and use
> >>> netlink to set the MAC address, but this is apparently failing.
> >>>
> >>> I checked on my Fedora 22 system with the igb driver, and found that
> >>> if the MAC address was originally set to something other than 0's,
> >>> it was restored properly by libvirt, but if it was set to all 0's originally, the
> attempt to set it back to 0 would fail.
> >>>
> >>> I then tried doing the same thing with the "ip" utility:
> >>>
> >>>      # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
> >>>
> >>> and I get the following response:
> >>>
> >>>      RTNETLINK answers: Invalid argument
> >>>
> >>> So it appears that either the kernel or the NIC driver is refusing
> >>> to set the MAC address to all 0's. I'm reasonably certain this is a
> >>> regression in the kernel,
> >> Sigh. It appears that this has "always" been the case - I just
> >> checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and
> >> 4.1 (Fedora 22) and both of them also refuse to set the MAC address
> >> to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or
> some basic code in the kernel.
> > It's considered to be an invalid address by is_valid_ether_addr() function.
> >
> > There appear to be different behaviour in some adapters. In current
> > upstream, it looks like a quarter of the VF capable drivers (bnxt,
> > enic, fm10k, sfc) permit VF mac setting of all zeros. The others
> > simply use is_valid_ether_addr function without specifically testing
> > for all-0.  I am not sure if this is HW related or simply oversights...  Might
> want to bringing this up on netdev.
> 
> Thanks, Vlad!
> 
> 
> Moshe,
> 
> It sounds like in your case it is caused by code in the mlx driver, so hopefully
> you can have some influence there. My path is a bit more difficult, as the
> failure on mine is in the igb driver :-)
Sure I will take it with Mellanox Driver team
Ok, just saw the latest thread mail you can discard my previous one
> 
> Sending a message to netdev is a good idea. It would be wonderful if all the
> vendors could agree to:
> 
> 1) initialize all VFs with a MAC address of 0
> 2) allow setting VF MAC address to 0 at any time.
> 
> I'll put that on my to-do list :-P


> 
> 
> > -vlad
> >
> >>
> >>> although I can't say how long it's been there, as I don't normally
> >>> pay attention to this (and as I said, many SRIOV NIC drivers don't
> >>> default their VFs to 0 MAC addresses)
> >>>
> >>> What distro and kernel are you using for your tests?
> >>>
> >>>
> >>>> [1]  - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> >>>> qdisc mq master ovs-system state UP mode DEFAULT group default qlen
> >>>> 1000
> >>>>
> >>>>      link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
> >>>>
> >>>>      vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>>      vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>>      vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>>      vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>> [2] - <interface type='hostdev' managed='yes'>
> >>>>
> >>>>    <mac address=' fa:16:3e:11:af:fe '/>
> >>>>
> >>>>    <driver name='kvm'/>
> >>>>
> >>>>    <source>
> >>>>
> >>>>      <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> >>>> function='0x7'/>
> >>>>
> >>>>    </source>
> >>>>
> >>>>    <vlan>
> >>>>
> >>>>      <tag id='190'/>
> >>>>
> >>>>    </vlan>
> >>>>
> >>>>    <alias name='hostdev0'/>
> >>>>
> >>>>    <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> >>>> function='0x0'/>
> >>>>
> >>>> </interface>
> >>>>
> >>>> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> qdisc
> >>>> mq master ovs-system state UP mode DEFAULT group default qlen 1000
> >>>>
> >>>>      link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
> >>>>
> >>>>      vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>>      vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>>      vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state
> >>>> auto
> >>>>
> >>>>      vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off,
> >>>> link-state enable
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> libvir-list mailing list
> >>>> libvir-list at redhat.com
> >>>> https://www.redhat.com/mailman/listinfo/libvir-list
> >>> F15
> >>>
> >>>
> >>> --
> >>> libvir-list mailing list
> >>> libvir-list at redhat.com
> >>> https://www.redhat.com/mailman/listinfo/libvir-list
> >>
> >

More information about the libvir-list mailing list