[libvirt] <interface type='hostdev'>vf configuration cleanup when VM is delete

[Vlad Yasevich]

On 12/15/2015 02:45 PM, Laine Stump wrote:
> On 12/15/2015 01:34 PM, Laine Stump wrote:
>> On 12/13/2015 10:51 AM, Moshe Levi wrote:
>>>
>>> Hi,
>>>
>>> I have a setup with libvirt 1.3.0 and OpenStack trunk.
>>>
>>> Before launched the VM ip link command show the following VF mac/vlan configuration [1]
>>>
>>> When I launch a VM with <interface type='hostdev'> via openstack api (OpenStack direct
>>> port)
>>>
>>> I can see that the VF get the mac/vlan according to libvrit xml [2] and ip link
>>> command  [3], but when I delete the VM the mac/vlan config are still shown as in [3]
>>> and not restored to [1]
>>>
>>> Shouldn’t  libvirt restore the mac/vlan to [1].
>>>
>>> The same problem exists when using <interface type='direct'> (OpenStack macvtap port) 
>>> but just for the MAC configuration of the VF.
>>>
>>
>> What libvirt does is to restore the MAC address to whatever it was before we set it up
>> for use with a guest. Although there are some sriov net drivers that (for some
>> unfathomable reason) think it's cool to assign a random MAC address to each VF at boot
>> time, the "normal" thing is for the VFs to have a MAC address of all 0's to start with.
>> So libvirt should be saving 00:00:00:00:00:00 (it will be in the file
>> /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC to use; when done,
>> libvirt will read the 00:00:00:00:00:00 and use netlink to set the MAC address, but this
>> is apparently failing.
>>
>> I checked on my Fedora 22 system with the igb driver, and found that if the MAC address
>> was originally set to something other than 0's, it was restored properly by libvirt, but
>> if it was set to all 0's originally, the attempt to set it back to 0 would fail.
>>
>> I then tried doing the same thing with the "ip" utility:
>>
>>     # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00
>>
>> and I get the following response:
>>
>>     RTNETLINK answers: Invalid argument
>>
>> So it appears that either the kernel or the NIC driver is refusing to set the MAC
>> address to all 0's. I'm reasonably certain this is a regression in the kernel,
> 
> Sigh. It appears that this has "always" been the case - I just checked on a 2.6.32-573
> RHEL kernel, and a 3.10.x RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also refuse
> to set the MAC address to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC
> driver or some basic code in the kernel.

It's considered to be an invalid address by is_valid_ether_addr() function.

There appear to be different behaviour in some adapters. In current upstream, it looks
like a quarter of the VF capable drivers (bnxt, enic, fm10k, sfc) permit VF mac setting of
all zeros. The others simply use is_valid_ether_addr function without specifically
testing for all-0.  I am not sure if this is HW related or simply oversights...  Might
want to bringing this up on netdev.

-vlad

> 
> 
>> although I can't say how long it's been there, as I don't normally pay attention to this
>> (and as I said, many SRIOV NIC drivers don't default their VFs to 0 MAC addresses)
>>
>> What distro and kernel are you using for your tests?
>>
>>
>>> [1]  - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
>>> ovs-system state UP mode DEFAULT group default qlen 1000
>>>
>>>     link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
>>>
>>>     vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>>     vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>>     vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>>     vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>> [2] - <interface type='hostdev' managed='yes'>
>>>
>>>   <mac address=' fa:16:3e:11:af:fe '/>
>>>
>>>   <driver name='kvm'/>
>>>
>>>   <source>
>>>
>>>     <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x7'/>
>>>
>>>   </source>
>>>
>>>   <vlan>
>>>
>>>     <tag id='190'/>
>>>
>>>   </vlan>
>>>
>>>   <alias name='hostdev0'/>
>>>
>>>   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
>>>
>>> </interface>
>>>
>>> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system
>>> state UP mode DEFAULT group default qlen 1000
>>>
>>>     link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff
>>>
>>>     vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>>     vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>>     vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto
>>>
>>>     vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, link-state enable
>>>
>>>
>>>
>>> -- 
>>> libvir-list mailing list
>>> libvir-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/libvir-list
>> F15
>>
>>
>> -- 
>> libvir-list mailing list
>> libvir-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/libvir-list
> 
>