[libvirt-users] VF MAC not reverted to all zero MAC/domain xml MAC on VM restart

Sat May 5 01:47:39 UTC 2018

On 05/04/2018 06:25 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL
TECHNOLOGIES LIMITED at Cisco) wrote:
>
> Hi Laine,
>
>  
>
> Thanks for taking the time to respond to my question. I think I have
> not described my problem clearly.
>
>  
>
> Let me explain my issue below with the information that you had requested.
>
>  
>
> My assumption according to the information you gave me is that the
> admin MAC and VF MAC are the same in my case.
>
> I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as
> you mentioned in your email.
>

If the VF has no separate netdev visible on the host, then it is not
being bound to the VF net driver for some reason. This should cause no
ill effect - it just means there is no "original" VF mac to restore.

>  
>
> * Given this assumption, when the host is booted, the admin MAC and VF
> MAC are both*00:00:00:00:00:00. *
>

Well, if there is no VF netdev on the host, there is no "vf mac" (and
the igb driver wouldn't allow it to be 00:00:00:00:00:00). But again,
not important

> **
>
> [root at nfvis ~]# ip link show GE0-0
>

"GEO-0"? That's not a normal name for an igb PF. Are you manually
selecting this name?

> 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
> <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
> <tel:1000>
>
> link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
>
> vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
>
> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
>
> [root at nfvis ~]#
>
>  
>
> * the VF is assigned to a guest, so the admin MAC is set to the
> configured value *(52:54:00:29:3c:bf) *as shown below
>
> [root at nfvis ~]# ip link show GE0-0
>
> 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
> <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
> <tel:1000>
>
> link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
>
> vf 0 MAC *52:54:00:29:3c:bf*, spoof checking on, link-state auto, trust on
>
> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
>
> [root at nfvis ~]#
>
>  
>
> * Configure bond interface on the guest and added member interface to
> the bond
>
>    On doing the above step, bond interface mac*(52:54:00:29:3c:be)*
> gets assigned to VF
>
> [root at nfvis libvirt]# ip link show GE0-0
>
> 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
> <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
> <tel:1000>
>
> link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
>
> vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, trust on
>
> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
>
> [root at nfvis libvirt]#
>

Okay, this part makes no sense, for 2 reasons:

1) once the admin MAC has been set by the host, a flag is set in the PF
driver marking this VF as having an "administratively set" MAC (that's
why I call it the "admin mac"), and after that point it should not be
possible for a guest to modify the MAC address. If your guest is
successfully setting the MAC of the VF, then either you don't have a
device that uses the igb driver, or there is a bug in the driver.

2) setting the MAC address of the VF shouldn't update the admin mac for
that VF - they are separate entities, and only synched up when the VF
driver is reloaded (and in that case it is the *admin mac* that is used
as the master copy to set both of them)

>  
>
> * The next step would be to restart the VM from within the VM console
> (* we are not doing a shut down and start from the host *).
>

Okay, so that reloads the VF driver in the guest, which would set it to
the admin mac (which you have said showed to be ....:be just before the
reset.

>    At this stage, the admin MAC is not reverted to the domain xml
> mac*(52:54:00:29:3c:bf)*
>
>    Instead it retains the same bond mac *(52:54:00:29:3c:be)* which
> was set before the VM was restarted from the console. **
>
>  
>
> [root at nfvis libvirt]# ip link show GE0-0
>
> 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216
> <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
> <tel:1000>
>
> link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff
>
> vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, trust on
>
> vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on
>
>  
>
> So the actual problem we face here, is after the VM is restarted and
> after the members of bond interface are released, VF mac is not
> getting reverted back to domain xml mac(*52:54:00:29:3c:bf*).
>
> Instead the VF mac and admin mac are set to the bond mac
> “*52:54:00:29:3c:be*” and leading to a duplicate mac issue.
>
> My expectation was that after the VM is restarted from the console,
> libvirt should revert the VF’s MAC to the original MAC,
>
> which in our case was the domain xml MAC, which is not happening in my
> case.
>

My expectation was that the guest would not be allowed to modify the MAC
address of the VF at all, and certainly that even if a change to the
guest MAC was allowed, that it wouldn't propagate to the PF's list of
admin mac addresses for the VFs. So we've both been disappointed :-)

Your VF and PF drivers are behaving strangely. But if you completely
shutdown the guest, then restart it, you should get the original
"...:bf" back. Even if that works for you, I would not count on the
ability of the guest to modify the VF's MAC address - that is not the
defined behavior and thus is likely to change in the future.

>  
>
>  
>
> Attached is the log file as requested.
>
>  
>
> *_Interface related contents from domain xml: _*
>
> <interface type='hostdev' managed='yes'>
>
> <mac address='52:54:00:29:3c:bf'/>
>
> <driver name='vfio'/>
>
> <source>
>
> <address type='pci' domain='0x0000' bus='0x02' slot='0x10'
> function='0x0'/>
>
> </source>
>
> <target dev='vnic1'/>
>
> <model type='virtio'/>
>
> <alias name='hostdev0'/>
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>
> </interface>
>
>  
>
> Thanks
>
> Thiru.
>
> *From:*sendmail <justsendmailnothingelse at gmail.com> *On Behalf Of
> *Laine Stump
> *Sent:* Wednesday, May 2, 2018 10:01 AM
> *To:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES
> LIMITED at Cisco) <tsengalv at cisco.com>
> *Cc:* Chanda Mendon (cmendon) <cmendon at cisco.com>
> *Subject:* Re: VF MAC not reverted to all zero MAC on VM restart
>
>  
>
> On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv -
> HCL TECHNOLOGIES LIMITED at Cisco) wrote:
>
>     Hi Laine Stump,
>
>      
>
>     We are running linux based VM on KVM based Hypervisor and facing
>     issue with respect to VF MAC.
>
>     Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and VF
>     driver IGBVF version 2.3.7.1
>
>      
>
>     *_Description of problem:_*
>
>     When passing a VF to a guest, libvirt sets its MAC according to
>     the domain xml.
>
>     On shutting down the VM or power off the VM, libvirt attempts to
>     restore its original MAC(all-zero mac). There is no issue with
>     this scenario.
>
>     *When we restart the VM, there is no attempt to restore/revert its
>     original MAC(all zero mac). *
>
>     This problem leads to duplicate MAC issue on the guest (if
>     configured with portchannel).
>
>      
>
>     Could you please help us in resolving the issue?
>
>
> Questions like this should be sent to libvirt-users at redhat.com
> <mailto:libvirt-users at redhat.com>rather than to an individual's email
> address. This makes the likelyhood of getting an answer much higher,
> and also creates an archive of the problem and eventual solution,
> which may help others in the future. I'm Cc'ing this response to that
> list so that any further communication will happen there.
>
> Before reading any of the following, it's useful to know this - for
> SRIOV network devices that *properly* support SRIOV (and I consider
> those using the igb driver to be in this category) each VF has two MAC
> addresses that need to be discussed:
>
> 1) the VF netdev MAC address (which I will just call the "mac address"
> - this is the MAC that is known to the VF net driver, and displayed in
> the output of "ip link show dev $vfnetdev".
>
> 2) the MAC address that is stored in the PF driver for each VF,
> displayed in the "VF" lines immediately following the "ip link" info
> of the *PF*, and which will be used to initialize the VF's own MAC
> address when its driver is re-initialized (I will call this the "admin
> mac")
>
> Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the
> Cisco enic driver) and some initialize it to a random number (e.g. igb
> and all the other Intel VF net drivers). Likewise, some PF drivers
> initialize the admin macs to 00:00:00:00:00:00 and some to random
> numbers (as a matter of fact, this behavior changes between different
> versions of the same driver in at least one case!).
>
> Beyond this, some PF and VF drivers allow setting the mac/admin mac to
> 00:00:00:00:00:00, and some prohibit one or the other. (in many cases,
> a driver that itself initializes the mac/admin mac to
> 00:00:00:00:00:00 also prohibits setting it back to that same value
> once it has been changed!)
>
> When libvirt sets up a VF to be used by a guest (either for vfio
> device assignment, or for macvtap passthrough), it saves both the mac
> and the admin mac with the intent of restoring them later.
>
> Once these values have been saved, libvirt will set the mac (in the
> case of macvtap passthrough) or the admin mac (in the case of vfio
> device assignment), and send the device on to qemu to be used by the
> guest.
>
> After the guest is finished running (or the device is detached),
> libvirt attempts to restore the settings it had saved at the
> beginning. Trouble comes up in 2 ways though - 1) once the admin mac
> has been set for a VF (via the PF), it is no longer possible to
> directly set the mac (in the VF) (this is done for security reasons,
> to prevent the guest from changing its mac address), and 2) as said
> above, many drivers don't allow setting 00:00:00:00:00:00, but in many
> cases that was the original setting.
>
> As of libvirt 3.2.0, libvirt uses the following strategies to work
> around these problems:
>
> a) if setting a non-0 mac (to the VF) fails, then libvirt will try
> setting the *admin MAC* to that value, then detaching/re-attaching the
> VF net driver. This will cause the PF to reinitialize the mac in the
> VF driver. (NB: if we do this, then we don't bother trying to later
> re-set the admin MAC to its own original value; it's really not
> important).
>
> b) if the failure was in setting the MAC to 00:00:00:00:00:00, then
> libvirt will set the mac/admin mac to 02:00:00:00:00:00 (which should
> be a legal value for any driver, but also should not conflict with and
> "real" mac on the network).
>
> Okay, enough background. On to your problem.
>
>
> To make sure I understand your situation correctly:
>
> * When the host is booted, the admin MAC is 00:00:00:00:00:00, and VF
> mac is random (call it rr:rr:rr:rr:rr:rr).
>
> * the VF is assigned to a guest (with <interface type='hostdev'> I
> guess?), so the admin MAC is set to the configured value (let's call
> it gg:gg:gg:gg:gg:gg)
>
> * the guest is shutdown, which causes *both* mac and admin mac to be
> set to rr:rr:rr:rr:rr:rr
>
> * the next time the guest is start *its mac is not changed*?
>
> There must be something I'm not getting. Can you maybe perform this
> test and run "ip link show $PF; ip link show $VF" (substituting the
> netdev names of the PF and VF for $PF and $VF) after each step to
> illustrate the problem? Also, it would be helpful to add this to
> /etc/libvirt/libvirtd.conf:
>
>   log_filters="1:util.netdev"
>   log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log
> <file://var/log/libvirt/libvirtd-netdev.log>"
>
> then restart libvirtd (systemctl restart libvirtd.service), and post
> the contents of libvirtd-netdev.log somewhere and reference it in your
> next reply. (don't forget to remove those lines and restart libvirtd
> again after your testing is done!)
>
> Finally, you should include the contents of the <interface ...>
> section of your config in your next response.
>
> (BTW, since you're using libvirt-3.2.0, I assume that you are using
> either RHEL 7.4 or CentOS 7.4. If you are using RHEL, you should open
> a customer case with Red Hat so that your problem is appropriately
> prioritized).
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180504/8bde190c/attachment.htm>