<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 05/04/2018 06:25 PM, Thirunavukarasu
Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at Cisco)
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:13.5pt;
font-family:"Courier New";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">Hi Laine,<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">Thanks for taking the
time to respond to my question. I think I have not described
my problem clearly.<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">Let me explain my
issue below with the information that you had requested.<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">My assumption
according to the information you gave me is that the admin
MAC and VF MAC are the same in my case.
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">I see a PF (GE0-0)
interface but I don’t see a vfnetdev interface as you
mentioned in your email.</span></p>
</div>
</blockquote>
<br>
If the VF has no separate netdev visible on the host, then it is not
being bound to the VF net driver for some reason. This should cause
no ill effect - it just means there is no "original" VF mac to
restore.<br>
<br>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">* Given this
assumption, when the host is booted, the admin MAC and VF
MAC are both<b> 00:00:00:00:00:00.
</b></span></p>
</div>
</blockquote>
<br>
Well, if there is no VF netdev on the host, there is no "vf mac"
(and the igb driver wouldn't allow it to be 00:00:00:00:00:00). But
again, not important<br>
<br>
<br>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><b><o:p></o:p></b></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
~]# ip link show GE0-0</span></p>
</div>
</blockquote>
<br>
"GEO-0"? That's not a normal name for an igb PF. Are you manually
selecting this name?<br>
<br>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">3:
GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
<a href="tel:9216" moz-do-not-send="true"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#141414;text-decoration:none">9216</span></a>
qdisc mq master ovs-system state UP mode DEFAULT qlen
<a href="tel:1000" moz-do-not-send="true"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#141414;text-decoration:none">1000</span></a><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">link/ether
a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
trust off<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
trust off<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
~]#<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">* the VF is assigned
to a guest, so the admin MAC is set to the configured value
<b>(52:54:00:29:3c:bf) </b>as shown below<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
~]# ip link show GE0-0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">3:
GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
<a href="tel:9216" moz-do-not-send="true"><span
style="color:#141414;text-decoration:none">9216</span></a>
qdisc mq master ovs-system state UP mode DEFAULT qlen
<a href="tel:1000" moz-do-not-send="true"><span
style="color:#141414;text-decoration:none">1000</span></a><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">link/ether
a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
0 MAC
<b>52:54:00:29:3c:bf</b>, spoof checking on, link-state
auto, trust on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
trust on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
~]#<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">* Configure bond
interface on the guest and added member interface to the
bond
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"> On doing the above
step, bond interface mac<b> (52:54:00:29:3c:be)</b> gets
assigned to VF<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
libvirt]# ip link show GE0-0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">3:
GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
<a href="tel:9216" moz-do-not-send="true"><span
style="color:#141414;text-decoration:none">9216</span></a>
qdisc mq master ovs-system state UP mode DEFAULT qlen
<a href="tel:1000" moz-do-not-send="true"><span
style="color:#141414;text-decoration:none">1000</span></a><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">link/ether
a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
0 MAC
<b>52:54:00:29:3c:be</b>, spoof checking on, link-state
auto, trust on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
trust on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
libvirt]#</span></p>
</div>
</blockquote>
<br>
Okay, this part makes no sense, for 2 reasons:<br>
<br>
1) once the admin MAC has been set by the host, a flag is set in the
PF driver marking this VF as having an "administratively set" MAC
(that's why I call it the "admin mac"), and after that point it
should not be possible for a guest to modify the MAC address. If
your guest is successfully setting the MAC of the VF, then either
you don't have a device that uses the igb driver, or there is a bug
in the driver.<br>
<br>
2) setting the MAC address of the VF shouldn't update the admin mac
for that VF - they are separate entities, and only synched up when
the VF driver is reloaded (and in that case it is the *admin mac*
that is used as the master copy to set both of them)<br>
<br>
<br>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">* The next step would
be to restart the VM from within the VM console (* we are
not doing a shut down and start from the host *).
</span></p>
</div>
</blockquote>
<br>
Okay, so that reloads the VF driver in the guest, which would set it
to the admin mac (which you have said showed to be ....:be just
before the reset.<br>
<br>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<div class="WordSection1">
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"> At this stage, the
admin MAC is not reverted to the domain xml mac<b>(52:54:00:29:3c:bf)<o:p></o:p></b></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414"> Instead it retains
the same bond mac
<b>(52:54:00:29:3c:be)</b> which was set before the VM was
restarted from the console.
<b><o:p></o:p></b></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">[root@nfvis
libvirt]# ip link show GE0-0<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">3:
GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu
<a href="tel:9216" moz-do-not-send="true"><span
style="color:#141414;text-decoration:none">9216</span></a>
qdisc mq master ovs-system state UP mode DEFAULT qlen
<a href="tel:1000" moz-do-not-send="true"><span
style="color:#141414;text-decoration:none">1000</span></a><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">link/ether
a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
0 MAC
<b>52:54:00:29:3c:be</b>, spoof checking on, link-state
auto, trust on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414">vf
1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto,
trust on<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal"
style="text-align:justify;text-autospace:none"><span
style="font-size:12.0pt;color:#141414">So the actual problem
we face here, is after the VM is restarted and after the
members of bond interface are released, VF mac is not
getting reverted back to domain xml mac(<b>52:54:00:29:3c:bf</b>).<o:p></o:p></span></p>
<p class="MsoNormal"
style="text-align:justify;text-autospace:none"><span
style="font-size:12.0pt;color:#141414">Instead the VF mac
and admin mac are set to the bond mac “<b>52:54:00:29:3c:be</b>”
and leading to a duplicate mac issue.
<o:p></o:p></span></p>
<p class="MsoNormal"
style="text-align:justify;text-autospace:none"><span
style="font-size:12.0pt;color:#141414">My expectation was
that after the VM is restarted from the console, libvirt
should revert the VF’s MAC to the original MAC,<o:p></o:p></span></p>
<p class="MsoNormal"
style="text-align:justify;text-autospace:none"><span
style="font-size:12.0pt;color:#141414">which in our case was
the domain xml MAC, which is not happening in my case.</span></p>
</div>
</blockquote>
<br>
My expectation was that the guest would not be allowed to modify the
MAC address of the VF at all, and certainly that even if a change to
the guest MAC was allowed, that it wouldn't propagate to the PF's
list of admin mac addresses for the VFs. So we've both been
disappointed :-)<br>
<br>
Your VF and PF drivers are behaving strangely. But if you completely
shutdown the guest, then restart it, you should get the original
"...:bf" back. Even if that works for you, I would not count on the
ability of the guest to modify the VF's MAC address - that is not
the defined behavior and thus is likely to change in the future.<br>
<br>
<blockquote type="cite"
cite="mid:8c3e3dbded7c406b8d29b1cc15ccb226@XCH-RTP-005.cisco.com">
<div class="WordSection1">
<p class="MsoNormal"
style="text-align:justify;text-autospace:none"><span
style="font-size:12.0pt;color:#141414"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:12.0pt;color:#141414">Attached is the log
file as requested.<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><b><u><span
style="font-size:12.0pt;color:#141414">Interface related
contents from domain xml:
<o:p></o:p></span></u></b></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><interface
type='hostdev' managed='yes'><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><mac
address='52:54:00:29:3c:bf'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><driver
name='vfio'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><source><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><address
type='pci' domain='0x0000' bus='0x02' slot='0x10'
function='0x0'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"></source><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><target
dev='vnic1'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><model
type='virtio'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><alias
name='hostdev0'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"><address
type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#141414"></interface><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;color:windowtext">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;color:windowtext">Thiru.<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> sendmail
<a class="moz-txt-link-rfc2396E" href="mailto:justsendmailnothingelse@gmail.com"><justsendmailnothingelse@gmail.com></a>
<b>On Behalf Of </b>Laine Stump<br>
<b>Sent:</b> Wednesday, May 2, 2018 10:01 AM<br>
<b>To:</b> Thirunavukarasu Sengalvarayan -X (tsengalv -
HCL TECHNOLOGIES LIMITED at Cisco)
<a class="moz-txt-link-rfc2396E" href="mailto:tsengalv@cisco.com"><tsengalv@cisco.com></a><br>
<b>Cc:</b> Chanda Mendon (cmendon)
<a class="moz-txt-link-rfc2396E" href="mailto:cmendon@cisco.com"><cmendon@cisco.com></a><br>
<b>Subject:</b> Re: VF MAC not reverted to all zero MAC
on VM restart<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 04/27/2018 11:08 PM, Thirunavukarasu
Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at
Cisco) wrote:<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi Laine Stump,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We are running linux based VM on KVM
based Hypervisor and facing issue with respect to VF MAC.<o:p></o:p></p>
<p class="MsoNormal">Our Libvirt version: 3.2.0, host driver
IGB version 5.2.16 and VF driver IGBVF version 2.3.7.1<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><b><u>Description of problem:</u></b><o:p></o:p></p>
<p class="MsoNormal">When passing a VF to a guest, libvirt
sets its MAC according to the domain xml.
<o:p></o:p></p>
<p class="MsoNormal">On shutting down the VM or power off the
VM, libvirt attempts to restore its original MAC(all-zero
mac). There is no issue with this scenario.<o:p></o:p></p>
<p class="MsoNormal"><b>When we restart the VM, there is no
attempt to restore/revert its original MAC(all zero mac).
</b><o:p></o:p></p>
<p class="MsoNormal">This problem leads to duplicate MAC issue
on the guest (if configured with portchannel).<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Could you please help us in resolving the
issue?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
Questions like this should be sent to </span><a
href="mailto:libvirt-users@redhat.com"
moz-do-not-send="true"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">libvirt-users@redhat.com</span></a><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> rather than to an individual's email
address. This makes the likelyhood of getting an answer much
higher, and also creates an archive of the problem and
eventual solution, which may help others in the future. I'm
Cc'ing this response to that list so that any further
communication will happen there.<br>
<br>
Before reading any of the following, it's useful to know
this - for SRIOV network devices that *properly* support
SRIOV (and I consider those using the igb driver to be in
this category) each VF has two MAC addresses that need to be
discussed:<br>
<br>
1) the VF netdev MAC address (which I will just call the
"mac address" - this is the MAC that is known to the VF net
driver, and displayed in the output of "ip link show dev
$vfnetdev".<br>
<br>
2) the MAC address that is stored in the PF driver for each
VF, displayed in the "VF" lines immediately following the
"ip link" info of the *PF*, and which will be used to
initialize the VF's own MAC address when its driver is
re-initialized (I will call this the "admin mac")<br>
<br>
Some VF drivers initialize the mac to 00:00:00:00:00:00
(e.g. the Cisco enic driver) and some initialize it to a
random number (e.g. igb and all the other Intel VF net
drivers). Likewise, some PF drivers initialize the admin
macs to 00:00:00:00:00:00 and some to random numbers (as a
matter of fact, this behavior changes between different
versions of the same driver in at least one case!).<br>
<br>
Beyond this, some PF and VF drivers allow setting the
mac/admin mac to 00:00:00:00:00:00, and some prohibit one or
the other. (in many cases, a driver that itself initializes
the mac/admin mac to 00:00:00:00:00:00 also prohibits
setting it back to that same value once it has been
changed!)<br>
<br>
When libvirt sets up a VF to be used by a guest (either for
vfio device assignment, or for macvtap passthrough), it
saves both the mac and the admin mac with the intent of
restoring them later.<br>
<br>
Once these values have been saved, libvirt will set the mac
(in the case of macvtap passthrough) or the admin mac (in
the case of vfio device assignment), and send the device on
to qemu to be used by the guest.<br>
<br>
After the guest is finished running (or the device is
detached), libvirt attempts to restore the settings it had
saved at the beginning. Trouble comes up in 2 ways though -
1) once the admin mac has been set for a VF (via the PF), it
is no longer possible to directly set the mac (in the VF)
(this is done for security reasons, to prevent the guest
from changing its mac address), and 2) as said above, many
drivers don't allow setting 00:00:00:00:00:00, but in many
cases that was the original setting.<br>
<br>
As of libvirt 3.2.0, libvirt uses the following strategies
to work around these problems:<br>
<br>
a) if setting a non-0 mac (to the VF) fails, then libvirt
will try setting the *admin MAC* to that value, then
detaching/re-attaching the VF net driver. This will cause
the PF to reinitialize the mac in the VF driver. (NB: if we
do this, then we don't bother trying to later re-set the
admin MAC to its own original value; it's really not
important).<br>
<br>
b) if the failure was in setting the MAC to
00:00:00:00:00:00, then libvirt will set the mac/admin mac
to 02:00:00:00:00:00 (which should be a legal value for any
driver, but also should not conflict with and "real" mac on
the network).<br>
<br>
Okay, enough background. On to your problem.<br>
<br>
<br>
To make sure I understand your situation correctly:<br>
<br>
* When the host is booted, the admin MAC is
00:00:00:00:00:00, and VF mac is random (call it
rr:rr:rr:rr:rr:rr).<br>
<br>
* the VF is assigned to a guest (with <interface
type='hostdev'> I guess?), so the admin MAC is set to the
configured value (let's call it gg:gg:gg:gg:gg:gg)<br>
<br>
* the guest is shutdown, which causes *both* mac and admin
mac to be set to rr:rr:rr:rr:rr:rr<br>
<br>
* the next time the guest is start *its mac is not changed*?<br>
<br>
There must be something I'm not getting. Can you maybe
perform this test and run "ip link show $PF; ip link show
$VF" (substituting the netdev names of the PF and VF for $PF
and $VF) after each step to illustrate the problem? Also, it
would be helpful to add this to /etc/libvirt/libvirtd.conf:<br>
<br>
log_filters="1:util.netdev"<br>
log_outputs="1:</span><a
href="file://var/log/libvirt/libvirtd-netdev.log"
moz-do-not-send="true"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">file:/var/log/libvirt/libvirtd-netdev.log</span></a><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">"
<br>
<br>
then restart libvirtd (systemctl restart libvirtd.service),
and post the contents of libvirtd-netdev.log somewhere and
reference it in your next reply. (don't forget to remove
those lines and restart libvirtd again after your testing is
done!)<br>
<br>
Finally, you should include the contents of the
<interface ...> section of your config in your next
response.<br>
<br>
(BTW, since you're using libvirt-3.2.0, I assume that you
are using either RHEL 7.4 or CentOS 7.4. If you are using
RHEL, you should open a customer case with Red Hat so that
your problem is appropriately prioritized).<o:p></o:p></span></p>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>