<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 12/16/2015 07:56 AM, Moshe Levi
wrote:<br>
</div>
<blockquote
cite="mid:AM4PR05MB152308FDA72FFCEBF05501E9D0EF0@AM4PR05MB1523.eurprd05.prod.outlook.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;
color:black;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">To clean up the VF I use
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">ip link set dev p4p2 vf 0 mac 0 and it
working
</span></p>
</div>
</blockquote>
<br>
Now *that* is interesting...<br>
<br>
<blockquote
cite="mid:AM4PR05MB152308FDA72FFCEBF05501E9D0EF0@AM4PR05MB1523.eurprd05.prod.outlook.com"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">24: enp3s0f0:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
master ovs-system state UP mode DEFAULT group default qlen
1000<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> link/ether e4:1d:2d:a5:f1:22 brd
ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 0 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 1 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 2 MAC 00:00:00:00:00:b1, vlan 190,
spoof checking off, link-state enable<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 3 MAC aa:bb:cc:00:00:12, vlan 190,
spoof checking off, link-state enable<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">[root@r-ufm160 devstack]# ip link set dev
enp3s0f0 vf 3 mac 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">[root@r-ufm160 devstack]# ip link show
enp3s0f0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">24: enp3s0f0:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
master ovs-system state UP mode DEFAULT group default qlen
1000<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> link/ether e4:1d:2d:a5:f1:22 brd
ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 0 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 1 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 2 MAC 00:00:00:00:00:b1, vlan 190,
spoof checking off, link-state enable<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"> vf 3 MAC 00:00:00:00:00:b1, vlan 190,
spoof checking off, link-state enable<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">It just put the address 00:00:00:00:00:b1
which I don’t know why, but as I remember the same behavior
is in intel cards (I think is related to iproute)</span></p>
</div>
</blockquote>
<br>
I just tried this with the igb driver on both 2.6.32 and 4.1
kernels, and a plain "0" is successful for me too. But, as you've
experienced, it doesn't actually set the MAC address to
00:00:00:00:00:00, but instead puts random numbers in the final two
bytes :-/<br>
<br>
So I investigated further, and found that if I use:<br>
<br>
ip link set dev p4p2 vf 0 mac 00:00:00:00:00 <-- note 5 bytes,
not 6<br>
<br>
then all bytes except the *final* byte are 0, and the final byte is
two seemingly random bytes. But if I re-run the same command many
times I find that it just rotates between 10 or so different values;
not so random (when I give "0", or "00:00:00:00" to ip link set, the
2nd to last byte is always the *exact same* value.<br>
<br>
So I looked in the source for the ip utility (in the iproute
package) and I found that the function parsing mac addresses from
the commandline just creates the buffer on the stack, doesn't
initialize it, then parses in as many digits as you specify, leaving
the rest with whatever happened to be sitting on the stack at the
time :-O.<br>
<br>
In other words, it's just a happy coincidence of a bug in iproute's
mac address parser that "ip link set .... mac 0" happens to be
successful (and that bytes 2-4 are 0 and 5-6 are non-0).<br>
<br>
I really don't know where to start / what to do with this
information. There is obviously a bug in iproute that should be
fixed, but if it is fixed before all the places in the kernel are
adjusted to allow an all-0 MAC, then users will be complaining that
their script which was working for years and years (although
probably not doing exactly what they believed) is suddenly broken.
And who knows what Hell-fury will be unleashed by some unknown bit
of code in the kernel if a 0 mac address suddenly shows up for the
first time ever. Sigh.<br>
<br>
(BTW, Cisco's enic driver, on the other hand, doesn't support
setting VF MAC addresses via a netlink message to the PF *at all*
(so libvirt has to make special accommodations), but it happily
accepts requests to directly set the MAC address to
00:00:00:00:00:00 via ioctl(SIOCSIFHWADDR) (and the interface MAC
address really does get set to all 0's). There is a script for ovirt
that uses a MAC address of all 0's to recognize that an interface is
unused, and can thus be included in a pool of interfaces in a
libvirt network. That won't work with any other SRIOV drivers
though, because even if they initialize their VF macs to 0 (e.g. mlx
and *new* (3.10+) igb (but *not* 2.6.32 igb!)), they can't be set
back to 0 when they are once again unused. Again sigh.)<br>
<br>
<blockquote
cite="mid:AM4PR05MB152308FDA72FFCEBF05501E9D0EF0@AM4PR05MB1523.eurprd05.prod.outlook.com"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">I used fedora 2.1 with kernel 4.1.13-100.
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">The most annoying part is that in
OpenStack if I use an SR-IOV VF (interface hostdev) for VM
and delete it I can’t reuse it for macvtap (interface
direct) so I have to clean the mac<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">by running ip link set dev p4p2 vf 0 mac
0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">I guess I will need to workaround it in
OpenStack.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in
0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span
style="color:windowtext"> sendmail
[<a class="moz-txt-link-freetext" href="mailto:justsendmailnothingelse@gmail.com">mailto:justsendmailnothingelse@gmail.com</a>]
<b>On Behalf Of </b>Laine Stump<br>
<b>Sent:</b> Tuesday, December 15, 2015 9:45 PM<br>
<b>To:</b> Libvirt <a class="moz-txt-link-rfc2396E" href="mailto:libvir-list@redhat.com"><libvir-list@redhat.com></a><br>
<b>Cc:</b> Moshe Levi <a class="moz-txt-link-rfc2396E" href="mailto:moshele@mellanox.com"><moshele@mellanox.com></a>;
<a class="moz-txt-link-abbreviated" href="mailto:vyasevic@redhat.com">vyasevic@redhat.com</a><br>
<b>Subject:</b> Re: [libvirt] <interface
type='hostdev'>vf configuration cleanup when VM is
delete<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 12/15/2015 01:34 PM, Laine Stump
wrote:<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 12/13/2015 10:51 AM, Moshe Levi
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi, <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have a setup with libvirt 1.3.0 and
OpenStack trunk. <o:p>
</o:p></p>
<p class="MsoNormal">Before launched the VM ip link
command show the following VF mac/vlan configuration [1]<o:p></o:p></p>
<p class="MsoNormal">When I launch a VM with <interface
type='hostdev'> via openstack api (OpenStack direct
port)
<o:p></o:p></p>
<p class="MsoNormal">I can see that the VF get the
mac/vlan according to libvrit xml [2] and ip link
command [3], but when I delete the VM the mac/vlan
config are still shown as in [3] and not restored to [1]<o:p></o:p></p>
<p class="MsoNormal">Shouldn’t libvirt restore the
mac/vlan to [1].<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The same problem exists when using
<interface type='direct'> (OpenStack macvtap port)
but just for the MAC configuration of the VF.
<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
What libvirt does is to restore the MAC address to
whatever it was before we set it up for use with a
guest. Although there are some sriov net drivers that
(for some unfathomable reason) think it's cool to assign
a random MAC address to each VF at boot time, the
"normal" thing is for the VFs to have a MAC address of
all 0's to start with. So libvirt should be saving
00:00:00:00:00:00 (it will be in the file
/var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then
setting the MAC to use; when done, libvirt will read the
00:00:00:00:00:00 and use netlink to set the MAC
address, but this is apparently failing.<br>
<br>
I checked on my Fedora 22 system with the igb driver,
and found that if the MAC address was originally set to
something other than 0's, it was restored properly by
libvirt, but if it was set to all 0's originally, the
attempt to set it back to 0 would fail.<br>
<br>
I then tried doing the same thing with the "ip" utility:<br>
<br>
# ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00<br>
<br>
and I get the following response:<br>
<br>
RTNETLINK answers: Invalid argument<br>
<br>
So it appears that either the kernel or the NIC driver
is refusing to set the MAC address to all 0's. I'm
reasonably certain this is a regression in the kernel,<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
Sigh. It appears that this has "always" been the case - I
just checked on a 2.6.32-573 RHEL kernel, and a 3.10.x
RHEL7.2 kernel, and 4.1 (Fedora 22) and both of them also
refuse to set the MAC address to 00:00:00:00:00:00. I'm
not sure if this limitation is in the NIC driver or some
basic code in the kernel.<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">although I can't say how long it's
been there, as I don't normally pay attention to this
(and as I said, many SRIOV NIC drivers don't default
their VFs to 0 MAC addresses)<br>
<br>
What distro and kernel are you using for your tests?<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[1] - 24: enp3s0f0:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
mq master ovs-system state UP mode DEFAULT group default
qlen 1000<o:p></o:p></p>
<p class="MsoNormal"> link/ether e4:1d:2d:a5:f1:22 brd
ff:ff:ff:ff:ff:ff<o:p></o:p></p>
<p class="MsoNormal"> vf 0 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> vf 1 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> vf 2 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> vf 3 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[2] - <interface type='hostdev'
managed='yes'> <o:p></o:p></p>
<p class="MsoNormal"> <mac address=' fa:16:3e:11:af:fe
'/>
<o:p></o:p></p>
<p class="MsoNormal"> <driver
name='kvm'/> <o:p></o:p></p>
<p class="MsoNormal"> <source>
<o:p></o:p></p>
<p class="MsoNormal"> <address type='pci'
domain='0x0000' bus='0x02' slot='0x00'
function='0x7'/>
<o:p></o:p></p>
<p class="MsoNormal"> </source>
<o:p></o:p></p>
<p class="MsoNormal"> <vlan>
<o:p></o:p></p>
<p class="MsoNormal"> <tag
id='190'/> <o:p></o:p></p>
<p class="MsoNormal"> </vlan><o:p></o:p></p>
<p class="MsoNormal"> <alias name='hostdev0'/><o:p></o:p></p>
<p class="MsoNormal"> <address type='pci'
domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/><o:p></o:p></p>
<p class="MsoNormal"></interface><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">[3] 24: enp3s0f0:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
mq master ovs-system state UP mode DEFAULT group default
qlen 1000<o:p></o:p></p>
<p class="MsoNormal"> link/ether e4:1d:2d:a5:f1:22 brd
ff:ff:ff:ff:ff:ff<o:p></o:p></p>
<p class="MsoNormal"> vf 0 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> vf 1 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> vf 2 MAC 00:00:00:00:00:00, spoof
checking off, link-state auto<o:p></o:p></p>
<p class="MsoNormal"> vf 3 MAC fa:16:3e:11:af:fe, vlan
190, spoof checking off, link-state enable<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>--<o:p></o:p></pre>
<pre>libvir-list mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/libvir-list">https://www.redhat.com/mailman/listinfo/libvir-list</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">F15</span></tt><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<pre>--<o:p></o:p></pre>
<pre>libvir-list mailing list<o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a><o:p></o:p></pre>
<pre><a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/libvir-list">https://www.redhat.com/mailman/listinfo/libvir-list</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
</div>
</div>
</blockquote>
<br>
</body>
</html>