[libvirt] two hostdev devices problem
Laine Stump
laine at laine.org
Fri May 24 15:49:45 UTC 2013
On 05/24/2013 09:16 AM, Dominik Mostowiec wrote:
>>> Do you have this problem with mac addresses when max_vfs is set to an
> even lower number?
First - I just got confirmation from the libnl maintainer that the bug I
mentioned causing problems with max_vfs > 50 *is* an issue in libnl
3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and
it should solve at least some of your problems.
>
> No,
> When max_fs=10, macs in VM are OK.
Interesting.
So now I'm curious about something - can you try adding a vlan tag to
your networks and see if the vlan tag is set properly when max_vfs is a
low number (10 or 7):
<network>
<name>vnet0</name>
<uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid>
<vlan>
<tag id='42'/>
</vlan>
<forward mode='hostdev' managed='yes'>
<pf dev='eth0'/>
</forward>
</network>
(you will have to do "virsh net-destroy vnet0; virsh net-start vnet0"
after making the modification to the network). If the vlan tag is
successfully set, it will show up next to the mac address for the VF in
the output of "ip link show dev $PF".
(BTW, it may get a bit confusing if you leave your networks named as
"vnet0" and "vnet1" - although it's a different namespace, libvirt uses
"vnetN" (where n is 0 - whatever) as the name for the tap device
associated with each guest interface).
To summarize:
1) when you have max_vfs = 10, you can assign the devices and mac
addresses are set properly, i.e. everything works.
2) when you have max_vfs = 35, you can assign the devices, but the mac
addresses are not set properly.
3) when you have max_vfs = 63, you can't assign the devices because you
get this error:
internal error missing IFLA_VF_INFO in netlink response
Is this correct?
>
> Phisical interfaces:
>> ifconfig eth0
> eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
Okay, so that's not the problem.
>
>> ifconfig eth1
> eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38
> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
>
>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag
>>> for ixgbe vfs was not working properly (although it works fine for igb
>>> devices)
> Domain config:
> <interface type="network">
> <alias name="hostdev0"/>
> <source network="vnet0"/>
> <mac address="52:54:0a:b1:48:fc"/>
> </interface>
> <interface type="network">
> <alias name="hostdev1"/>
> <source network="vnet1"/>
> <mac address="52:54:0a:b1:48:fc"/>
> </interface>
>
> Networks defined:
> <network>
> <name>vnet0</name>
> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid>
> <forward mode='hostdev' managed='yes'>
> <pf dev='eth0'/>
> </forward>
> </network>
>
> <network>
> <name>vnet1</name>
> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid>
> <forward mode='hostdev' managed='yes'>
> <pf dev='eth1'/>
> </forward>
> </network>
>
> Regards
> Dominik
>
> 2013/5/24 Laine Stump <laine at laine.org>:
>> On 05/24/2013 04:45 AM, Dominik Mostowiec wrote:
>>> Hi,
>>> I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G.
>>> Upgrade libnl to 3.2.16 not helps,
>>>
>>> I don't know - this is probably another problem, when I have set
>>> max_vfs=35,35 VF is attaching to VM but mac in VM is not set
>>> propertly.
>> Do you have this problem with mac addresses when max_vfs is set to an
>> even lower number?
>>
>> Ah, another thing - did you "ifconfig up" the PF before you started the
>> guest?
>>> Libvirt probably set this macs because in syslog:
>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe
>>
>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag
>> for ixgbe vfs was not working properly (although it works fine for igb
>> devices)
>>
>>
>>> 0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0
>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe
>>> 0000:01:00.0: Reload the VF driver to make this change effective.
>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe
>>> 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0
>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe
>>> 0000:01:00.1: Reload the VF driver to make this change effective.
>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub
>>> 0000:01:12.4: enabling device (0000 -> 0002)
>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4
>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub
>>> 0000:01:12.5: enabling device (0000 -> 0002)
>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5
>>> But in VM i have another macs probably old from VF.
>>>
>>> ---
>>> Regards
>>> Dominik
>>>
>>>
>>>
>>> 2013/5/24 Laine Stump <laine at laine.org>:
>>>> On 05/22/2013 06:32 PM, Dominik Mostowiec wrote:
>>>>> I tested on 1.0.5 patched version and vm with 2 vfs working fine.
>>>>>
>>>>> I have another problem
>>>>> When i set max_vfs=63:
>>>>> internal error missing IFLA_VF_INFO in netlink response
>>>>>
>>>>> Its working when max_vfs=31
>>>>>
>>>> What is the platform, kernel version, and what version of libnl are you
>>>> using (libnl 1 or libnl3?)
>>>>
>>>> There have been multiple bugs recently filed and fixed in this area on
>>>> RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that
>>>> had exactly this symptom. It was solved by a patch to libnl-1.1. If your
>>>> platform uses libnl-1.1, this could be the source of the problem. It is
>>>> fixed in the upstream libnl-1.1.4 maintenance release.
>>>>
>>>> I unfortunately am not sure where the libnl-1.1 git is (I only have the
>>>> git tree for libnl3), so I can't give you the exact commit message, but
>>>> in short the problem was that libnl was using too small of a buffer for
>>>> the netlink socket, so when there were a lot of vfs, the netlink message
>>>> containing vf info was truncated.
>>>>
>>>>
>>>>> --
>>>>> Dominik
>>>>>
>>>>> 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec at gmail.com
>>>>> <mailto:dominikmostowiec at gmail.com>> napisał(a):
>>>>>
>>>>> Hmm,
>>>>> It seems to be working (after only simple tests).
>>>>>
>>>>> --
>>>>> Dominik
>>>>>
>>>>>
>>>>> 2013/5/21 Ján Tomko <jtomko at redhat.com <mailto:jtomko at redhat.com>>
>>>>>
>>>>> On 05/21/2013 01:37 PM, Laine Stump wrote:
>>>>> > On 05/21/2013 04:03 AM, Ján Tomko wrote:
>>>>> >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote:
>>>>> >>> hi,
>>>>> >>> I try to add 2 VF by "hostdev".
>>>>> >>> Networks (vnet0, vnet1) with:
>>>>> >>> <forward mode='hostdev' managed='yes'>
>>>>> >>> <pf dev='eth1'/>
>>>>> >>> .....
>>>>> >>>
>>>>> >>> Domain:
>>>>> >>> <interface type="network">
>>>>> >>> <source network="vnet0"/>
>>>>> >>> ....
>>>>> >>> <interface type="network">
>>>>> >>> <source network="vnet1"/>
>>>>> >>> ....
>>>>> >>>
>>>>> >>> virsh create error:
>>>>> >>> "error: internal error process exited while connecting to
>>>>> monitor: kvm:
>>>>> >>> -device
>>>>> pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4:
>>>>> >>> Duplicate ID 'hostdev0' for device"
>>>>> >>>
>>>>> >> Hi,
>>>>> >>
>>>>> >> it seems we have been assigning the same id to all network
>>>>> hostdevs until this
>>>>> >> recent commit (not yet released):
>>>>> >>
>>>>> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25
>>>>> >
>>>>> >
>>>>> > If that patch has such an effect, it's purely accidental.
>>>>> Have you
>>>>> > tested this? (I plan to test a before and after as soon as
>>>>> I've had
>>>>> > breakfast)
>>>>> >
>>>>>
>>>>> I haven't tested it and looking at the code again, I might've
>>>>> been wrong :(
>>>>>
>>>>> Sorry about that.
>>>>>
>>>>> Jan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pozdrawiam
>>>>> Dominik
>>>>>
>>>
>
>
> --
> Pozdrawiam
> Dominik
>
More information about the libvir-list
mailing list