[libvirt] RFC: setting mac address on network devices being assigned to a guest via PCI passthrough (<hostdev>)

[Paolo Bonzini]

On 01/20/2012 10:50 PM, Laine Stump wrote:
> To refresh everyone's memory, the origin of the problem I'm trying to
> solve here is that the VFs of an SRIOV-capable ethernet card are given
> new random MAC addresses each time the card is initialized. If those VFs
> are then passed-through to a guest using the existing <hostdev> config,
> the guest will see a new MAC address each time the host is restarted,
> and will thus believe that a new ethernet card has been installed. This
> can result in anything from a dialog claiming that the guest has
> connected to a new network (MS products) to a new network device name
> showing up (Linux - "hmm, eth0 was unplugged, but here's this new
> device. Let's call it "eth1"!)
>
> Several months ago I sent out some mail proposing a scheme for
> automatically allocating network devices from a pool to be assigned to
> guests via PCI passthrough:
>
> https://www.redhat.com/archives/libvir-list/2011-August/msg00937.html
>
>
> My idea was to have a new <network> forward mode combined with guest
> <interface> definitions that would end up auto-generating a transient
> <hostdev> entry in the guest's config (and setting the VF's mac address
> in the process). Dan Berrange pointed out in that thread that we really
> do need to have a persistent <hostdev> entry for these devices in the
> domain xml, if for no other reason than to guarantee that the same
> guest-side PCI address is always used (thus preventing surprises in the
> guest, such as re-activation demands from Microsoft OSes). (There were
> other reasons, but that one was the real "hard stop" for me.)
>
> I've come back to this problem, and have decided that, while having the
> actual host device auto-allocated at runtime would be nice, first
> implementing a less ambitious solution that uses a hand-picked device
> would not preclude adding the more complicated/useful functionality
> later. So, here's a new simpler proposal.
>
>
> Step 1
> ------
>
> In the end, the solution will be that the VF's auto-generated random MAC
> address should be replaced with a fixed MAC address supplied by libvirt
> prior to assigning the VF to the guest. As a first step to satisfy this
> basic requirement, I'm figuring to just extend the <hostdev> xml in this
> way:
>
> |<hostdev mode='subsystem' type='pci' managed='yes'>
> |<source>
> |<address bus='0x06' slot='0x02' function='0x0'/>
> |</source>
> |<mac address='11:22:33:44:55:66"/>
> |</hostdev>

In view of the discussion on SCSI passthrough, it seems to me that this 
should be attached to an <interface> element:

   <devices>
     <interface type='hostdev'>
       <source>
         <address type='pci' bus='0x06' slot='0x02' function='0x0'/>
       </source>
       <mac address='00:16:3e:5d:c7:9e'/>
       <address type='pci' .../>
     </interface>
   </devices>

> 3) I've seen requests from 2 places to do host-side virtual port
> association (i.e. vepa / 802.1Qb[gh]). Would it be feasible to do that
> association with the device after setting MAC address and before
> assigning it to the guest? (and likewise for the inverse) Or would the
> act of PCI assignment screw that up somehow? (one of the messages in the
> earlier thread says something about the device initialization by the
> guest un-doing necessary setup) (if it would work, a <virtualport> could
> just be added along with <mac address>).

I know almost nothing about this, but it does sound like another hint 
that augmenting <interface> is a better plan.

> Step 2
> ------
>
> Once the basic functionality is in place, a further step would be one
> just to simplify the admins job - we could do this by replacing this
> config:
>
> | <source>
> | <address bus='x' slot='y' function='z'/>
> | </source>
>
> with:
>
> | <source>
> | <address netdev='eth22'/>
> | </source>

   <devices>
     <interface type='hostdev'>
       <source dev='eth22'/>
       <address type='pci' .../>
     </interface>
   </devices>

> To further simplify configuration, it would be very nice if the choice
> of network device could be done automatically. Since libvirt's networks
> already have the concept of a pool of devices (and also of portgroups
> which can be used to set <virtualport> parameters), it kind of makes to
> sense to use that. In this case, a network would be defined something
> like this:
>
> | <network>
> | <name>passthrough-net</name>
> | <forward dev='eth20' mode='hostdev'> <!-- or "hardware" or "device" -->
> | <interface dev='eth20'/>
> | <interface dev='eth21'/>
> | <interface dev='eth22'/>
> | ..
> | </forward>
> | </network>
>
> (it could also contain a virtualport definition and/or portgroups
> containing virtualport definitions. Obviously, we would have to prohibit
> <bandwidth> elements (and several other things) in the definitions>)
>
> Then, in lieu of a pci address or network device name (as "netdev"), the
> <hostdev>'s <source> would have a reference to the network:
>
> |<hostdev mode='subsystem' type='pci' managed='yes'>
> |<source>
> |<address network='passthrough-net'/>
> |</source>
> |<mac address='11:22:33:44:55:66"/>
> |</hostdev>

   <devices>
     <interface type='hostdev'>
       <source network='passthrough-net'/>
       <mac address='11:22:33:44:55:66"/>
       <address type='pci' .../>
     </interface>
   </devices>

> (or, again, maybe use the separate <network> element: "<network
> name='passthrough-net'/>) At attach time, the pool of devices in
> passthrough-net would be searched for a free device, and if found, that
> device would have its MAC address changed and be assigned to the guest.
> In this case, the live XML would be updated with the pci address
> information, but when the guest was destroyed, the device would be
> handed back to the pool, and the pci address info once again removed
> from the config.

This sounds really nice, especially together with the auto-add VF 
functionality that was committed recently.

Paolo